Chapter 5

Student Assessment in Mathematics

A Note on How You Might Read this Chapter
Assessment is a major part of individual teaching and of departmental administrative affairs and in this
chapter I have tried to cover the wide range this represents. If you are new to teaching and assessment, it
may be that you want to get down as quickly as possible to the actual nitty gritty of setting and marking
exams, for example. In that case you can probably skip the rst sections and jump straight in at Section
5.5. However, the earlier sections give an overview of the importance and purposes of assessment and
you may like to return to them when you have nailed the practical side.
5.1 The Importance of Assessment
Student assessment is traditionally regarded as the bane of the academics life. Unlike schoolteachers,
most of us mark our own students work, and the load it represents is not always welcome. But in fact this
should be regarded as positive and valuable aspect of teaching in higher education, it is an opportunity
to see how your students have progressed and the impact of your teaching. It is a serious business and
taken seriously it is difcult, but it is also important. If you are really interested in your teaching then it is
informative and (hopefully) rewarding. The skills of good assessment are developed over long and varied
experience. All we can do here is provide advice, theory, suggestions and good practice in assessment
in mathematics, that may give you ideas for assessing your own students. By far the most important
input to this will however come not from literature such as this, but from talking to colleagues to get to
as wide a range of views and experience as possible. The aim is to provide fair assessment for as many
students as possible and so provide stakeholders such as employers with some idea of how the students
may be expected to perform in their professional roles. This is a tall order and requires careful thought
about the purpose and context of any assessment we undertake, including a wide range of human as well
as intellectual aspects of teaching and learning (Principle 2).
There are many issues to consider in assessing students. We will want to ensure good standards as
viewed by a wide range of stakeholders, the department, the university, employers, government, the
public, and so on. We will want to be fair to the students and give them every opportunity to show us
what they can do. We will need to ensure that the assessment provides an accurate picture of what the
bulk of the students have learned, so that colleagues can rely on that material in presenting their own
modules. Sometimes these different drivers can pull in conicting directions. For example an examina-
145
146
tion paper may appear unduly difcult (easy) to an external examiner, whereas in fairness to the students
you have made efforts, in the teaching, to take account of their strong (weak) background to prepare them
appropriately for the examination. In each case of strong or weak background you may still be providing
the students with high standard challenges, but this may not be apparent to the outsider.
There is of course a vast literature on assessment, both generic and mathematics based. As overview
material for generic assessment we mention the Generic Centre Assessment Series available on the Higher
Education Academy website. For assessment in the context of mathematics we mention the Good Practice
in Assessment Guide for Students (Challis et al, 2004) available on the website of the MSOR Network of
the HEA. Also see Supporting Good Practice in Assessment in Mathematics, Statistics and Operational
Research on the same website.
In the spirit of assessment we will also look at the evaluation of our teaching in this chapter, in Section
5.8. We have touched on this in the context of lecturing and tutoring, but give more general ideas in this
chapter.
5.2 MATHEMATICS for Student Assessment
The MATHEMATICS mnemonic of Section 1.7 can be applied to assessment, to remind us of the sorts
of things we have to keep in mind when assessing students. The Mathematical content of the module
obviously determines the content and nature of the assessment, and the main thing to ensure is that
we assess an appropriate breadth and depth of the material, and that the students know the syllabus
on which they will be assessed. And particularly in mathematics we need to be specic about what is
meant by content. The classic mathematics example here is the question of whether the students will be
assessed on all of the proofs of theorems they are taught. Such content will be spelt out more precisely
in the Aims and objectives of the course, and we have to ensure that the assessment is alligned with the
learning objectives through an appropriate Teaching and learning strategy and assessment strategy. For
example you may set coursework that is an integral part of learning for the students. This coursework
may assess learning objectives different from the examination, allowing more extended developments.
Or, if using numerical techniques on your course you may have an examination in a computer laboratory.
The question of Help and support for the students in terms of assessment may for example include the
issue of a specimen examination paper, particularly if it is the rst time you have examined the module.
You may also have a couple of revision classes before the examination.
Evaluation of assessment used to be just a matter for the external examiner, but now most modules will
usually have a moderator to check the examiners work. Also there tends to be much more scrutiny of
the outcomes of the assessment these days. Examination results that are way off the norm are scaled with
more reluctance these days and the emphasis tends quite rightly to be on ensuring that the examinations
are set properly in the rst place to avoid such practices.
For the Materials for the assessment we will have to prepare the examination paper, or other forms of
assessment, and such things as specimen solutions and marking schemes. Assessment - well this is it, the
main subject of the chapter!
Time and scheduling considerations arise throughout the assessment process - coursework distribution,
submission and feedback times, length of examination and time allowed for questions, submission times
for the papers, examination timetable and boards, etc. Indeed, assessment is one of the most fraught parts
of the curriculum so far as scheduling and time constraints are concerned because the lecturer often has
little control over it. If you get time estimates wrong in the actual teaching then you can usually handle it
yourself in subsequent lectures. But in assessment, so much relies on or affects other people.
The issue of Initial position of the students will come in of course when you set questions. Since the
147
students have just completed your module you know what their background is in your subject, and so
can design questions accordingly. However, it is rare that questions do not also call on knowledge outside
your particular course. For example in differential equations you will be relying on such basic knowledge
as integration and algebra. If the purpose is to nd out what they know about differential equations then
you might arrange for the parts of solutions relying on such background to be relatively straightforward.
Coherence of the curriculum can be reected in coherence in the assessment. So for example questions
might extend across the content to test that students have understood the linkages between topics. As
always, we need to think about the Students when considering assessment. For them, assessment is one
of their most worrying times - after all it is very important to them and can affect their condence and
possibly their career prospects. So be considerate and understanding on this issue, this is one of those
times when they need all our support.
5.3 Assessment and the Basic Principles of Teaching Mathematics
As with everything in this book we take the basic principles in Section 2.4 to underpin our design and use
of student assessment. Below we offer some suggestions about how these principles underpin student
assessment in mathematics.
Practicalities of providing the learning environment
P1. All teaching and learning must take place within limited resources (the most important of which is
time) available to you and your students
One of the major requirements of good assessment is that it be practical. All assessment is a com-
promise between ensuring the validity and reliability of the assessment and working within limited
resources. For example we could almost certainly get a much better idea of what a student really
knows from regular interviews, and indeed much use of oral examinations is made in Europe and
Russia, but this is simply too (wo)manpower intensive and beyond the resources of most UKdepart-
ments with current student numbers. Also we have already noted the crucial issue of scheduling
and time constraints in assessment.
P2. Teaching is a human activity that requires professional management of the curriculum, the group
and the interpersonal interactions that implies
Nowhere is it more important to balance professional position and ordinary human regard for stu-
dents than in assessment. We of course want to help them all we can, but there are limits to the
extent and manner in which we can do that. And we have to put aside any personal feelings to
ensure objective decisions on the outcomes of assessment.
P3. There must be clarity and precision about what is expected of the students and how that will be
measured
Our expectations of the students are expressed in terms of the learning objectives. But it is a delicate
matter. If you are over specic about objectives, then that is tantamount to telling the students pre-
cisely what the assessment will be. The precision has to be in telling the students what is expected
of them and that the assessment is based on that alone.
P4. The teaching, learning and assessment strategies must be aligned with what is expected of the
students
This is the issue of alignment between learning, teaching and assessment. For given learning ob-
jectives, the teaching and learning strategies and activities must be designed to achieve them. The
148
assessment strategy and tasks must be designed to measure achievement of those objectives, taking
into account how we have taught the students.
How students learn
P5. The workload, in terms of intellectual progression, must be appropriate to the level and standards
of the course
As in the actual amount of material taught, so the assessment also needs to be reasonable in terms
of the workload it represents. Not too much (or too little) coursework for example. Questions and
examination papers need to be of a reasonable length and difculty. We need accurate assump-
tions about prior knowledge, so for example some form of initial assessment might be necessary to
determine the real abilities of incoming students.
P6. Mathematics is best learnt in the away it is done, rather than in the way it is nally presented
And if this is how we teach, then it is how we should assess. If we have only taught students to
regurgitate set proofs of specic theorems, then we have not trained them in the skills required to
generate their own unseen proofs and so should not examine this. Of course, the best option is to
teach them how to produce unseen proofs, because this is what mathematicians really do, and then
we can set such questions - tough ones in coursework, less demanding in examination questions.
And we might have essay type questions that allow students to demonstrate their appreciation of
topics for which they have not yet mastered the details, or to test their command of higher order
skills. We might for example reward conjecturing skills in the exam, even if they do not lead to the
correct conclusions. Excellent discussions of these issues can be found in Niss [59].
P7. Mathematics is most effectively learnt if the student reconstructs the ideas involved and ts them
into their current (corrected!) understanding
This requires more open-ended types of questions, which bring out how the student processes,
analyses information and then synthesizes it into new forms. Of course this is easier to assess in
coursework or via project work but is possible in examination questions by for example testing
unseen generalizations and abstractions.
P8. The meta-skills required for the previous learning points may need to be explicitly taught
Again, such skills are much easier to assess in projects or coursework, but it is not so much the skills
themselves that need to be assessed as the deep learning that they produce. In an examination paper
for example one might ask a student to describe how they would approach a particular previously
unseen problem, without actually solving it. They could describe the actions they would have to
take, the new skills they might need to learn, and so on.
Teachers tasks (Explain, Engage, Enthuse)
P9. Good skills in explanation are required to assist students in learning efciently and effectively
(Explain)
One sometimes sees such instructions as Explain how you would ... in an examination question. If
this is a verbatim regurgitation of a set piece technique - say the adjoint matrix method of inverting
a matrix - then it is hardly testing skills of explanation. Just as good explanation is a skill that the
lecturer needs, so we should try to develop such skills in the students. We can test such things in an
oral for a project, for example, or through an essay type question in the examination or coursework.
And of course the teacher will need to exercise good explanation skills in order to set carefully
explained and unambiguous questions.
149
P10. Students best learn mathematics if they are actively engaged in the process of doing mathematics
(Engage)
No problem at all in a typical examination situation - it is one time when you can guarantee active
participation from the students! But of course coursework and project work are a different matter,
and part of such assessment might indeed comprise the extent to which the student has contributed
and engaged with the work.
P11. High levels of motivation are essential for effective learning (Enthuse)
As noted elsewhere we are quite condent of the motivating qualities of assessment! However, we
perhaps do not always think of making the assessment interesting. Generally we might describe
an interesting question as a nice question, meaning that it requires some clever twist, or perhaps
leads to an unexpected result. Of course, we have to be careful that it is not what some might
describe as a trick question, so there is a balance between interesting and ingenious to be drawn!
Also, we might construct an interesting question from a new or topical application (ensuring that
the students will have the required background).
5.4 Denition and Purpose of Assessment
5.4.1 Denition of Student Assessment
It is worth being precise about what we mean by assessment. A typical dictionary denition is [1]
To assess : to estimate, judge, evaluate, e.g. persons work, performance, character.
Actually, the use of the term assessment in the context of student assessment is relatively new - circa
1970s. Prior to that terms such as testing, examining, grading, tended to be used. Here, by assessment
we mean the measurement of the extent to which students have met the learning objectives of a course
of study.
The importance of assessment and its inuence on student learning has long been recognised. Indeed,
like it or not, in practice most teaching and learning is assessment driven. For in-depth discussion of the
denition of assessment in the context of mathematics, see Niss [59].
5.4.2 Purposes of Assessment
Assessment serves a number of purposes, some of the most important are listed below [13]:
to judge the extent to which knowledge and skills have been mastered
to monitor improvements over time
to diagnose students difculties
to evaluate the teaching methods
to evaluate the effectiveness of the course
to motivate students to study
to predict future behaviour and performance
150
to qualify students to progress.
The purpose intended for the assessment will naturally inuence the formand conduct of the assessment,
but whatever its purpose it is primarily a measure of student learning. Of course, it is not always a good
measure, and poor assessment can adversely affect student learning. So the outcomes of assessment
have to be taken in context - for example, the sort of coursework assessment used to encourage student
learning should be used carefully in predicting students future performance, if only because there is no
certainty that it is solely the students work.
Perhaps it is worth thinking about how the aims of assessment have evolved over the last few decades.
When participation rate in HE was about ve percent, university mathematics assessment was really
about nding the best of the best. The UK HE system has changed. With 40-50% participation rate the
emphasis is on ensuring that as many students as possible achieve high levels of competence in math-
ematics. First class results are now meant to reect the highest level of achievement and are no longer
intended to identify the very best, relegating the rest of the best to second class results. And indeed nowa-
days there are frequently questions raised over whether the traditional assessment and awards system
is t for purpose. We dont discuss such issues in this book, but focus on what is overwhelmingly the
mainstream practice in current mathematics assessment - coursework and examination.
Niss [59] goes into great depth on the purposes of assessment of mathematics, and makes the point that,
uniquely, mathematics has often formed a lter for entry into many disciplines and professions. This
makes assessment in mathematics particularly important. Niss also provides details of the history of the
development of assessment in mathematics. He reminds us that the written exams we currently cherish
so much were in fact introduced originally as an efcient way of assessing students as numbers increased
and made the previously universal oral examinations too costly.
5.4.3 Some Key Denitions in Student Assessment
Below are listed denitions of some commonly used terms in the study of student assessment. One can
nd plenty of material on these in the books listed in the Bibliography ([4], [15], [40]) and depending on
the educational level and objectives of the book one can nd various depths of treatment. Indeed, there
is even disagreement about the proper use of these terms. Here we cannot delve very deeply into the
relevant topics, yet we must have a rough appreciation of the terms. We therefore provide denitions
and brief discussions that hopefully give an accurate common sense notion of what is intended, while
still carrying adequate precision and rigour.
Formative assessment
This is designed for developmental purposes and does not contribute to students marks or grades, allow-
ing students to make mistakes without penalty. It includes such things as diagnostic tests, non-assessed
coursework, quick classroomquizzes. It is often argued that students wont do such work because it does
not attract marks, and various devices are employed to get round this. For example, awarding marks for
the best 50% of such work, or an unspecied selection from a number of assignments. Many lecturers
issue weekly sheets of exercises designed to reinforce the lecture material and aid learning, and may
even mark and provide feedback on these. But if there is also assessed coursework, then the students
invariably concentrate on that and may ignore the exercise sheets altogether. However, this should not
deter us from setting such formative exercises, provided the resultant workload does not exceed a rea-
sonable level. Some students may do the exercises out of interest in the subject, and those who dont take
advantage of the extra opportunities for learning that we have provided, well, that is their problem.
Summative assessment
Summative assessment is designed to establish students achievement at stages throughout a programme
and normally contributes to their marks and grades. It includes the usual examinations and assessed
151
coursework. Of course, any assessment that attracts marks has to give valid and reliable results (See
below). Examination conditions can usually guarantee this, but take away coursework cannot. In the
well-intentioned move to aid learning, coursework is used in many undergraduate courses. But of course
it is open to plagiarism, which can be difcult to prove in mathematics. For this reason marks awarded
to such coursework are normally kept a low proportion of the whole assessment (say 10-20%).
Criterion-referenced assessment
An assessment system in which students performance is marked and graded according to specied cri-
teria and standards. In theory all students could fail completely or achieve the highest grade if they fail
to meet or achieve the required standards. Such assessment is sometimes used in situations where it is
vital that the student actually has mastered key topics - say in medicine. The trouble is, it is costly for
everyone concerned, because inevitably many students will need a number of attempts before achieving
all the required outcomes. An example of this type of assessment is a test given to new students when
they start at the University of Warwick Mathematics Institute. The Institute identied a number of key
areas, differentiation, integration, trigonometry, inequalities, in which the rst year intake lacked suf-
cient skills. These are regarded as so important that incoming students were given a test in each topic,
with a requirement that they achieve at least 80% in order to be eligible for gaining full marks on one of
their modules. The basic idea is that students are encouraged to master these key techniques. Of course,
this serves the double purpose of bringing them up to speed in essential material and also emphasises
the importance of these topics to all the students. Similar criterion-referenced assessment is used in many
vocational qualications such as BTEC. Students are given phase tests incorporating key skills, which
they have to take until they achieve them all. However, the crucial difference here is that they are allowed
to take the same work away and rework it until they get it right. In such circumstances it is difcult to
guarantee effective and lasting learning.
Norm-referenced assessment
An assessment system in which students are compared with each other and placed in rank order on a
(normally!) normal distribution curve. Only a proportion of students will obtain a particular grade or
class of degree. This is the conventional mode of assessment in undergraduate courses. It has of course
the disadvantage that even a high mark cannot guarantee mastery of all aspects of the topic - it is only
evidence that they have learned selected material well enough to gather sufcient marks together.
Validity of assessment
This is the requirement that the assessment measures attainment of the learning objectives set. In math-
ematics it is usually possible to ensure this. For example it is easy to set a question that validly tests
whether a student can solve a linear second order inhomogeneous differential equation. However, there
are some notable instances where validity can be more of a problem. For example, suppose one objective
is that they will be able to construct mathematical proofs. Then a question that asks them to prove say the
irrationality of

2 would not be a valid test of this objective if they had thoroughly rehearsed the proof
in class.
Reliability of assessment
This is the requirement that the outcome of the assessment is consistent for students with the same
ability, whenever the assessment is used, whoever is being assessed, and whoever conducts the assess-
ment. Again, this is not too difcult to guarantee in mathematics because of the use of detailed marking
schemes. However, as we will see later, even when these are used there can be quite signicant variations
between different markers. Any scheme that is sufciently detailed to always guarantee reliability is al-
most certain to be complex and unwieldy, or to render the question so anodyne as to pose little challenge.
152
5.4.4 Framework for Discussion of Student Assessment
We will need some overall structure for discussing assessment, and in this chapter we will take this
simply as the order in which things usually have to be done in assessment, that is:
setting assessment tasks (Section 5.5)
marking and moderation (Section 5.6)
coursework and feedback to students (Section 5.7).
In this chapter we also look briey at the issue of evaluation of our teaching (Section 5.8). We need not say
much about this here because it will probably form part of your institutional and departmental training.
However, we can say a few things about the mathematical aspects of such things, which you might nd
useful. In particular this involves the way we use the outcomes of assessment.
Essential to developing skills in assessment of mathematics is a detailed study of the actual practice of
setting and marking examination papers. To provide this we include in the appendices copies of papers,
module specications, mark schemes and sample students solutions for examinations in engineering
mathematics and abstract algebra. These are used in the following sections to provide nitty gritty ex-
posure to the real thing. None of it is intended as good practice to imitate. Rather, regard them as case
studies designed to stimulate your own ideas, and to illustrate how you might discuss such papers in
your own department.
5.5 Setting Assessment Tasks
5.5.1 The need for Learning Objectives
By assessment tasks we mean any sort of assessment - examinations, coursework, projects, computer
aided assessment, etc. Following Principles 3 and 4 we have to design the assessment to measure what the
student is expected to have learnt, as expressed in the aims and objective of the course and to take account
of how it has been taught. The following exercise is intended to highlight this aspect of assessment. It is
of course not an exercise in mathematics, but an exercise in teaching, indeed of any subject, which makes
a crucial point that is sometimes overlooked, even in mathematics teaching at advanced levels.
Exercise - sum marking
Students have been asked to evaluate the product 1234 34 without a calculator. Mark out of ten
this attempt by one student:
1 2 3 4
3 4
4 9 3 6
3 7 1 2 0
4 2 0 5 6
Compare your mark with those of a few colleagues. Do you all agree? Discuss any differences. Why
are there differences? What is the correct mark?
In fact, without further information about the objectives (which would dene the context and purpose)
of the question in the previous exercise there is no correct mark. If the objective is to test whether a
153
student knows how to multiply such numbers then although the answer is wrong, this student clearly
knows what to do and has simply made a small slip in mental arithmetic in one carry. Ahigh mark would
therefore seem to be in order. But if the objective is to obtain exactly the right answer, with accuracy of
calculation paramount throughout and the method assumed known, then this student has not achieved
the objective and a modest mark would be not unreasonable - some might even give a low mark.
The above exercise gets to the core of setting assessment tasks - we have to be quite clear that what
we are asking the student to do will measure what we expect them to have learned. Furthermore, the
student should have been told what we expect of them. All this could simply be summarised under the
requirement that the assessment is fair, and taken very much as common sense, and indeed was taken
in this way say twenty years ago. However, we are now expected to articulate much more precisely what
it is that is being assessed. This is essential these days because we are now assessing a much wider range
of the population than a decade or two ago. As well as leading to a much more varied background in our
students it also means that with less personal contact with them (Modularisation, larger classes, etc) we
have to rely on assessment results to a much greater extent than in the past.
The common vehicle for specifying what we expect students to learn is the learning objective (Section
2.6). Learning objectives must be dened and published for everyone to see for a given module (See, for
example, Appendices 2 and 6), and then the assessment must measure the attainment of these objectives.
We say the assessment must be linked to (or alligned with) the learning objectives. This is the main
subject of this section, to design assessment tasks for various types of objectives. This amounts to the
assessment strategy, in the way that the teaching and learning strategy links teaching activities to the
learning objectives (Section 2.7). For an in-depth critique of learning objectives in mathematics assess-
ment, see Niss [59]. Like many books however, criticism of learning objectives is often on the basis of
intended learning that cannot easily be expressed in behaviourist objectives, neglecting the fact that there
is much intended learning that can and perhaps should be so expressed.
Example - the assessment of proof
This issue about clarity of objectives and its importance for high quality teaching and learning
is illustrated very well by the assessment of proof in mathematics, referred to already in this
book. If we set a question requiring a proof it makes a great difference whether the student has
been forewarned about the possibility of it coming up. If the student feels it might come up
and they rehearse it verbatim in advance then the answer to the question reveals little about
their real abilities in unseen proof - the skills they need may in fact be at the lowest level. On
the other hand, if it is intended that they will be able to handle unseen proofs in examinations
then they must be made aware of that possibility, and must be trained accordingly. Acommon
type of compromise here is to teach a particular method of proof, such as induction, but in the
examination we might set new examples of this that they have not yet seen. The point is that
we need to be clear about what skills our questions will aim to assess, and we have to inform
our students about this in the learning objectives of the module. A related issue is the extent
to which we expect students to be precise about the statements of theorems. For mathematics
students we might require explicit statements of the conditions of theorems, whereas for say
engineers we might not be too fussy as long as they have the right idea and can use a theorem.
Either way such issues need to be claried in the learning objectives.
154
5.5.2 Designing Assessment Tasks for given Learning Objectives
Exercise
You have to write a question to test what rst year students have learned about Pythagoras
theorem. Construct three questions, easy, medium and hard for this purpose. Discuss your
questions with a colleague, particularly focusing on your interpretation of easy, medium and hard in
relation to the skills that the students are expected to develop.
Almost certainly, any two attempts at this exercise by different lecturers will produce very different re-
sults. It is just like the rst exercise in this section, the objectives have not been specied clearly and so
we are all free to interpret the exercise as we wish and set questions accordingly. On the other hand, if
we used the examples of a learning objective and the guidance on MATHKIT given in Section 2.6 then it
is likely that different attempts at the exercise would be more consistent. In practice of course we would
not go into anything like this detail, but instead would use intuitive notions of what we expect from the
students and how we would assess them. The point of the exercise is to emphasize that in setting an as-
sessment task we need to decide on the balance we want between K, I, T and design the task accordingly
(Or use your own preferred means of categorizing cognitive skills that you want to assess). As already
touched on above note that K will usually include a whole range of factual items or results or elementary
techniques, and any practical assessment will only sample this. Also care is needed in ensuring that the
assessment of I and T is authentic. (If either have been seen before to any great extent then they simply
become K - a drilled higher order skill becomes a lower order skill) - these days some lecturers ensure
this by stating what proportion of a question is bookwork or previously seen.
Contrary to the views expressed by some authors, most cognitive skills can be assessed by appropriately
set unseen examinations. It has to be said that this becomes increasingly difcult as the duration of the
examination shortens. And it is a matter for debate whether this can be done to any signicant extent
in the ninety-minute module examination that is becoming common these days. The traditional three-
hour examination seems preferable here in that this is a more appropriate length of time to assess real
cognitive skills. Of course for the so-called transferable skills of communication, time management,
team working, etc, then one needs specialised forms of assessment such as projects or group exercises.
And for practical topics such as numerical methods one might have computer laboratory assessment, but
again this is a specialised area not dealt with here.
Straight K can be easily assessed by short objective questions, even by multiple choice or computer aided
assessment. Of course Kcan constitute a signicant piece of work, albeit routine - for example in inverting
a matrix or nding eigenvalues. So such skills may take up a sizeable proportion of a question. But one
would normally include also an aspect of IT - in the case of matrix theory for example this is often quite
easy to do in the form of unseen proofs, or application of routine techniques to novel situations. Another
way of incorporating assessment of IT is by essay type questions (see example below). Or one might
ask for a particular technique or method to be extended or generalised to an unseen situation. Most IT
questions will be multi-step requiring students to weave ideas together (but again unseen).
An often neglected aspect of assessment of mathematics is the issue of howit is presented. You dont have
to be teaching long to come across the poor level of mathematical presentation and arguments used by
students in their solutions to examination questions. Often this comes about because they simply copy
what they have seen the lecturer do on the board - write out a list of equations (sometimes minus the
equals signs!) with little explanation. As long as they get the answer, they are happy. You may try to
encourage better presentation skills by awarding some marks for this in your mark scheme - but make
sure the students know about this.
155
Example - the RSA encryption algorithm
The RSA (Rivest, Shamir, Adleman - [75]) encryption algorithm is a public key encryption
technique which uses prime factorization as the trap-door one way function. It is at present
one of the most powerful encryption systems known, and since it has relatively modest back-
ground requirements in mathematics it is an excellent motivating example in a rst course
in Abstract Algebra (See Appendices 5-8). The problem is that the algorithm itself is fairly
easy to remember with practice, and it is possible for a student to work through numerical
examples in recipe mode, requiring little real understanding of the rationale and theory be-
hind the method. That is, the actual coding and decoding can be, and frequently is, reduced
to straight K. It relies on three key theorems but even these can be learnt verbatim, reducing
them essentially to K. But the students are required to understand and be able to adapt and
use the algorithm, that is they are expected to develop skills at the IT level. One way that
this can be fostered is to get them to express and describe the algorithm and its underpinning
theory themselves, in their own words. So, they are expected to do this in coursework set, and
the relevant examination questions normally contain an essay component, as in the example
shown in the abstract algebra paper in Appendix 5. As one might expect, most students do
ne on the routine numerical example part, demonstrating good K skills, but struggle on the
descriptive part and usually lose marks on this IT component.
Example - integration
Integrating by partial fractions is often learnt as a routine Ktechnique, even though it is multi-
step. Indeed it is not unknown for students to do well on this question yet slip up elsewhere on
the paper on questions that utilize the component skills of splitting into partial fractions and
log integration. In the case of integration by partial fractions the process as a whole probably
provides prompts for doing the component techniques. The sort of variation on this problem
that might require IT skills (provided it is unseen) is provided by the integral included in
Question 15 in the engineering mathematics paper of Appendix 1,
I
3
=

e
2x
e
2x
4e
x
+ 3
dx.
This is actually quite subtle at this level - the student does have to link together a couple of
ideas, substitution and partial fractions, they need to know their exponential function very
well and need to be able to transfer the idea of partial fractions to a new environment. Of
course, if they have seen such things before then it immediately becomes K again!
Exercise - assessing the objective
Think of a topic you teach and formulate a learning objective on some particular aspect - remember,
you need:
an action verb
conditions
standards
some categorisation of the types of cognitive skills involved
Discuss, with a colleague, how you would assess student achievement of the objective by a suitable
task/question.
156
5.5.3 Matching Assessment to the Student Prole
Just as we need to consider the background of the students when we are actually teaching them, so we
also have to think of this in assessing them. In mathematics there is one example of this issue that par-
ticularly emphasises its importance - namely the service teaching issue. For example there are many
ways that a question in elementary integration might look completely different for a rst year engineer-
ing class and for a rst year mathematics honours class. Of course the language might look different.
With the mathematicians we may be talking about continuous, differentiable, integrable functions, etc.
For engineers it may be better to avoid such terminology - maybe put them under the well behaved
umbrella. But there is more to it than that.
With engineers, for example, one might be more lenient on the level of rigour and precision expected (up
to a point). Questions may be more oriented towards applying techniques, with less proof expected
from engineers. We may specify the method to help them, or we may allow them to use any valid
method if they have shown evidence of learning. Because they may have seen mathematics from a range
of different perspectives (say in their other engineering subjects), we may be more tolerant of slips in
notation, terminology and approach. The point is that with engineers our attitude to their abilities in
mathematics might be something like This is only a tool to them, has this student shown that (s)he can
use it provided they have the resources, and could in practice iron out the few slips they have made in
their solution? On the other hand for the mathematicians it might be This is fundamental material for
them, do they have a thorough understanding and facility in this, able to work quickly and accurately,
adapting as necessary?. The difference between these two approaches will clearly inuence the type of
questions set.
Example
In complex analysis we often refer to regions of the form0 < |z| < , as in say the statement
of Laurents theorem. This should be ne for the educated mathematics student. But for
engineers for example, when doing complex variable, it might be better to simply refer to it
as a punctured disc. In the heat of an examination a proliferation of symbols in a question
might wrong foot the engineer, when all we really want to know is whether they can actually
use the relevant result. On the other hand, mathematics students should be uent in such
things and one would expect them to be comfortable, even to prefer, a symbolic formalism.
Another area in which assessment may need to be geared to the background of students is if we have a
signicant number of foreign students whose rst language may not be English. Whilst mathematics is
virtually the only universal language it is also true to say that when a native language is used in math-
ematics it is often to convey subtlety which may be lost on non-native speakers. Amongst all the other
things we have to consider in assessment it is very hard to set papers that take such things into account.
Usually it is only after the event, when the scripts come in, or students ask you questions in the exami-
nation, that you realise a question may be ambiguous or even meaningless to a foreign student. When
this happens, you may need to take a lenient view in the marking (although of course with anonymous
marking you wont know who the foreign students are, so you will have to play this by ear!!).
And, of course, you may be a foreign lecturer. Your whole approach to such things as assessment may
be inuenced by your own cultural background, as well as language. Some nationalities provide less
support for their students than might be the norm in the UK. Some may be used to quite severe policies
on marking. So, if you are a foreign lecturer, new to UK teaching then discuss widely with colleagues to
nd out as much as possible about UK conventions and other matters of assessment .
Another way in which you might need to learn about the students is in terms of your expectations of
their abilities. Research on the expectations that lecturers have of their students abilities has shown
[20] a signicant gap between expectations and reality. And that was before the widening participation
157
agenda reached the levels of some 40% of the eligible age group! In some ways, being younger than
many academic staff the new lecturer will be nearer to their students and perhaps understand better
what their backgrounds might really be. On the other hand the chances are that the new lecturer will
come from an institution with highly qualied entrants where the standards are much higher than for
the students they have to teach. Then older more experienced hands may have a better idea of what the
base-line is. Either way, we need to familiarise ourselves with the real abilities of our students and pitch
our assessment accordingly. This is not lowering standards, but good teaching (Principle 5).
5.5.4 Validity and Reliability of Assessment Tasks
Ensuring that an assessment task measures the skills it is intended to (validity), and that it will do so
consistently in different situations (reliability) is important and difcult. These days we usually have
moderators, checkers, collaborators, etc to help us in setting our examination papers, and this may help
improve validity and reliability. Other practical measures that have been suggested are given below, most
of these being self-evident in the context of mathematics.
Improving validity:
matching assessment to the learning objectives
testing a wider range of objectives
using a number of assessments that can be compared against each other
testing under secure conditions to avoid cheating
improving the reliability of the assessment.
Improving reliability:
setting clear unambiguous questions.
insuring questions are at the right level and standard
imposing realistic time limits
use a rigorous marking scheme with precise criteria
use moderation to check setting and marking
minimizing the choices available on an examination
increasing the range of assessment methods and the duration of the assessment.
5.5.5 Methods of Assessment
As noted earlier, methods of assessment, viewed as measuring the level of attainment of learning objec-
tives (i.e. what the students have learnt), must be matched to the learning objectives, to the student prole
and be practical. Across the range of academic disciplines there is a wide choice of assessment methods
to use, but often practical issues determine the choice. Challis et al [18] describe the many assessment
methods one might use in mathematics, and discusses the functions they may full.
158
Overwhelmingly, the usual assessment method in mathematics is the unseen time-limited examination.
There is a great deal of criticism of such assessment in the literature, but as mentioned earlier, despite
claims to the contrary, a properly set unseen examination of sufcient duration can assess any cognitive
skill. No one would suggest that Olympiad test papers are in some way decient in exercising cognitive
skills!
Between 1998 and 2000 the Quality Assurance Agency conducted what turned out to be the last detailed
department by department evaluation of mathematics provision in the UK [27]. This was the most thor-
ough and detailed investigation of what mathematics departments did for their students in the UK, and
it produced volumes of useful information. One of the aspects studied was student assessment [8], and
in this there were many comments about the range of assessment, probably because this is regarded
as one of the desirable features of good effective assessment in some disciplines. The argument is that
there should be a range of different types of assessment methods, to assess the various types of skills one
wishes to develop. In some subjects this principle is essential and has long been employed. For example,
in a practical science such as Chemistry students have to develop practical skills in the laboratory, and so
they are also assessed in these.
In the past, mathematics has not been seen as a practical subject and has traditionally been examined by
unseen examinations usually lasting three hours (Or module examination of 90 minutes, while Interna-
tional Mathematical Olympiad examinations, for example, range from three to ve hours). But in recent
years, with increased use of computers in core mathematical subjects, and the increasing emphasis on
transferable skills, mathematicians are expected to develop a wider range of skills, hence the call for a
wider range of assessment methods. However, it is not the wider range per se that is necessary, but the
need to match assessment methods to the learning objectives. Thus, if a particular degree programme
is solely devoted to pure mathematics, then it is quite possible that traditional examination assessment
methods are appropriate, although even here such things as coursework and projects are now becoming
commonplace [41]. In what follows we are going to focus mainly on the time-limited unseen examination,
since that is probably what most of us will be involved with.
It is only right that in our discussion of examinations we should be aware of the reservations that some
experts have about the unseen time-limited examination as a method of assessment. The problem is that
it is easy to set bad examinations, and very difcult to set good ones, and this often brings them into
disrepute. While here we take this as an argument for thorough training (one should say education)
in setting examinations, the case for replacing them should also be considered. One of the most schol-
arly discussions along these lines is by Heywood ([40], page 274) who is pessimistic about traditional
examinations. He notes that:
other than objective tests and grading systems, examinations have been shown to be relatively
unreliable
apart from errors in scoring, assessors are not always agreed on the purposes of testing or what
they should be testing
question setting is often treated as an art rather than a science
few assessors have any knowledge of the fundamentals of educational measurement
little attention is paid to the way students learn or of howassessment should be designed to enhance
learning
if the processes of examining and assessment are to be improved then more explicit statements of
criteria showing how these should inuence learning will have to be made.
159
All this leads Heywood to propose a move towards outcomes based assessment. Heywood is actually
talking about generic aspects of assessment, and his call for sharper statements of criteria may have less
force in mathematics because of our widespread use of detailed marking schemes. But there is no doubt
that student assessment is an imperfect pseudo-science and much needs to be done to put it on a sound
foundation. This said, we still have to make the best of the current system, and any new lecturer will
have to learn to operate within that.
5.5.6 The Unseen Time Limited Examination
This, usually closed book (although there may be formulae sheets, for example), traditional form of as-
sessment is easy to present, and each student has exactly the same assessment to complete. The usual
format for the unseen time-limited examination used to be the end of year three-hour paper, but with
modularisation this has sometimes been reduced, for example, to one and a half-hour examinations (Or
two hour, particularly in the nal year) for each module. Often such changes have been brought in with-
out careful thought for their implications. For example, current modular examination schemes do not
always allow time for the assimilation of deep learning or for a student to demonstrate their in-depth
understanding of a topic if the examination immediately follows the module. The assessment of higher
order thinking skills usually requires longer examinations. In this section we are however talking about
the unseen time-limited closed book examination in general, regardless of duration.
In putting together such a paper, we have to think about the number of questions, the choice open to the
student, the duration of the paper and individual questions, etc. There is a wide range of formats - eg, 5
from 8 three hour examination, 3 from 5 one and a half hour examination. Your department may have a
standard format that you can adopt. Usually the examination is a compromise between what is desirable
for enabling the student to demonstrate what they have learnt, and what is practicable with the resources
available. One could for example assess a 100 hour module with a half hour multiple choice computer
marked test. This would be very efcient and save a lot of time, but it would not really give the students a
fair chance to showwhat they can do. On the other hand we could give each student a six-hour paper and
a one-hour viva. This would probably give every student a more than fair opportunity to demonstrate
their abilities, and would give a very accurate picture of what they know and can do. But it would be
impractical for most departments. The three-hour end of year exam evolved over a long period of such
compromise and had probably got it about right (at least for mathematics). The assumption that two one
and a half examinations are the same as one three hour examination is however debatable.
One examination paper format sometimes used, particularly when we have a very wide range of student
ability, such as in engineering mathematics classes, is the Section A and B type paper (See Appendix
1). In this Section A contains a largish number of relatively straightforward mainly K type questions
spreading across the full syllabus, and all these questions have to be attempted. Then Section B contains
a smaller number of longer, harder questions, fromwhich there is some limited choice, and which contain
signicant IT components. This gives the weaker students the opportunity to demonstrate some range
of basic understanding across the module, while still providing a challenge for the stronger students.
Once an overall paper format has been decided we have to design the questions to test what we expect
the students to have learnt. This involves a lot of professional judgement and experience. Here we can
only give some rough ideas, and by far your best input on this will be from discussions with as many
colleagues as possible, looking at past papers, or those of other institutions. And of course as a student
yourself you will have seen many examination papers. But, this said, when you actually come to put the
paper together your main thought should be for the students you have actually taught and for whom the
examination is intended. It is not good practice to take questions off the peg, from books or other papers
- they may be used for ideas, but should be rewritten with your students in mind. The questions should
be your questions, for your module, for your students. While you are designing the questions you will
160
probably have lots of ideas that you cant use all at once - just bank them for future use.
Overall the paper should cover the bulk of the main ideas of the module, and should test a range of
cognitive skills. How you measure this and ensure reasonable coverage is again a matter of judgement.
Using something like the KIT categorisation most questions should have a fair distribution of these types
of skills, and the paper as a whole should be structured to ensure that a high mark is only possible with
considerable evidence of higher order skills. This can be a difcult balancing act.
Example
Suppose you are examining a rst year methods course comprising advanced calculus, dif-
ferential equations, complex numbers, matrix theory, and vectors. Each of these topics would
have to have a corresponding question or part of a question. Also across the paper as a whole
the student should be called upon to demonstrate each type of skill fromKIT. It is well known
that, at least at the elementary level, it is difcult to set high I and T questions in matrix and
vector algebra, and equally difcult to set lowI and Tquestions in calculus. As a consequence,
if the paper is not carefully set it is quite possible for a student to concentrate on say the alge-
braic questions and achieve a good mark with relatively little use of higher order skills, and
having done little calculus. One way round this is to mix algebra and calculus in a single
question, but then it is difcult to t in signicant higher order skills in both topics. This dif-
cult issue is ultimately a matter for your judgement, and whatever departmental mechanisms
there are for supporting you in designing questions.
A few general examples might help. Thus a typical basic knowledge objective question testing little
more than memory would have mostly K, a good applied question, testing applications would have
some K and a lot of T. A good all round question might have a fair spread. A standard type of question
might have say 8/20 K, 6/20 I, and 6/20 T. This is a sort of quantication of what an experienced lecturer
might suggest - Start the question with some routine material to give all the students a chance, have
some more substantial material of medium difculty for the good students, and them something hard for
the very best. It is a lecturers informed professional judgement that must decide such matters and for
example would take into account what (s)he has done in class with the students. Thus, as noted earlier
the proof of the irrationality of

2 would be mainly K if done and drilled in class, but mainly I if only

sketched and requiring much work from the student. Related to the format of the paper is the question
of what aids the students are allowed in the examination. If they are allowed a formulae sheet then this
will inuence the types of questions that can be set. For example we cannot set straight K questions such
as the derivative of a trig function if that is actually on the formulae sheet.
Once you have constructed questions and the examination paper to cover your required learning out-
comes a nal touch you might consider to the paper is to indicate the marks awarded for each part of
a question. This is now standard at A-level, and is becoming commonplace in HE, indeed it may even
be departmental policy. My personal opinion however is that is one of the major culprits in eroding the
mathematical skills of students. It encourages the bitty approach we see so often in our students, the
mark hunting that replaces real learning. In a question for which a correct solution will take a good
student say 30 minutes I see no educational value in indicating the marks allowed for each part of the
question - indeed the question should not be in parts at all.
Example
In some A-level papers there is a tendency to replace a straightforward question such as
Sketch the function f(x) = x
3
3x
2
x + 3, indicating any intercepts with the axes and
the location of any stationary values with a sequence of sub-questions such as:
i) Factorize the function f(x) = x
3
3x
2
x + 3 and hence determine its roots [4]
161
ii) Determine the stationary values of f(x) [6]
iii) Discuss the behaviour of the function f(x) for small and for large values of x [4]
iv) Sketch the function f(x) [6]
The marks are illustrated for each part of the question. Clearly this form of the question is
much easier to answer than the shorter version. It leads the student through the question and
removes completely any skills of synthesis required to put the different steps together. It is
often argued that it helps the weaker student and gives them the opportunity to demonstrate
at least parts of what they can do. But this is a misconception of what such questions are
meant to test. What it in fact does is replace higher order thinking and problem solving skills
(I and T) by low level routine tasks (K). It fails completely in educating the student, even the
weak student, and provides them with nothing more than bitty, isolated techniques, which
will easily be forgotten because the student has not had to initiate their use.
To illustrate the above ideas the next two subsections will look at some example papers in detail.
Exercise
Survey as many examination papers as you can, if possible from other institutions and on a range of
mathematical modules, not only those similar to yours. In each case try to determine the skills that
the examiner is attempting to assess.
5.5.7 Example - a Sample Examination Paper for Service Mathematics
In Appendix 1 an examination paper for rst year engineering mathematics is reproduced. In Appendix
2 is the corresponding module description and in Appendix 3 the model solutions and marking scheme
are given. Appendix 4 contains the scripts of three students. It is not suggested that these appendices
represent good practice that should be emulated, setting examination papers is and should be, within
professional reason a personal thing. Rather it is intended as a case history on which to base discussion
and criticism illustrating points made in the text. The more such papers you discuss with your colleagues
and department the better.
The module is designed to consolidate students background knowledge to help in the transition to uni-
versity. The entry requirement is A-level mathematics grade E or equivalent, but the actual range of
background knowledge is very wide and varies greatly from student to student. Also, the subsequent
needs of the students vary greatly - some will go on to relatively mathematical rich programmes such as
electronic engineering and some will proceed to programmes with modest mathematical requirements,
such as some chemistry or biology programmes. So the examination has to be designed to ensure that
a certain basic level of mathematics is assessed, while students who need them can demonstrate higher
level skills. These requirements determine the rubric. There is a compulsory Section A earning 50% of
the marks and then a Section B has a selection of three from ve longer questions. It is not difcult for
the conscientious student to pass this examination (40%) so long as they have command of basic skills in
algebra, trigonometry and calculus, but it is also not easy to get 100% even for the best students. No aids
of any kind are allowed, no calculators or formulae sheets, so what the student produces is what they
really know. Pretests give a good idea of the incoming students background and the examination can
be used to measure the added value during the module. A substantial proportion of the incoming stu-
dents have non-standard vocational equivalents to A-Level mathematics, or lower qualications such
as BTEC and their mathematics is sometimes relatively weak. This accounts for the elementary nature
of the requirements in Section A particularly. As noted earlier, this is not a lowering of standards. We
162
accept these students in full knowledge of their background and it is our job to provide curriculum and
assessment that takes account of this.
One of the great problems with this sort of module is ensuring that well prepared students cant rely
entirely on what they have met before coming to university, since most of the material revises what they
have already seen at school. For example in the binomial theorem we want to wean them off the Pascals
triangle. And in calculus we want them to develop sufcient facility that they no longer need to rely on
explicit formulae for things such as the product rule. This is reected in the mark scheme - for example
no marks are given for the use of Pascals triangle, and time allocated for questions is intended to assume
that they have the required speed and facility. Of course, it is impossible to be sure that all this works for
all students, but at least the attempt is made.
There is no breakdown of marks on the questions. As noted above I regard that as bad practice, encourag-
ing a fragmented approach to learning and teaching. Even rst year students should be able to cope with
a 30 minute question without having to be led through it. In fact, itemising marks simply distracts the
student, especially since they only have ninety minutes. They will only waste time in trying to approach
a question strategically, weighing up which bits to try according to the marks allocated. And then there
is no guarantee that they will get it right. Also it is an essential skill for them to be able to mentally skim
through a question and assess what they can and cant do, and how much that contributes to the overall
solution. Certainly, when they go into the workplace they wont be given tasks broken down into So
much for this part, so much for that, they will just be given a job and told to get on with it. The introduc-
tion of part marks on examination papers has probably contributed more than any other innovation to
the lack of learning of higher order skills in schools (and now universities). Also, the higher the order of
skill you try to assess in an examination, the greater the likelihood that you will get the marks allocation
wrong and may need to adjust your marking scheme after the event when you nd a question is too hard
or too easy.
There are some points worth making about particular questions. For example, Question 11 is not a good
question because if students cannot do the rst part, or make a slight slip then that effectively denies them
from the rest of the question. This is a common mistake in setting papers. It could have been avoided by
actually giving the answer to the rst part in the show that form. However, the good student will check
the result they obtain by multiplying the results out, if they sense something is wrong for the subsequent
parts. In this sense one could regard the question as it is as largely IT. Question 12 is straightforward
and may appear to be rather generous, but in fact these students have great difculties with the innite
binomial series and its applications and very few attempt or complete such questions (It is interesting
that many foreign students cope easily with such things however).
Question 13 might not look it, but is all IT. In the module a point is made with trigonometric identities
that they only need to remember (very well!) two - Pythagorean and the compound angle identity for
sine. The students are then encouraged to derive all the others themselves, so they are not trotting out rote
proofs in this question. This is different to what most of the students will have seen at school where they
tend to use formulae sheets or booklets of identities. Here they are not expected to learn a lot of different
identities, rather they are expected to learn one or two very well and then develop the higher order skills
to deduce the rest for themselves. Question 14 is quite hard for students at this level. Although they have
seen this sort of question - doing integration by parts twice to evaluate an integral - in class, it is unlikely
that they would have remembered it and they will nd the plethora of symbols difcult to deal with.
However, even if they cannot do the rst part, they can still get marks for the fairly routine IT of the last
part. Finally, Question 15 is largely IT since the students have to decide on the methods to use for quite
similar looking integrals. I
3
is a test of T from I
2
but is the sort of thing that few students at this level can
cope with.
The important point from the above discussion is that to most lecturers the questions on this paper will
appear elementary and all very much K and routine, but viewed from the students perspective, taking
163
into account their limited background and the way they have been taught, the skills involved are in some
cases quite sophisticated.
5.5.8 Example - a Sample Examination Paper for Pure Mathematics
Appendices 5-7 give an examination paper in Abstract Algebra, the module specication and solutions
and marking scheme. Three student scripts are given in Appendix 8. This is a rst course in Abstract
Algebra taken by mathematics students in their second year. This is another example of matching the
assessment to the level of the students. Even students with good A-levels have (at least in the UK) had
little exposure to abstract structures and ideas. Some are not even capable of separating inductive and
deductive thinking and will sometimes prove things by giving an example (They are not so quick at
disproving things by giving a counter example however!). So they have to be eased into this sort of work
very gently. There is certainly no remorseless denition, theorem, proof format. They only meet key
theorems, such as Lagranges Theorem, and the proofs of these are dealt with fully and carefully. Also,
they get practice on fairly simple proofs such as the uniqueness of the identity. The overall objective
of this course is to expose them to the basic ideas of Abstract Algebra - elementary formal set theory,
relations and functions, simple algebraic structures, with the focus mainly on groups. A large proportion
of the course is spent on an important application intended to re the students imagination and that
demonstrates clearly how applicable abstract algebra really is - the RSA Encryption algorithm. Most of
the students are excited by the fact that with what they have learnt on the module they can understand
this very powerful coding system, can use it to code and decode and can prove all the necessary theorems
on which it is based.
For many of the students this module is the most sophisticated mathematics that they have met to date
and is more abstract than they normally meet. So to ease it in gently most of the abstract ideas are
introduced and exemplied through concrete numerical examples, and this is reected in the examina-
tion questions. In fact, there is relatively little straight K in this subject - denitions, standard theorems
and proofs, but demonstrating understanding of these and applications is largely IT. Thus, Question 1
commences with a condence boosting straightforward exercise using set operations on simple sets of
numbers. This is routine K. However, this is followed by formal proof and some riders, all of which is
IT, as it is unlikely that they will have memorised the particular proofs involved. Predictably, few stu-
dents complete this question, despite the easy start. Equivalence relations, examined in Question 2 cause
problems for many students, and again they do better on concrete numerical examples such as modular
arithmetic. There is no harm in catering for this. The students have two years of mathematics ahead of
them where formal abstract thinking will be extended and consolidated, and this module is only a start.
In fact, most of Question 2 is IT, albeit at a modest level. Virtually all of Question 3 will be new to the
students. Apart from the well-drilled proof of the uniqueness of the identity, this is largely IT again. In
Question 4 the proof of Lagranges Theorem is (hopefully!) K as it has been done carefully in class and
the students know they are expected to know it. The rest is IT. Question 5 splits into too halves. The rst
part is largely IT, since the students really have to understand the RSA algorithm to be able to express it
in their own words (Some regurgitate the summary of the actual algorithm, but few marks are awarded
for this). The second half is, provided the students have thoroughly mastered the algorithm, mainly K,
and the students invariably get most of their marks on this part.
164
Exercises
Draft an examination paper for one of your modules, giving thought to the issues raised above.
Compare at least ve examination papers from courses in your department, with the
appropriate module specications and marking schemes. Analyse the papers bearing in mind
the issues discussed in this section. If possible discuss with the setters of the papers.
5.5.9 Administrative and Organisational Matters
The traditional end of year/module examination gives the impression that we only need to think about
assessment near the end of the course or module. This is far from the case however - assessment is
one of the prime considerations during the planning of the module. As soon as we have decided on the
learning objectives we have to start thinking about howand when we will assess the students. Apart from
the actual intellectual aspects of the assessment this also includes a great deal of tiresome but essential
administrative drudgery to complete, and we will get this out of the way in this section.
There are two main administrative and organizational aspects to attend to in assessment. Firstly there
are the regulatory arrangements that your department/institution have for the assessment process, with
schedules, paper-trails, etc. All this should come to you via your institutional induction and/or through
your department assessment ofcer, so what we say here is a summary of the sorts of things that most
of us have to deal with in this respect. However, it covers the main stages that most assessment regimes
would have to conform to these days, and the checklist below can be easily adapted to your own envi-
ronment if you nd it useful.
The main milestones are:
match assessment to objectives during course planning
decide on the nature, weighting and timing of the various components of assessment
nd out the dates and venues for submission of examination papers, examinations, and examina-
tion boards
inform students in writing about the assessment process
prepare assessment, with solutions, marking schemes, etc
submit papers to examinations ofcer/ external examiner for moderation
respond to moderation of papers
attend examinations/invigilations
mark examination scripts/assessment outcomes
forward results to appropriate ofcer
attend examination boards
learn any lessons from the outcomes
other - that we have omitted and that you can add yourself.
165
Secondly there is the process of informing the students on all relevant aspects of the assessment process.
This is very important and you should make sure that they have all the information they need. Examples
of the sort of information that the students should have, in writing, are:
the units of assessment of their degree programme
the status of your course/module as a unit of assessment
the mode of assessment of your course (e.g. coursework/exam)
the examination requirements (duration of examination, rubric, etc)
the relation between units of assessment and the nal degree classication
format of the examination or other mode of assessment
materials allowed in the examination or other mode of assessment
date, time, location of examination
date, time location of publication of results
assessment schedule for coursework
coursework
feedback on coursework
Other?
One stressful administrative issue that might arise is what to do when a student disputes the way in which
you have marked their work. Now of course this depends on the degree of the dispute. But, if it cant
be settled amicably refer to the departmental processes for dealing with such things. Dont be bullied
or intimidated, but also dont be dismissive of the students case. There might be a cultural component
here - some nationalities consider it reasonable to barter and negotiate for their marks - others believe the
markers word is law and wouldnt dream of questioning your decisions! Of course, you should always
be prepared and able to justify your own marking decisions. In fact, a good test of the rapport you have
with the students is how smoothly such matters can be dealt with.
Exercise
Devise a checklist for your own purposes, enabling you to keep track of the administrative and
organizational requirements for your assessments, such as course-work and exams.
5.6 Marking and Moderation
Marking time, the moment of truth. Most of us become apprehensive at this time, not only because of the
intense work required, but because we worry about how our students have done. You dread that your
paper was too hard, or too easy, and you will get embarrassing results - average of 80% or, more likely,
60% failure rate. When the students scripts arrive on your desk you cant help but feel moved - one way
or the other! In this section we are going to look at writing marking schemes and marking the traditional
examination.
166
5.6.1 Writing Marking Schemes
The Learning and Teaching Support Network Generic Centres Assessment Series 3 [13] gives a lot of
advice on writing criteria for assessment, which in the case of mathematics usually takes the form of a
marking scheme. The essentials of good criteria are that they:
match the assessment tasks and learning objectives
enable consistency of marking
can pinpoint areas of disagreement between assessors
can be used to provide useful feedback to students.
In fact, the generic literature on criteria is confused and contradictory, and less than helpful for practi-
cal purposes. It is similar to the issue of taxonomies of learning objectives - there are a lot of different
approaches, and some contradictory advice in the literature. Your department may have a standard
format/convention/practice, which may make the choices easier for you. Here we will suggest quite de-
tailed practical advice on writing the standard types of mathematics marking schemes, which hopefully
will be easily adapted to your own needs. Essentially, we are trying to allocate marks/reward to different
parts of questions in a way that reects what we value about the skills that have been learned by the
students. There is however a range of opinion about the need and the format for this.
Firstly, some would argue that one only needs a quick run through the solution to check it is satisfactory
- in many cases this can be done in a few scribbles by the experienced lecturer. The production of a de-
tailed marking scheme is held as being unnecessary and even impossible because students may produce
different solutions. You may want to change the scheme with hindsight after the examination, it is not
always clear how best to allocate marks to parts of questions, and a global overview of what students
have done is necessary before deciding on this. There are elements of sense in all this, but there are also
good reasons why it is better to at least produce a rst draft of a scheme. It is acceptable to ne tune the
scheme after seeing what the students have done, and usually only a small proportion of students will
produce a completely different version of the solution for which you may have to bin the scheme and
treat the case on its merits.
One advantage of preparing a marking scheme is that it should force you to think through the solution as
the students are likely to do it in the light of what you have taught them. This focuses your mind on how
difcult it really is for them and exactly what it tests. Also, it is a way of communicating to a third party,
such as the examination moderator, or the external examiner, exactly what is expected from the students,
what you have done with them, what value you place on various ideas, concepts, techniques, skills, etc.
And of course it is now standard practice to provide marking schemes for internal and external quality
review processes.
So how does one construct a mark scheme for a given question? It is quite possible for a moderator to
disagree with aspects of any scheme that you produce. They might say too many marks for this, too few
for that, etc. Now, in a way, they are placed in a difcult position, because they havent sat through your
teaching, and they dont know what you have covered with the students, or what you expect from them.
However, if you have stated the learning objectives for the module and the criteria for assessment then
it should be easier for a moderator to understand your intentions. The learning objectives and criteria
should be sufciently transparent for a fellow mathematician, or for the students, to judge whether the
examination covers appropriate material. But there is more to it than that. We all know that one of the
rst questions of clarication a moderator will ask is Have you covered this is the lectures - have they
seen an example like this before? Usually this is because they are concerned about the difculty of part
of a question. For example, if part of a question asked for the proof of the result for the sum of the rst n
167
integer cubes then it would make a big difference whether or not you had gone through this in the class.
Some people now note on their scheme whether or not a part is bookwork, or seen/unseen. This is
progress, but a moderator (colleague) may still feel uncomfortable.
Going back to the sum of cubes example mentioned above, suppose you say it is bookwork and you have
done it in class with the students. One could imagine two extremes. In one you have gone through it in
great detail, giving copies of a full proof, maybe setting it for coursework, and the students have a fairly
good idea that it will come up, either because you told them, or it appears frequently on past papers. In
other words, you have drilled them in this proof. In that case the cognitive skills needed for answering
the question are not very high - barely more than K in fact. On the other hand you may have only glossed
over the proof, it is one amongst many, and students have no grounds for believing it is likely to come
up. You may for example have covered proof by induction and done examples such as the sum of the
squares of the integers, but not explicitly done the sum of the cubes. In this case a full, correct solution
would be quite an impressive piece of work, and even if the student has learnt it and remembered it, they
have still had to use the associated higher order thinking skills to do so. In this case you are more justied
in saying this demonstrates IT and allocate marks accordingly. The moderator would then have a better
idea of what the question entails and how many marks it is worth. This is why we have a hierarchy of
learning objectives - it makes it much more transparent what we mean when we say things like This is
a hard question, or This part is fairly easy. The moderator may be happy to accept your judgement on
that, but using a classication such as KIT they have more ground to do so and understand better what
you mean. Or of course any similarly informative scheme is better than nothing (e.g. [7], page 194; [63],
page 14).
Having categorised the skills required for doing a particular question, we then have to decide whether
the parts of questions requiring the higher order thinking skills (HOTS) get fewer or more marks. If
given a lot of marks and students cant do them then the average mark comes down. If the HOTS are
awarded low marks then the students wont learn them and can get respectable marks on lower order
skills alone. The common approach to give substantial marks for an initial K section such as routine
application of a method followed by lesser marks for harder parts to sort out the better students needs to
be applied carefully. It should not reinforce the idea that weaker students, who will get less marks, are
only capable of lower order skills - mainly K. Any student admitted to university should be capable of
developing higher order thinking skills, and should be expected to. Certainly, it should not be easy for a
student to pass with K only. This can be ensured by minimising the amount of real K and constructing
questions such that the bulk of the marks come only for I and T. This is not easy and, in fact, setting good
examination questions is a difcult job requiring substantial time and effort, and normally involving a
number of iterations.
To construct the marking scheme rst draft the paper, maybe getting ideas from other examination pa-
pers, textbooks, etc. Unless the course is completely new then there will be a few past papers about. Now
work through each question, writing the solutions out as if for a real examination, putting yourself in
the students place and obviously not using any short cuts that your superior knowledge provides you.
Some lecturers for example use mathematics software to construct and work through questions to save
time. Certainly, use such aids for ideas and for nding good numbers that work out conveniently, but
in the end you have to go through it long hand yourself, exactly as the students will have to do it in the
exam. Then nd some comparison factor for your solution and a typical students solution, so as to get
an idea of the length and time taken by a student. Ask around to get ideas for this. Examples are two A4
sides per half-hour question, your time = two-thirds of student time, etc. There are clearly problems with
all such rules of thumb, but they are better than nothing. The idea is to be sure that the typical student
can make a reasonable attempt at the required number of questions in the time allotted.
Next, start allocating marks to the various parts, now that you have a rough idea of the relative difculty
of each. In parallel with this classify the parts by KIT, or whatever other scheme you use to check the
168
learning objectives are covered ([7], page 194; [63], page 14). Your part marks will depend on this, and
on how much you expect students to develop each skill. Again, this may take a number of iterations.
And you will need to compare this across the different questions, as well as across the possible choices
the students can make ([7], page 194, classies the difculty of examinations using a similar idea) to get
an overview of what various overall marks for the paper would mean. For example if a paper asks for 3
questions out of ve, and two of the questions have 12/20 marks for K alone, then a student could pass
(at 40%) this paper employing no HOTS at all. So allocation of marks is clearly a delicate optimisation
problem, guided by common sense!
Having done all this, tidy up the paper, the solutions, with your mark scheme, and ask someone else to
look at it. If it passes this hurdle, then it can go off to the examinations ofce. But, that may not be the end
of it. It takes a lot of experience to set a good, fair, challenging paper that wont result in a high failure
rate (above 10%). You dont really know how fair a paper is until you see how the students nd it, at
which point you should be prepared, in consultation with colleagues, to change your views if need be!
Finally some summary advice for writing mark schemes:
consult other marking schemes to get ideas
rst write out a model solution - as the student would do it
allocate each mark so that it is associated with something that is there or not there, right or wrong
check the overall balance across the paper
aim for a scheme that is usable by the educated layman
make it reliable, so that anyone can mark the answers and broadly agree on the marks
penalise once only for each error - carry through consequential marks
pilot the marking scheme
be exible in accepting solutions that dont t into the marking scheme.
Afrequent concern in constructing a marking scheme is the level of detail to employ. For sake of argument
let us focus on a typical half-hour examination question, and suppose we have to award this a total of
20 marks. This is a common level of award, the usual intention being that one would only award to the
nearest integer. This is about the maximum level of resolution that is possible when assessing how the
students have performed on such a question - and some would argue that this is too optimistic. Of course
this can still lead to problems. The most trivial mathematical statement can be broken down into parts
almost indenitely and we might feel a need to split our mark scheme further. Dont bother - assessment
is simply not that exact a science and if our mark scheme is a blunt instrument there is little point trying
to sharpen it! The same applies when we actually come to mark the scripts - see later.
5.6.2 Example of a Marking Scheme in Service Teaching
In Appendix 3 we give the marking scheme for the Engineering Mathematics examination of Appendix
1. Again it is not intended to demonstrate good practice, but simply to form a basis for discussion. As
always the dry numbers conceal a great deal of hidden assumption and background knowledge of the
students, and here we make a few of the more important points.
Most Section A questions are not broken down in submarks. These questions are so straightforward
that one can view attempts at them overall and make judgements on the hoof. As we will see later, the
169
variation in approach to these questions, and the types of errors made can be so wide that any ner mark
scheme would simply get in the way. Questions in Section B are broken down into clear submarks, but
this will not always be appropriate for all students solutions, so can be overridden by judgement on
the y, if need be. Also, there are a number of hidden assumptions of which only the setter would be
aware. For example, in Question 6 it is well known that many students will leave the answer in the form
sin(2x + x) - and indeed if you ask them to differentiate something like this some will retain the 2x + x
split in the process. Now although it is not explicitly stated in the mark scheme I would certainly deduct
a mark for this. The point is that no marking scheme can cover all such eventualities.
In Section B questions the breakdown of marks is very crude. Question 15 nicely illustrates the folly of
putting part marks on the examination paper. Part of the test here is to see whether the student has an
overall appreciation of the different methods involved and can judge for themselves the likely amount
of work in each case. How does it benet the student to know that there are the same marks for I
2
and
I
3
? Do they then assume they are equally difcult, which for most students they are not, of course? And
in some cases displaying the marks can give away the game - for example when a question involves
evaluating contour integrals by a range of methods such as Cauchys integral formulae, parametrisation
or Cauchys Theorem. The last is much the easiest and if you reveal that by displaying the marks the
students will know to use that method. We are assessing the students understanding of mathematics,
not their strategic skills in harvesting marks.
Overall then, a marking scheme has a signicant personal dimension, informed by what and how you
have taught the material, what you expect from your students, what you know about them and so on.
And it is a framework only, not a straitjacket.
5.6.3 Example of a Marking Scheme in Abstract Algebra
Appendix 7 gives the marking scheme for the Abstract Algebra paper of Appendix 5. This is quite de-
tailed in some cases, reecting specic tasks set in each question. It does make the marking harder, but
on the other hand we are looking for quite specic knowledge and skills. The marks are fairly self-
explanatory. Some people might break down the rst part of Question 5 into more detail as one can with
essay questions - listing the points required, perhaps allocating some marks for clarity and presentation,
etc. However experience shows that it is very difcult to bend such schemes to t the attempts of different
individuals - it is better to treat each attempt on its merits and rely on overall judgement for consistency
and fairness. In any case, marking essays is notoriously unreliable.
The overriding point about the above examples is that the marking scheme is very personal and reects
the setters interests and priorities, always tempered of course by the usual collegial agreements on what
the students need to know and be able to do. For example in a rst year methods class a lecturer short
of time may decide to skip through hyperbolic functions, regarding them as not very fundamental or
important, setting them, if at all as minor riders on an examination question. But if a subsequent course,
say in partial differential equations or complex analysis, makes signicant use of them then they might
need a more thorough foundation. So, for the same questions, different lecturers may have different
marking schemes.
170
Exercises
Construct a marking scheme for the examination paper you drafted in Section 5.5 explaining
your rationale for the scheme.
Discuss the marking schemes for the ve papers you looked at earlier with the setters of the
papers.
5.6.4 Marking Examination Papers
With a good marking scheme at hand marking is not too bad, except that it almost invariably has to be
done quickly and accurately, which some people nd quite stressful, because after all it can mean a great
deal to the students. You have to be fair without being too harsh or too generous (This actually depends
on whether the marking really matters - for example you can afford to be a bit harsh in coursework, and
maybe less so in a nal examination). You need to be sensible - time spent anguishing over a precise
mark is rarely fruitful, student assessment is simply not that rigorous a process and it may be better
to step back and look at a question globally to get an overall feel for what the solution is worth. If a
student makes a minor error early on in the question which renders it more difcult to continue than you
intended then try to nd clues that the student did know how to proceed but got bogged down in the
calculations. This is another area where your best source of advice is discussion with your colleagues. The
suggestions discussed here may be useful, but again you will nd varying opinions between lecturers.
However most will agree on one aspect of being sensible about marking, which is the policy of revisiting
borderline cases. For example few would return a mark of 39%, and indeed nowadays most departments
have automatic replacement by 40% in such cases.
Marking a question at a time (horizontal marking) improves reliability although it is wise to reorder
scripts between each question so that the same students are not repeatedly victims of any weariness and
irritation you might have after marking the same question many times! On the other hand papers of short
questions can be done in one go, and indeed this might be the most efcient method if there are a lot of
such questions on the paper. For example Section A of the paper in Appendix 1 could be marked in one
go for each script. These days most departments have anonymous marking, at least for examinations, and
this minimises any personal bias in marking. Avoid the halo effect in marking (allocating excessive/low
marks for reasons other than objective interpretation of the marking scheme). An example here is to
avoid being unduly inuenced by untidy work, poor handwriting, etc. Only up to a point of course, but
if you can read the material and the student clearly knows what they are doing, then unless you have
specically indicated that there are marks for presentation it seems churlish to penalise for such things.
Also, as you mark your mood can change - dont let it inuence your marking.
Sometimes you are really not able to read a students handwriting, how are you going to mark his/her
work? It depends on the degree here. One unfortunate consequence of widening participation has been a
greater variety of presentation styles, to put it kindly! Some work is now so badly presented that some
lecturers have indeed taken to awarding presentation marks to encourage greater care in student work.
The bottom line is that you cannot award marks for things you dont know are there. So if you cannot
decipher what a student has written, then you have no grounds on which to award the marks - although
there are of course shades of giving the benet of the doubt that you might exercise. You might look for
evidence elsewhere that the student knows what they are doing, but quite frankly if you need to work
too hard to compensate for their bad writing then they probably need pulling up over it anyway.
A practical point is worth making here. Mainly for short questions, but sometimes also for longer ones, it
is not unreasonable to simply use the students answer (Provided it is correct and not given in the question
171
of course!!) in allocating marks. Provided you are sure that the student couldnt have got the answer by
guesswork or some other means avoiding the necessary working, then you can take the working for
granted. This speeds up the marking considerably. What you cant do however is automatically award
zero marks if the answer is not correct - them you have to look more deeply into what the student has
done.
In general you should stick to the marking scheme unless there is a compelling reason to change it.
However, you should always be prepared to rethink a question in the light of experience. For example
you might realise halfway through the scripts that one question is actually harder than you thought. This
is quite a common problem, particularly when you are taking a course for the rst time. As always it
depends on a number of things. If the implications of the resulting marks are not that great (formative
non-assessed work, or coursework that contributes very little overall) then dont worry about it too much
and just tell the students the marks are low on that question because it was too hard. But be consistent
in your marking and dont change the way you mark for subsequent scripts. On the other hand, if
the impact of the marks will be signicant (e.g. a nal examination paper), then you may need to do
something about it. Two extremes are to go back and remark according to a more appropriate marking
scheme or to continue marking in the same vein, but at the end scale the marks in some way. The rst is
by far the fairest but of course takes longer, while the second will hardly slow you up at all, but runs the
risk of treating students inconsistently. Provided there is time, the responsible teacher will go back and
remark, taking this as yet another lesson of how important it is to set questions carefully in the rst place!
Indeed one should be continually evaluating ones questions as you mark to see how they could have
been improved. Incidentally, the danger of this problem is one more argument for horizontal marking,
and also for having a brief skim though the scripts before starting to mark to see how students have fared
generally on each question.
On giving the benet of the doubt generally, think about what the student has learnt, rather than what
they have not. You can make mistakes in setting the paper - maybe the question is less precise than
it could have been and was open to misinterpretation. Or perhaps you have not allowed for the weak
English of foreign students. Perhaps adopt the general policy - be hard on mistakes when the results do
not affect the student too much, less so in important summative work.
While we are constantly exhorted to be consistent and totally objective in our marking it is worth remem-
bering that this is an imperfect world and occasionally we might need to exercise generosity or severity
in the wider interests of the students and the teaching. The ofcial line might be - You should never be
generous and you should never be harsh!, and we should be scrupulously fair and objective in all your
marking. But of course, like all generalisations, it isnt always like that. Basically your yardstick in this
decision should be What is in the best interests of the student. This doesnt mean giving them marks
they dont deserve, and may even mean being a bit harsh when deserved. On the other hand there is no
point in being gratuitously hard on them, we must always have a considered educational reason for our
decisions. After a long late night spell of marking it is easy to become irritable and dock marks unrea-
sonably if someone makes a silly mistake. Resist this and stop marking if need be - you are getting too
tired.
We mentioned above that that the degree of resolution of the marking scheme can cause problems, and
particularly for new lecturers these sometimes emerge at the marking stage. For example you might have
a short question attracting four marks. The student makes a minor error so you dont think it is worth
full marks, but dropping it to three seems a bit severe, so you feel like awarding a half mark. Of course,
if that is down to you and you are doing all the arithmetic to collate the marks at the end, then no matter.
But you might consider whether it is worth anguishing over anyway, and in fact some departments wont
allowawarding half marks in this way. The issue about half marks is really one of not having a sufciently
ne grading scale to resolve decisions on what to award and being forced to perhaps penalise something
too heavily, or to give too much benet of the doubt. The rst thing to do is clarify how important such
172
decisions are. If we are talking about marking a nal year examination then they are quite important. On
the other hand if it is a piece of coursework that only contributes a couple of percent to overall assessment
then clearly 3 or 4 is neither here nor there. Some argue they will never give full marks. Why the heck
not - if the student has achieved all you have asked of them what hidden requirements are you basing
your penalties on? On the 1 4 scale talked about here that means students can never achieve more
than 75% - one hopes the reason for that is explained to them carefully! On the other hand, if you are
doubtful about giving three even though they have made a small mistake, remember that this is still a
rst class mark. Another ploy is to compensate for what you consider might be a harsh judgement on
one question by being a bit lenient on another question. On a typical half hour question attracting say
twenty marks the use of half marks is a waste of time, a resolution of 1/40 overall is simply not reliable
in such assessment. No instrument is capable of reliably distinguishing between say 12 or 13 in such a
question, so just make a choice. In the unlikely event that it will make any difference (say a borderline
fail case), then look back at it and take any action necessary. The main thing is to be sensible, practical
and fair throughout.
Sometimes a students solution departs fromthe marking scheme and they produce a completely different
but also correct solution that you had not allowed for. This may mean the student has shown air and
initiative and deserves good marks. It may also mean that they have not learnt what you taught them,
but have relied on previous knowledge from another source - that is their solution does not constitute
evidence that they have achieved the learning objectives of the course.
Example
Suppose in a differential equations paper you ask the students to nd the general solution to
the rst order equation
x
dy
dx
+ y = x.
Now this can be solved in at least three ways. It can be treated as homogenous and solved by
substituting y = xv. It can be treated as linear and solved by integrating factor. It can also be
solved by direct integration by noting that
x
dy
dx
+ y =
d(xy)
dx
.
You may not care how the students solve it and might regard the student who uses the third
method as showing air and command of the subject. But if the intention was to test whether
they can solve homogenous equations, then clearly they have not demonstrated this - ditto for
integrating factor method. Of course one could say Solve by integrating factor, or some such
clarication, but that is partly giving the game away. The proper approach, if you want to test
whether they can use a particular method is to give an equation which can only be solved by
that method. Or, you could ask them to solve it by all three methods.
But the question remains, what do you do if your intention was indeed to test the integrating
factor method, thats what your mark scheme assumes, and yet a student uses a method you
havent taught them? If they treat it as homogenous then it may be that they are using knowl-
edge gained elsewhere. Their solution contains no evidence that they have learned anything
from you, they have demonstrated no knowledge of the integrating factor method and you
would be entitled to award low marks. On the other hand if they use the third method then
it could have been that they have been quick-witted and spotted the integrating factor imme-
diately (1!). On the other hand they could simply have spotted the product rule and have no
knowledge at all about the integrating factor method. This sort of situation is difcult. On
your mark scheme you will probably have a proportion of the marks allocated to nding the
173
actual integrating factor, and you can simply not award the student these marks. But then
you might be doing them an injustice, and it will be your fault because you didnt make the
methods clear or the question unambiguous enough. The message is, think carefully about
how students might attempt questions before you set the mark scheme, but if you do inad-
vertently get into this sort of tangle, act in the best interests of the students, and be prepared
to accept responsibility if need be.
These days most departments/institutions have quality control measures requiring checking or double
marking of samples of scripts. This is a well-intentioned attempt to improve reliability of marking and
fair treatment of the students. Find out what the policy is in your department. There is much variability of
practice in this area across the MSOR HE community. At the time of the 1998-2000 Subject Review ([27],
[8]) none of the top performers in assessment did very much in double marking examination papers.
Most simply did clerical checks on the addition of marks, and checked that every page of answers had
been marked. Examiners decisions on marks allocated to the students attempts are rarely questioned.
The marking scheme could still be changed after marking, if the marker thinks a question too hard/easy,
or was uncertain about how the class might get on with the question. The fact is that double marking
needs additional resources and there is no strong evidence that it actually improves reliability [13].
Once marking of all questions is completed, total the marks and nd out how the students have done on
the paper. Few academics are unfeeling enough to be relaxed about this. You want the students to have
done well, as evidence of your good teaching, but on the other hand you dont want marks that high
that you are seen as having low standards. If your results are at all unusual (low average, high failure
rate, wide standard deviation, lack of discrimination, etc), you must be prepared for some objective soul
searching. Particularly if other examiners of the same students have reasonable marks, the exceptional
results are more likely to be your responsibility than the students, and you have to deal with it. Normally
the department will have processes in place for this, possibly some sort of scaling procedure. However,
even among the departments with proven good practice in assessment, as recognised in the 1998-2000
Subject Review, there is little consistency about this. Approaches range from having processes in place
that ensure examinations are set properly and therefore need no scaling, to a blanket policy of scaling
precisely to ensure uniform marks across all modules. For the new lecturer this issue will probably be
out of your hands anyway.
Remember that the prime purpose of the marking scheme is to ensure consistency between different
markers or between different scripts. In practice, one will still get variations. The Open University has
very detailed marking schemes for some of its tutor marked assignments in Mathematics and yet they still
have moderators to check on peoples marking. The fact is, the reasons people award different marks are
varied and unpredictable. We get vindictive villains and soft-hearted fools in all walks of life, academia
not excepted. Strategies such as double marking and marking schemes can mitigate against too much
variation in this respect, but wont eradicate it completely. The fact is we live in an imperfect world. One
may take consolation for this by regarding the unfairness and inconsistencies students experience in as-
sessment as valuable preparation for the unfairness and inconsistencies they will undoubtedly experience
throughout their lives in all sorts of judgements of their abilities.
Example
Suppose we have a question on solving a second order inhomogeneous linear differential
equation with initial values by complementary function plus particular integral method. Of-
ten a student will obtain the general solution as the sum of the complementary function and
particular integral correctly, but then apply the initial conditions to the complementary func-
tion alone, not including the particular integral. This is quite a serious misunderstanding of
the method that few marking schemes would allow for in advance. It might attract a large
penalty, even though it might be considered that the bulk of the work has been done when the
174
general solution is obtained. On the other hand, it is not uncommon for a student to get the
general solution, and apply the initial conditions correctly, only to make an elementary arith-
metic error in nding the arbitrary coefcients. In this case, it is not easy to justify docking
any marks at all. It is unlikely that a marking scheme designed for such a question beforehand
will cover both these cases, allocating marks according to what the student actually does. It
simply calls for sensible judgement after the event - and no two lecturers are likely to come to
the same conclusion on the marks.
5.6.5 Detailed Examples of Specic Questions
Here we are going to look in detail at the marking of a couple of examination questions from the Engi-
neering Mathematics paper given in Appendix 1. This is not to suggest these are good questions, or good
marking, but simply to bring out and illustrate a number of points that frequently crop up in marking.
Also, it illustrates the sort of detailed thinking one has to go through even in the simplest questions. And
this has to be done quickly. And sometimes you might skim quickly through a students solution and
jump to the wrong conclusion at rst glance. Remember that in this paper no aids such as calculators or
formulae sheets are allowed. The rubric insists that solutions be legible and that explanation is clear, but
there are no explicit presentation marks.
Question 3. Expand (2 + 3x)
5
by the binomial theorem.
This would seem to be an innocuous question, as it was intended to be. It is basically K with perhaps a
little I because of the coefcients. There are 6 marks for this question out of a possible maximum of 120
for the paper as a whole. It has been used in workshops with over 300 delegates so has been thoroughly
worked over!
The rst point about this question is that most students will have seen the binomial theorem before but
almost invariably by Pascals triangle. However, it was made clear in the course that they would be
expected to learn the general binomial expansion with the coefcients n(n 1)...(n r + 1)/r! because
this is easier to use for large powers and applies without modication when the power is not a positive
integer. Now predictably many students simply use what they already know and stick with Pascals
triangle. Strictly speaking such students have given no evidence that they have actually learned what
was expected. On the other hand they may simply have chosen for them the simplest option if they can
actually remember the coefcients from Pascals triangle. This is a typical situation where the student
departs from the mark scheme and you are left with the problem about what marks to award. In fact,
for engineers one would probably not be very severe here and would award most of the marks - another
question on the paper might test what they do for negative powers for example. Unless one phrases
earning outcomes and questions in a clumsily precise manner - such as Expand by the binomial theorem,
but not using Pascals triangle then the student has achieved the task set in the question, but on the other
hand might not have learnt an important technique. In general, re-teaching a topic by a different method
is very difcult - You cant teach a young dog new tricks! Of course you can get round this by using a
marking scheme heavily weighted towards the correct choice of method - indeed all the marks may be
awarded for this, with errors in the details of calculation ignored. On the other hand, if you are happy
for them to continue with the method they prefer, then the marks would be more evenly spread across
method, accuracy, etc. Either way the students must be aware of this beforehand.
Some students might evaluate the expansion by longhand expansion of the brackets without using the
binomial theorem, and then clearly they have not demonstrated that they have learnt this and you would
be entitled to give zero marks. But if they for some obscure reason they expanded it in the form (2 +
3x)
2
(2 + 3x)
3
using the binomial theorem, for the second factor and then expanding longhand, then you
might show some generosity and award say half marks. As noted earlier, it also depends on the outcome
175
of any marks for the student and the purpose of the assessment.
Another point about this question is that it is possible to get the answer almost completely wrong by
making the single small error of forgetting to include the 3 in the expansion - that is they write just x
r
or 3x
r
instead of (3x)
r
. Now this is a very simple oversight for the student to make in the heat of the
examination and otherwise the student may understand the binomial theorem perfectly well. It is not
over generous to penalise this with the deduction of just one mark out of the six, even though the nal
result might have ve out of six of the coefcients wrong. Possibly, we didnt allow for this error in the
mark scheme. Few schemes can be so detailed that they pick up every possible error, and then experience
and judgement is called for. Ask around to get a range of opinions. Give the benet of the doubt where
appropriate. Ask yourself whether the student has really demonstrated that they have learnt what the
question was intended to test. If they have not, have they still demonstrated some ability that is worthy of
merit? Is the question explicit enough to render the mistake important (Cf: the multiplication question in
the SumMarking exercise of Section 5.5 - howserious is the error for the learning objective being tested?).
Does the student really know what they are doing?
Now lets turn to the actual marking of this question. Appendix 4 gives the attempts by three students,
A, B, C. The attempt of Student A has been used in a number of workshops to illustrate the variation in
marks between different markers. Since the students have no formula sheet they are expected to knowthe
binomial theorem for (a +b)
n
for positive integer n and be able to use it for any a and b. Now the student
has indeed learnt the form required and the rst line has the correct factorial coefcients. However, they
have made the common error referred to above of forgetting that in this case b = 3x, instead taking
b = x. This is the only error in the solution, which is subsequently continued correctly. In a more detailed
marking scheme I might for example give 3 marks for writing down the correct general expansion of
(a + b)
n
, two for the correct choice of a, b, and n and one mark for the subsequent arithmetic calculations
and nal result. So for the slip of using x instead of 3x I might deduct one or at most two marks, and this
student deserves at least 4 marks out of the six - I wouldnt quibble over 5. In fact, the marks given for
this by 25 delegates in one workshop were:
Mark: 0 1 2 3 4 5 6
Number of Markers: 1 3 3 9 6 3 0
with an average of three. Even though the scheme is not explicit enough, three out of six, is still harsh.
This student clearly knows the binomial theorem and has made a small slip in applying it in this case. To
award only half of the marks is simply unfair. And to award less, as done by over a quarter of the markers
is unacceptable. The zero mark was awarded by a German delegate who was used to awarding zero for
any solution that was not perfect! This exercise was not completely realistic because normally you would
be marking your own paper, with which you would be familiar (However, it is worth remembering that
it is exactly how the UK Gold Standard A-level is marked - by commissioned markers marking to a
scheme provided by someone else).
Question 5. An arc of a circle of radius

3 subtends an angle of 30

at the centre. A tangent is drawn to the circle

at the midpoint of the arc. Find the areas between the tangent, the arc and the extended radii bounding the arc.
What is the length of the arc?
Again we look at the solution of student A in Appendix 4. In this case the student needs to Know (no
formulae sheet) how to nd the area of a sector of a circle and the length of arc of a circle, including
the need for conversion to radians. In addition they need to know simple geometrical facts such as the
radius to a tangent bisects the angle and meets the tangent at right angles, and they also need to be able
to nd the area of a triangle and be familiar with the trig ratios. No credit would be given for the latter
elementary points, which we would assume is part of their background knowledge. But marks would
be awarded for knowing the area of a sector and the length of arc, because these were in the module
syllabus. In addition the student had to have Internalized these results to the extent that they could see
176
when they could be used in this problem and had to Transfer them to a new situation, synthesize ideas
and solve quite an involved problem for this level of student. This is a typical example of a question
that is quite unpredictable when you set it. It is one you suspect a large number of students wont even
attempt - and indeed this was the case. But it is one that sorts out the better students. It is the sort that is
extremely difcult to mark fairly, and this was reected in the nal spread of marks among 22 markers at
one workshop:
Mark: 3 4 5 6 7 8
Number of Markers: 3 2 4 1 1 11
The students solution is in fact a good attempt. They have structured the overall approach, nding the
difference of two areas, and they have made correct use of the area of sector (OK the student says area of
arc, but this can happen to anyone in the heat of an examination). In fact, they have done this from rst
principles. They have also found the length of arc, with correct use of radians where appropriate. But, of
all things, the student has got the area of the triangle wrong because they mistook

3 for the length of

side of the triangle when it is actually the radius of the circle. This is just an elementary slip in calculating
an area that was not really being examined. So what is fair mark for this student? It is almost certain that
we would have to depart from any prior mark scheme set up, since it is unlikely to have anticipated this
sort of error. This is where common sense comes in. This student has shown a great deal of higher order
skill and has made one elementary slip while using an inappropriate formula for calculating the area of
a triangle. One could not in all reasonableness deduct more than two marks for this (25% of available
marks), perhaps justifying it by noting that their explanation is not that clear. So an overall mark of 6 out
of 8 would not be over generous. 7 would not be outrageous, but clearly 8, given by half of markers is
not acceptable. The average mark was in fact near enough 6, but the range is clearly a cause for concern.
But no matter how we interpret these results it makes clear how variable marks can be between different
markers. When marking your own papers, or marking to a very detailed mark scheme, the variability
should be considerably reduced, but you can see that even then one has to make a lot of complicated
decisions and judgements very quickly. And all of this is important - we have to strike a balance between
being as fair as possible to students, while maintaining good standards, and not spending too much time
in the process. This sort of question is one where you have to read the students attempt very carefully to
locate the error and judge its signicance. On the other hand as noted earlier it is also the sort of question
which most students do not even attempt anyway!
Exercise
Mark the attempts at Question 3 made by Students B and C and compare with a colleague if possible.
If you have time you can also look at the marking of other questions in Appendices 4 and 8. As before
you will get more out of this if you can compare your marking with a colleague and discuss the
results.
Or, better, repeat a similar detailed analysis of your own marking or that of a colleague.
5.6.6 An Example of Marking a Full Paper - Engineering Mathematics
Exercise
Mark the full scripts of students A, B, C, given in Appendix 4, for their attempts at the engineering
paper of Appendix 1 .
To illustrate the variability of marks one can get in marking even a straightforward paper we give below
177
the marks (out of 60) for the attempt of Student A at the Section A part of the engineering paper. These
are marks given by 21 delegates at a workshop for new lecturers.
Mark: 49 50 51 52 53 54 55 56 57 58
Number of Markers: 1 3 3 1 2 2 1 3 3 2
This ranges from 82% to 97%. In practical terms this wide variation would not be too much of a problem,
since they are all rst class marks. And bearing in mind that the markers had not set or seen the paper
before the variations in the results are perhaps not surprising. But for students near the borderline of
First/Upper Second such a variation could have signicant implications. As it happened the variation for
two other students marked in the exercise was much less - probably because the highly variable Question
5 considered above was effectively taken out for these students. The average of the mark for Student A is
around 54, i.e. 90%. The actual moderated mark eventually given to this student, in the actual exam, was
56 (93%). The difference can be explained by the fact that the actual marker had a clearer idea of what the
students had been taught and what was expected.
The rst lesson to learn from this exercise is that even on a relatively straightforward script marking is
simply not a rigorous enough activity to be anguishing over the odd percentage or two. In particular,
as noted earlier, no one would seriously quibble over turning a mark of 39 into a pass mark of 40%.
Some people, for example, used
1
2
a mark (out of 6) in their marking and again as mentioned earlier it is
doubtful whether it is worth worrying about such detail.
The marks given by different markers for the individual questions in this exercise also illustrate a number
of important points. Questions 1, 2, 4, 6-10 all had virtually perfect solutions and nearly all markers
gave full marks for these. There were some rather ridiculous outlier marks for these questions - for
example a 2 for Question 2, a 3 for Question 10, but these are problems for the marker, not the student.
The results overall illustrate that marking is easy and highly consistent, as you would expect, when the
solution is virtually perfect and gets full marks. And of course the same applies at the other extreme - no
solution means no marks! Section A is designed to have the bulk of such questions, so that the student
who works reasonably but has limited ability can pick up some basic credit. This is appropriate for
engineering students, but not necessarily for mathematics single honours students. Such straightforward
questions are essentially single step questions, requiring low level Knowledge type skills. However,
Questions 3 and 5, studied in detail above, each have a number of stages and do require some higher
order thinking skills, such as putting ideas together, analysing complex situations, or applying quite
complicated formulae. It is such questions that students nd more difcult to do, and lecturers more
difcult to mark. Often the student only manages part of the question, and it is difcult to decide how
many marks to award. This of course is where the link with the learning objectives is most important and
most difcult, and where a good but exible marking scheme is important. It is in such places that we
get wider variations in marks between markers, and we have to examine such cases more carefully.
In the same workshop some delegates marked part B questions, which carry 20 marks each and are
more substantial. The issues discussed above are even starker in such questions, which are longer and
more involved. Most students will only complete parts of these. For student A, for example, the marks
awarded by the delegates ranged as follows:
Question 11 4-12
Question 12 8-18
Question 15 15-20
Clearly, irrespective of the low resolution of the marking scheme, the ranges evident for Questions 11 and
12 could generate much debate along the lines of the earlier discussions.
178
Of course, one cannot discuss every question and every script in the detail we have here when it comes
to real marking. One has to use common sense. Usually, the quality of ones marking is only revealed
to scrutiny after the event, when all marks are in and one can make comparisons between different sub-
jects and different examiners. Only then are any extreme results detected, and various departmental
mechanisms such as scaling may be brought into action to correct the situation. In order to reduce in-
consistencies in marking one may be tempted of course to use examination questions that are easily
marked, because they are relatively straightforward and examine only low order cognitive skills (such
as Questions 1, 2, 4, 6-10). This then means that students are neither required nor encouraged to develop
higher order skills, or to learn the material in any depth, storing up trouble for subsequent learning.
Fromsuch exercises as that described above one picks up a fewminor points worth noting. When writing
marks on the script it is better to them as a proportion rather than as an absolute value e.g. 5/8 in
Question 6 rather than just 5. This helps the moderator or external examiner. It is only worth writing
comments on the script if they are going to be useful to someone (See Feedback later). Now, in the UK
(but not for example in some parts of the US) students dont normally get to see their marked examination
scripts. So any comments on the students scripts are really a waste of time, unless they are intended to
aid the moderator or external examiner. Usually on a script there is a proper place for marks to be entered
- a column on the right for example. In real marking always try to mark in a different colour to that used
by the student, for obvious reasons.
Question 4 highlighted an interesting point that some delegates picked up - namely, what is the simplest
form, is 5 lnx simpler than lnx
5
? In fact of course we would not really quibble, and such questions as
Simplify ... can sometimes be problematic. Either would be acceptable. The key point is whether the
student demonstrates knowledge and understanding of the key rules of exponentials and logarithms and
student A, for example, clearly does. But in general, we need to be sure that questions such as Simplify
... lead to an unambiguous answer. Another similar situation arises when the result of a calculation
asked for (For example differentiating a function) is not reduced to simplest form. If this is not asked for
and the student has performed the required calculation correctly then they deserve full marks.
We now look at each script in turn highlighting the sorts of thoughts one might go through in marking
such papers. Here and elsewhere I explain my own rationale for the marks awarded not to exemplify
good marking, but to illustrate the many factors one has to take into account and the continual need for
fast but fair decisions.
Student A
The Section A questions are short and easy to mark and can be all done at once. This student is a clear
pass and is pretty easy to mark. Here one simply skims through and checks the answers are OK and only
homes in when they are not, as in Questions 3 and 5. Then one makes a very quick assessment of what
marks to deduct. When you are marking a couple of hundred scripts you are not going to mess about
when the outcome is so clear!
We take more care over the Section B questions, but can still run through them quite quickly as this
student gives clear and concise solutions. In Question 11 for example one forms an overall impression
as there isnt a direct match to the mark scheme. The student has uffed the factorisation and so has
struggled with the subsequent partial fractions. As noted earlier this is a bad question in this respect,
because errors in the rst part of the solution render the rest far more difcult than intended. In Question
12 the binomial expansion is OK, but there is a confusion in the choice of the numbers for the approxi-
mation. Rather than break down the marks I simply make an overall judgement, and 16/20 seems fair.
No problem with Question 15 except for I
3
, where the substitution is correct, but has not been imple-
mented properly - less than half marks for this part. The eventual overall mark of 78% seems reasonable,
particularly as this is an engineering paper.
179
Student B
This student is also very easy to mark as they are a clear fail as is evident from a quick skim through the
paper. Section A was quite easy to mark. Most solutions were either perfectly correct or not attempted at
all. The student got the bulk of their marks here, offering only token attempts at part B questions. This
student is a clear cut case, they simply have not developed the skills to proceed and the eventual mark
achieved was 28%.
Student C
A quick glance through the paper suggests this student might pass, but they are not strong. Here we
have to look more carefully at even the Section A questions. In Question 8 the one mark reects perhaps
over generous recognition that the student has a vague notion of the product and chain rules, but cannot
implement them. In Question 9 the student at least knows the log integral. Again in Question 10, the
student has a vague idea of the integration by parts and knows what a denite integral is. In all of these
there is some benet of the doubt in order to give at least some recognition, or to avoid total zeros. So in
the section B questions I can afford to be more rigorous.
In Question 13 the student has simply shown that they can remember the key trig identities (No formula
sheets remember), but they dont really know how to use them in this question, so they get a low mark.
In Question 14 they know enough to start off with integration by parts, but dont get very far, so again a
low mark. In Question 15 the student has done I
1
ne. But I
2
and I
3
have been approached in the wrong
way but show some merit, and a mark of 10/20 seems reasonable here for this type of student. Going
quickly through the solutions again, it is clear that this is very border-line. Is this a case for a rethink? If I
docked a couple more marks this student might well end up spending the summer revising for the resit.
There seems no point in this, and their personal tutor will almost certainly have a word with them, so we
(The moderator and I) let it go. No doubt the student will be relieved. Many non-specialist students are
content to simply scrape through their mathematics in this way, and have no ambitions for good marks.
5.6.7 An Example in Marking a Full Paper - Abstract Algebra
Exercise
Mark the full scripts by students A, B, C given in Appendix 8.
We will now look at the marking of the Abstract Algebra paper given in Appendix 5, for the attempts
given in Appendix 8. As stressed earlier the discussion here is not meant to be denitive, you might take
different views in places, it is just simply to show the sorts of things one thinks about during marking.
Student A
It is clear from a quick glance at the paper that this student has not done very well and is a clear fail. So
the rst thing to note is that we would not invest too much time anguishing over this script. The student
will almost certainly have to resit the paper and a rough mark will sufce. Note that the student has
struck out some of their work. Ofcially you would be entitled, indeed possibly obliged, to ignore this
attempt altogether. However, I always look at such things. If the student has crossed out worthy work
that would have attracted useful marks I may turn a blind eye and take it into account in some way,
particularly if the student might otherwise fail. In this case the student returns to the question, so the
issue doesnt arise.
In fact, this is a very easy paper to mark. Clearly there is nothing of substance in questions 2 and 3, and
the marks of one for the attempt in each case are token and pretty irrelevant. The solution to Question
1 has some substance to it. The student has the right idea and accuracy for the rst part and there is no
harm in awarding the full marks here. For the second part the student appreciates the idea of proving
180
equality of two sets by showing each is contained within the other, but the details are garbled and not
convincing. 3/5 is a generous compromise here and makes no difference to the eventual outcome of a
fail. In an effort to nd marks I initially awarded one mark for the last part, but on closer inspection it
is probably too generous, the student is writing nonsense. However, it is not worth thinking about. The
upshot is that this student, whatever we could do for them has failed. This student achieved 21% on the
paper.
Student B
That this script was border-line was clear from a quick skim through, so one would take some care on
this. In Question 1 the rst simple part has been presented perfectly and so gets full marks. For the rest
of the question the student has made the common error of continuing the example rather than doing a
general proof. In such cases I try to look for some evidence that the student has some idea. This student
appreciates the idea of disjoint sets, but that is about all, They have confused order of a set with the set
itself, and it is arguable whether 2 or even one mark is generous here.
In Question 5 the student has in fact reproduced verbatim the summary of the encryption/decryption
process provided in the class notes - so at least they have a good memory! But they have produced little
evidence that they have any deep understanding of the RSA system, so a bare pass seems to be in order
for this part. I have been generous elsewhere. Such questions are in fact very difcult to mark and are as
subjective as mathematics gets. If there had been any danger of this student failing I would perhaps have
found one or two more marks here. But in the second part of the question the student has demonstrated
mastery in all respects and reassures me they actually know what they are doing, so I am relaxed about
giving full marks here. The nal outcome of 14/20 is still a rst class mark, so I am pretty happy with
it. Incidentally, this question illustrates well the tendency of most students to prefer the routine, recipe-
based aspects of questions to the more sophisticated explanation and deeper understanding.
In Question 2 the student has stated the denition of an equivalence relation, but they have not dened
their terms and from the ensuing example it is clear that they dont really understand the idea. It is
difcult to allocate marks in such cases, and to decide exactly what the student does know. Here I tend
to try and allocate marks according to the mark scheme and see what results, in this case 5/20. I then
scan though the solution as a whole and try to see if the nal mark is a reasonable reection of what the
student really seems to know. In this case I was satised it was not far out. Finally, I look at the total, feel
relieved I have a pass with a clear conscience, and quickly move on to the next script. I know this student
is weak, but overall I am condent that they have enough grasp of the ideas to cope with future needs.
The nal mark for this student was 45%, and they were probably told by their personal tutor that while
this was a pass, they were very weak in this subject and should do some revision.
Student C
Ah, no problem! In Question 1 the easy numerical part is faultless, so full marks. The general proof is
mixed up with the argument for the disjointness of the sets and it is difcult to separate out what the
student actually knows, so I combined the marks and gave a near pass mark for these two parts. The
nal proof is by no means clear or complete, but the student has given enough of the gist, and in their
own words, that I see 2/4 as reasonable. And they have given a relevant example. Overall, I am happy
with the mark 13/20 and as this student is a clear pass I am not going to waste much time on it.
In Question 2 the denition is good enough for full marks. The application to mod 6 is faultless, so
full marks there. The equivalence classes part is awkward. The student seems to know what they are
doing, and has listed the equivalence classes, but has not given any formal denition of them. I settle for
4/6. Again, in the nal part of the question the student has argued their way through the proof with
some conviction, even though they havent expressed it in precise algebraic terms. One gets a sense that
they understand what is going on, so 2/4 seems reasonable. This student is right on top of equivalence
relations so I am comfortable with an overall mark of 16/20. Ditto for Question 3, an excellent solution,
and no hesitation for the high mark. The overall mark of 80% seems appropriate.
181
We emphasize again that the above examples are not intended to display good practice that you should
adopt. Other equally capable teachers may justiably take different approaches to the same material. The
point of the examples is to illustrate the sorts of things that we may have to consider in our marking, the
plethora of quick decisions needed, the judgements and compromises, the need for practical common
sense and so on.
Exercise - marking to a mark scheme
The examples given here are only a rather articial start for you. With the cooperation of a colleague
try to work though some past scripts in a similar way, and learn as much as you can from the
rationale given for marks awarded. As here, you are not expected to go along with everything your
colleague says.
5.7 Coursework and Feedback to Students
5.7.1 Coursework
For most examinations the only feedback students receive is their overall mark/grade. Most departments
now use a certain proportion of continuous assessment in the form of coursework. This provides an op-
portunity to provide helpful feedback for the students, and for them to accumulate credit by constant
working throughout the year. However, coursework is controversial, mainly because it has low validity
and reliability as a method of assessment. Coursework submitted by a given student cannot be guaran-
teed to represent exactly what they know, and will not necessarily produce similar results under different
circumstances - we cant control the conditions of assessment. Most departments are prepared to accept
this in exchange for the promotion of learning and the extra opportunities for students to demonstrate
their attainment. However, fears of plagiarism and collusion (See below) do tend to limit the proportion
of coursework to the examination. Of course it is possible to set plagiarism proof coursework of the
mini-project type, where the students are unlikely to be able to nd such material elsewhere, such as
on the web. But this wont offset collusion between students, or the possibility of them getting help
from friends and family. And it is very labour intensive for the lecturer. Another ploy is to set differ-
ent questions for different students, but this is again enormously labour intensive for the marker, also
it cannot be guaranteed that every student is assessed equally, and of course it still does not offset help
from others. And if they learn from it, what is wrong with help from others anyway? Some people use
class tests instead of coursework, but these are simply mini-examinations (And sometimes not under fair
examination conditions) and defeat the object of helping the student to learn in the process of doing the
assessment. Quite simply, in whatever form it takes, unsupervised coursework is neither valid nor reli-
able, but for many students it is valuable to learning and for this reason is useful, if only in small doses!
Also, whatever its form coursework is always labour intensive for the lecturer.
The amount of coursework to set is a difcult issue and you should seek advice on this from colleagues.
Also, particularly in mathematics, we need to be careful about the sorts of assessment tasks we set for
coursework. Questions that are routine in nature (Mostly K) should he avoided, since these are easily
done or copied in the absence of examination conditions. Essentially coursework questions should test
higher order thinking skills (High I and/or T), which makes it even more difcult to estimate how much
to set. It also makes the marking more difcult and less valid and reliable. On the other hand the level of
skill required should not be excessive - this is not an Olympiad environment. Perhaps the best strategy is
to build up such a rapport with your students that they will let you know very quickly what they think of
the coursework load. Combining this with early coursework results should give you an idea of whether
you have the balance right. Some lecturers set coursework that uses past examination questions. This
182
gives students ideas of what to expect in the examinations, and it also means that the actual examination
questions can be more challenging.
Some lecturers set coursework frequently, others at specic points such as mid-term or in the run-up to
the examination. There are arguments on both sides. Frequent coursework assignments have to be less
substantial to keep workload reasonable, and are therefore more open to plagiarism. A small number of
substantial assignments can lead to workload bunching. Also, coursework should not be a replacement
for regular independent self-disciplined learning on the part of the student. One of the problems with
excessive coursework is that the students can end up only doing work on which they are assessed, and
so get little practice in self-motivated and self-directed learning.
Technically, marking coursework is the same as described above for marking examinations, except for
three important differences:
you are likely to have more time to deliberate and comment on the students work
you are likely to need the extra time because, being coursework the students will have done much
more work on the questions, giving fuller solutions for you to work through, and also calling for
more feedback
you can save some time by skim marking the many copies of the same solution presented by differ-
ent students working together!
5.7.2 Plagiarism
This, referred to above, is a hot topic nowadays - but somewhat overcooked. There are far greater prob-
lems in mathematics assessment, but we ought to say something about it here, in the context of math-
ematics. Obviously there should be little possibility of cheating in supervised examinations, and this is
dealt with, when detected, by standard institutional disciplinary measures. Your only involvement here is
likely to be in reporting and witnessing to the event. Much more problematic of course is the increasingly
widespread use of unsupervised coursework. While this is perhaps an issue in essay-based assessment as
say in the arts or social sciences, where plagiarism is almost impossible to conceal, it is really a non-issue
in subjects like mathematics, where it is almost impossible to detect, investigate and prove. Particularly
in techniques-based topics two completely independent solutions to the same problem can look virtually
identical. The only real give away is when a number of scripts come in together with identical silly mis-
takes, and even then, unless someone coughs it is difcult to assign responsibilities. And, despite all the
hype about plagiarism, does it really matter too much? Quite frankly, if your assessment regime is such
that signicant gain can be made through cheating and plagiarism, then it is not really reliable or valid
assessment. For example some lecturers set the same coursework each year, issuing the same worked
solutions. Obviously the students will get to know this and soon start communicating! We should en-
courage students to work together, and coursework is a good vehicle for that. Just encourage the students
to present their own work in the end. If one student obtains a few percent by verbatim copying off a fel-
low student then they will be found out in the exam. The only way to make such coursework assessment
harmless in this respect is to keep it at a low proportion of the overall assessment. But if its too low, some
students wont bother doing it (but then thats their funeral!). This problem is well appreciated at school
level where coursework at GCSE has now been reduced for this very reason.
On the practical matter of dealing with students copying work, note that this has to be distinguished
from working together as opposed to out and out verbatim collusion or plagiarism. The latter is almost
certainly above your pay grade, and if it is an issue then you should simply report it to the responsible
authority. You might be kind and give the students an initial warning. For non-assessed work where
there is no question of gain involved about the best you can do is simply ask them what do they hope
183
to gain from such behaviour. Suppose only one student in the class understood a topic and everyone
else copied from them. Then the lecturer sees 100% understanding and carries blissfully on their way
believing that topic has been covered adequately! So he puts it in the examination .... You will reduce the
chance of copying not by excessive vigilance of what the students are doing, but by creating an open and
secure atmosphere in which students are not afraid to expose their weaknesses and problems. Ask them,
do they want to get help from an unpaid, unqualied fellow student, or from a paid, ofcial, trained
teacher who has the experience to sort out their problems?
5.7.3 Quality of Feedback to Students
One of the prime purposes of coursework is for students to benet from the feedback you provide on the
work they submit. This is a key tool for helping students to learn, and it is important that feedback on
coursework is returned as promptly as possible, while the work is still fresh in their minds. Tardiness in
returning feedback on coursework was one of the signicant criticisms of MSOR providers found in the
Subject Review of 1998-2000 [8]. It is also an area, along with assessment generally, that often receives
lowest satisfaction rating in the UKs National Student Satisfaction Survey (THES Sept 9, 2005).
Principle 2 is particularly important in providing feedback, especially the human aspect. This is prob-
ably the most sensitive area of teaching in terms of human interaction. In all other situations you have
sufcient distance, physically and ofcially, between the students and yourself to control interactions be-
tween you. But in Coursework you are directly involved with the students and you are interacting on
matters that directly affect their future. Some students take it very seriously and will want their marks
explained and justied in detail. Some regard your mark as nothing more than an initial offer on which
negotiations can commence - usually in the upwards direction! It doesnt help to stand on ceremony in
such situations or to pull rank. If you cannot easily justify your marks you perhaps ought to consider
whether the student is right. In my experience, if you are considerate to your students, respect them, and
are honest and rigorous with yourself then such things are invariably resolved amicably.
There is a wide range of feedback mechanisms possible, including full specimen solutions and notes,
annotations on students scripts, oral feedback in class, classroom demonstrations, personal summary
sheets, feedback forms, group sheets identifying common problems, posting on notice boards or the
web, workbooks, follow-up teaching sessions, effective open-door policy, individual discussions with
students, reworked examinations, etc. The important thing is to make every effort to ensure that all
students receive the feedback in one form or another. See [15], [53], [63], [72] for advice on providing
feedback. Below we summarise some pointers on providing oral and written feedback.
Oral feedback:
start by asking the student what they think rst
be sensitive to the students feelings when giving feedback
describe issues that need to be addressed without being judgemental about the students work
mix positive and negative feedback in a constructive way
let the students respond with their own ideas
focus on specic issues with clear objectives for improvement
indicate actions that can be taken to improve
check the students have really understood the feedback
184
be realistic and tolerant in your expectations and assumptions about the student
only feedback with an amount of information the student can use, and structure this as efciently
as possible, organising recommendations most helpfully
listen carefully when a student is responding to your feedback, be sure you understand them
agree on actions to be taken, and follow these up.
Feedback on written work:
make constructive comments rather than just noting errors
suggest where they need to concentrate their efforts, with guidelines for doing this
balance encouragement and criticism, ending on an optimistic note
focus on a few points that will result in greatest improvement rather than highlighting every little
error
when appropriate relate to marking criteria, pinpointing where the student needs to improve
produce a summary of common errors to issue to all students, with constructive comments for
dealing with them.
Follow up with oral feedback when possible
tailor your feedback to the mark given and try to indicate where and why marks have been lost
conclude with an overview of the feedback.
Another point about feedback is that there is such a thing as being too helpful to students. It is often
argued for example that one should not issue worked solutions to coursework, because they le them
away and never look at them again (until too late!). Instead it is advised that one picks up any error in
their script and suggests ways forward fromthere, addressing the students particular difculty. Nowthis
seems to be very commendable and very helpful to the student, but apart from being time consuming,
especially with large classes, it is also not quite so clear cut as might appear at rst sight. Firstly, if a
student loses marks then that should immediately tell then they have a problem to sort out, and if their
response to this is to le away the solution then that is on their own heads - we cannot police every
students learning methods. They could come to you to sort out the problem, but they really need to
develop the skills of criticizing their own work, comparing their solution with the specimen solution and
identifying where and why they have gone wrong. These are important skills for students to acquire to
develop as independent learners - crudely, if you keep spoon-feeding them, they will never learn to feed
themselves!
Perhaps the best advice is to decide an upper limit on the time you can commit and within that identify
the really essential messages you must get across to the student and then do this in the most efcient way.
Dont bother too much with details - perhaps simply put a circle round such mistakes and let the student
sort them out themselves. If there is a really important point that does need careful and full explanation,
then take some care over this - maybe it reects a misunderstanding that you could have anticipated and
avoided in your lectures or tutorials. Perhaps go through it in class next time. Some people recommend
making comments even when a student has not made an error - simply to extend on what they have done
or give them a different viewpoint. In practice you will probably nd your time adequately employed
just covering those issues where students have made mistakes. Unless a student specically asks for
185
further enlightenment, it is unlikely that they will welcome it (do you?). In general, you shouldnt take
a long time over anything in teaching - time should be used efciently for all concerned. In fact, there
is some research that suggests that when students get their coursework back they only look at the marks
achieved and dont read the feedback. Possibly they have some preconceived idea of what they should
get and if the mark accords with that then they are simply think fair enough and get on with other
more immediate things (like the next coursework!). So some argue that you shouldnt give marks for
coursework. The problem is, of course, that if you dont give marks for work then most students just
wont do it!
Example
In Appendix 9 we reproduce the attempts of students A, B, C on Question 3 in the engineering
mathematics paper of Appendix 1 - the binomial theorem question. If we pretend these are
coursework questions on which we have to provide feedback then we might adopt a strategy
as follows.
All students have got the binomial theorem mostly right, but there are some minor glitches.
So in general feedback to all we might rst congratulate them on having the basic idea, but
then re-emphasize the need to get the details of the theorem exactly right - remind them they
wont have a formula sheet in the examination. Two of them have forgotten the three so in
general feedback note that the typical term should be 2
nr
(3x)
r
.
Having summarized the common errors, there is not much to say on each individual script.
For student A you might say nothing at all, or simply You missed out the 3. For student B,
something like Your coefcients are out of synch:
5
1!
2
4
(3x) not
5(5 1)
1!
2
4
(3x)
etc. For student C we might simply scribble on the script (3x)
r
not 3x
r
and na
n1
b not
a
n1
b. These individual comments, along with issue of the full solutions with comments on
common errors should be more than enough for these students to sort out their problems.
Exercise - feedback
Assuming the work of students A, B, C in Appendices 4 and 8 are formative coursework, provide
feedback materials and individual comments in the most educationally effective and efcient way.
5.8 Evaluating your Teaching
As well as assessment of the students we also have to assess or evaluate our own teaching. Nowadays
someone else will anyway, through such things as student questionnaires. Note that in the US for exam-
ple evaluation is usually used for what we call assessment of students, so be aware of this if reading
American books on teaching. Also, note that we are not talking here about formal mechanisms of eval-
uating your teaching that will be provided by your department and institution. You will get more than
enough literature and information on this from your university. Here we are making some brief com-
ments about our own reective evaluation of our teaching - what all good academics have always done,
as a matter of course, long before administrators and managers turned their attention to it.
You may think that evaluation of your module may come late in the timetable, when you have deliv-
ered most of it. This is not so however - you should continually be evaluating what you do, as you go
along, and you can design evaluation in from the start. Remember that every aspect of your teaching
186
needs evaluating - your planning and design of the programme, preparation of materials, teaching and
learning activity, student assessment - even the evaluation itself can be evaluated! That is part of be-
ing a professional. For example a colleague can advise on your draft design for the curriculum, draft
examination papers, and so on.
Whatever form of self-evaluation you use, you will have to integrate it with your institutional mecha-
nisms, you dont want to overload the students, or yourself, by repeating what is done by the institution
or department. So what we say here should be taken in the context of your institutional mechanisms.
By far the best and most efcient evaluative tool you can have is good rapport, excellent communications
and mutual respect with the students. Aim for a situation where they will tell you if they are dissatised.
But student satisfaction is not the last word in good teaching. Contrary to popular opinion, most students
prefer a good education to simple satisfaction, and there are many more inputs to evaluation besides stu-
dent feedback. Thus, input from trusted experienced colleagues, knowledge of the educational literature,
expert advice, outcomes of assessment can all be triangulated to give an overall picture of the quality and
effectiveness of your teaching.
If, for example, you have institutional student feedback forms, then dont duplicate what they survey,
but consider if you need any additional information. Often, because such forms are machine-processed
they do not have facility for open-ended feedback from the students. If you would like this then just ask
them to jot down at some point their likes and dislikes. Often, such things can be found simply by a
show of hands in the class. Always look at the big picture. You will not please everyone, and you will
sometimes get some discouraging, even insulting, responses, but focus on the majority concerns and try
to learn from these, dont take them personally. Particularly if you are giving a course for the rst time,
try to get some feedback early on in the course, to help steer your development. And let students know if
they have benetted from previous student feedback: I tried an example like this last year, but it didnt
go down very well, so I have changed it a bit - let me know what you think.
So far as input from experts and the literature is concerned, the UK HE Mathematics community is, at
the time of writing well served through the MSOR Network of the Higher Education Academy and its
nation-wide contacts. Through this any lecturer is able to access good and innovative practice across the
sector - what your departmental colleagues cant help you with, the Network knows someone who can!
The results of assessment are also a very good indicator of howyour teaching is going - if you are rigorous
and honest enough with yourself about it. You can learn a great deal from how your students have done.
At the supercial level, you need to ensure that your results are reasonable. Your examination board
Chair or Examinations Ofcer will soon let you know if they are not! But this is not the end of it, and
dont sit back and relax just because the nal data doesnt raise any eyebrows. Study the scripts carefully
and see if there are any common misconceptions or errors that you can anticipate next time you teach
the course. If no one has attempted a particular question then that might be saying something about
the teaching of that particular topic. Examples include practical versus theory questions in numerical
methods, calculus versus matrix methods in engineering, etc. Of course some assessment is designed
specically to tell you particular things about the students - diagnostic testing for example, on which
there is plenty of literature (http:/mathstore.ac.uk/).
A simple reminder of how to obtain and use evaluation of any aspect of your teaching is the ve As [28].
These reminds us that whenever we evaluate our teaching this is only the rst step towards developing
to improve it, possibly with assistance from others.
Assess the particular area of interest, say clarity of presentation in a lecture. Remember assessment
always needs to be as valid and reliable as possible, so you might use a number of methods such as
a student feedback form, observation of your teaching by a colleague, possibly even videoing your
presentation.
187
Analyse the results of the assessment as candidly as possible and identify where improvements can
be made. For example perhaps English is not your native language and this shows, or perhaps you
speech is monotone and lacks enthusiasm.
Advice from others should be sought, or you may consult the literature, in order to decide on the
most practical way to make improvements. For example you may, after advice, decide that you
need some voice projection training.
Assistance is then required from some quarter. For example your staff development department
may be able to arrange the voice training, and then you may want to practice in front of a colleague.
Action is then required by you to implement the improvements, which may include letting the
students know that you have taken their views on board, and would value their comments on the
improvements.
Now, as with most such things, the above is basically common sense and once it is absorbed requires no
conscious deliberation. However, particularly if your are doing a large scale evaluation, say of a complete
module, then it may help to remind you of the boxes that need ticking.
Exercise
Choose some area of your teaching that you feel needs improvement and carry out an evaluation
triangulating a number of methods. Talk with colleagues if necessary and make improvements with a
clear rationale. Implement these and reassess the impact of your changes - and so on. This exercise is
really nothing more than an example of what being a professional is all about!
188