Adaptive Remediation for Novice Programmers through
Personalized Prescriptive Quizzes
Reza Soltanpoor Charles Thevathayan
RMIT University RMIT University
Melbourne, Victoria, Australia Melbourne, Victoria, Australia
reza.soltanpoor@rmit.edu.au charles.thevathayan@rmit.edu.au
ABSTRACT
Learning to program is a cognitively demanding activity. Students
need to combine mental models of various concepts and constructs
to solve problems. Many students new to IT and CS programs have
little or no prior experience with abstract reasoning and problem-
solving. Instructors attempt to present the core concepts early to
allow adequate time for students to complete their programming
assignments. However, misconceptions of basic concepts formed in
the early stages often get propagated blocking any further progress.
Such students often begin to form poor opinions about their capa-
bility leading to low self-esteem and performance.
This paper proposes a framework to help individual students
to overcome their misconceptions through personalized prescrip-
tive quizzes. These quizzes are generated by combining the rich
meta-data captured by each quiz question with analysis of past
responses to class quizzes. The personalized prescriptive quizzes
generated helped to improve student engagement and performance
substantially. Over 91% of the students surveyed indicated that per-
sonalized quizzes helped them to clarify their own misconceptions
and made them more confident of their progress. Students using
the prescriptive quizzes performed significantly better than others
in subsequent class assessments and the final exam.
CCS CONCEPTS
• Computer systems organization → Embedded systems; Re-
dundancy; Robotics; • Networks → Network reliability;
KEYWORDS
Personalized Learning, Formative Assessment, Active Learning,
Learning Analytics, Prescriptive Analytics
ACM Reference Format:
Reza Soltanpoor, Charles Thevathayan, and Daryl D’Souza. 2018. Adaptive
Remediation for Novice Programmers through Personalized Prescriptive
Quizzes. In Proceedings of 23rd Annual ACM Conference on Innovation and
Technology in Computer Science Education (ITiCSE’18). ACM, New York, NY,
USA, 6 pages. https://doi.org/10.1145/3197091.3197097
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus
© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-5707-4/18/07. . . $15.00
https://doi . org/10 . 1145/3197091 . 3197097
51
Daryl D’Souza
RMIT University
Melbourne, Victoria, Australia
daryl.dsouza@rmit.edu.au
1 INTRODUCTION
Problems faced by novice programmers stem from misconceptions
of basic concepts, which varies from student to student [19]. Such
misconceptions can be easily (and contextually) identified by min-
ing the collected quiz answers, such as problems in understanding
nested loops [28]. In a similar work, misconceptions with variables
and boolean operators became apparent only when loops were
used [13]. Quizzes have been used effectively as instruments for
correcting misconceptions by setting quiz choices and their feed-
back based on knowledge of how such concepts are commonly
misunderstood [26]. Offering weekly quizzes from the beginning
of a semester enables teaching staff to monitor student progress
and to identify their misconceptions early [29]. However, the lack
of student-specific data in the initial stages prevents creation of
personalized quizzes.
Though quiz-enhanced learning improves performance in exams,
the extent of improvement has been shown to vary depending on
frequency and timing; greatest gains result from review quizzes at
the end of the semester [22]. The type of quizzes should also reflect
the motivational level and the background of students. Ungraded
quizzes, void of any anxiety about marks, have been shown to be
more effective in fostering deep learning, while graded tests help
to ensure all students make steady progress [14]. Ungraded quizzes
are more likely to benefit intrinsically motivated students, thus
needing other forms of intervention to help stragglers. Stragglers
who fare poorly in early programming assignments and class tests
often develop a poor self-esteem (belief about their personal capabil-
ities) and become even more withdrawn [4]. Small remedial classes
have been used to provide more personal attention and to boost
confidence through positive and encouraging feedback; however,
the mandated use of large classes has made such an option unvi-
able. A more cost-effective strategy is to provide feedback through
personalized quizzes, which are gradually made more difficult, a
key strategy for building self-esteem [4].
Intelligent tutoring systems (ITS) provide individualized instruc-
tions tailored for specific students, the key feature being the support
of cognitive diagnoses and adaptive remediation. They can be made
to supplement traditional teaching and have been shown to outper-
form traditional teaching for specific cohorts [20]. ITS effectiveness
stems from immediate response-specific feedback and increased
opportunity for interaction. Such features would allow student spe-
cific pathways to be created considering what students are expected
to learn, the strategies for tutoring, and the current knowledge of
the students.
In this paper we bring together the foregoing ideas using adap-
tive remediation, based on the notion of a Personalized Prescriptive
Quiz (PPQ). PPQs are compiled using learning analytics (LA) data
ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus
from past assessments. LA has emerged in recent years, combin-
ing aspects of data collection, insight extraction and prediction. It
is primarily concerned with the measurement, collection, analy-
sis and reporting of data about learners and their contexts with
the purpose of optimizing learning outcomes [1, 31]. Several ap-
proaches combining the benefits of LA and ITS have been proposed
to provide the best learning pathway and adaptive feedback to
students. By combining aspects of learning analytics to ITS, our
proposed approach allows learning difficulties to be discovered
from the data collected. Effectiveness of such systems in helping
students to learn can be evaluated by using “pre” and “post” tests
with standardized language-neutral misconceptions known as con-
cept inventories [10]. Moreover, our system logging all student ac-
tions can play a key role in formulating conceptual change theories
designed to study the acquisition of knowledge and the evolution
of misconceptions [24].
We used PPQs in this study to address the following research
questions: Can prescriptive quizzes be effectively used to:
(1) improve learning outcomes of diverse student cohorts?
(2) develop higher-level cognitive skills needed for problem
solving?
(3) promote engagement and self-esteem?
The PPQ approach allows us to capture common misconceptions
at single and multiple concept levels, as many misconceptions oc-
cur when two or more concepts are combined. In the anonymous
student feedback, 95% of the 120 respondents wanted the adaptive
remediation approach to be extended to other cognitively demand-
ing courses. The final class test results at the end of the semester
revealed that students using the PPQ showed significantly greater
improvement over others.
2 RELATED WORK
Introductory programming courses have been experiencing high
failure and attrition rates (up to 50%) partly reflecting the diversity
and background of incoming students [3, 5, 7, 11, 17]. An earlier
multi-institutional study revealed significant performance differ-
ences, highlighting the need to customize teaching methods to
specific student cohorts [21]. Even simple concepts introduced in
the first week cause some students to form nonviable mental mod-
els [16, 25]. Incorrect mental models of basic constructs, when
not corrected early, get propagated to other concepts, leading to
frustration and poor performance in summative assessments [30].
Providing early qualitative feedback through formative assessments
can rectify misconceptions and improve engagement [6].
Prescriptive analytics, a branch of analytics, uses data to help
mitigate risks and remain competitive by guiding management
with adaptive and automated courses of actions in a timely man-
ner [32]. The use of LA in higher education, incorporating key
prescriptive functionalities, is fairly recent, with much of the ana-
lytics based on logs of learning management systems (LMS) [31].
Several studies have been conducted to identify students’ patterns
of learning [8, 15], using LA techniques to capture, for example,
idea development and flow within discourse, and understanding of
learning pathways [6, 18]. Unlike summative assessments, forma-
tive assessments should fine-tune instructions for each individual
students by personalizing feedback in areas of attainment, areas in
52
Reza Soltanpoor, Charles Thevathayan, and Daryl D’Souza
need of improvement, and further guidance for improvement [33].
Automating individualized feedback therefore requires concept-
based data gathering and adaptive learning techniques to tailor
separate pathways for each individual student [27]. For LA to be
effective, the frameworks for gathering data should incorporate
ways of measuring depths of learning outcomes using established
taxonomies for cognitive processing [12]. ITS and LA can be ju-
diciously combined to improve learning outcomes. ITS software
performs tutoring functions, constructs robust multidimensional
student models, and adapts a set of tutoring functions relevant to
each student using the analysis done by LA modules [20].
Despite the surge in LA use, no current framework explicitly
captures the relationships between concepts, which is necessary
to identify the root causes for conceptual difficulties. Moreover, a
flexible framework that allows tasks with composite constructs is
needed, as novices mostly experience problems when combining
constructs [1, 2]. Finally, catering to increasingly diverse student
cohorts requires techniques that target different student groups by
incorporating metrics such as discrimination indexes [9]. The PPQ
approach is intended to address these gaps. We elaborate further in
the next section.
3 DESIGN OF OUR FRAMEWORK AND PPQS
This section describes the rationale for our design decisions and
our methodology used in evaluation before describing the algo-
rithm used in PPQ. The PPQ approach was applied in the course
Introduction to Programming, during Semester 1, 2017, with 274
undergraduate computer science students. The rationale behind
selecting this course was that (1) technical courses are acknowl-
edged to be among the most challenging for first years [34, 35]; (2)
they are fundamental to computer science majors in terms of core
concepts taught, and the skills developed [11, 23]; and (3) they have
continuously been experiencing high dropout and failure rates,
which corroborates the view that learning to program is tough for
novice learners [5, 6, 35].
Responses to class quizzes used in the first 8 weeks allowed stu-
dent specific misconceptions to be captured. Misconceptions were
identified based on multiple choice and multiple selection questions
with corresponding distractors (designing the incorrect options per
question in a way that distract students with limited knowledge
[26]). Questions were also designed to cover multiple concepts as
most misconceptions are known to occur when two or more con-
cepts are combined. Dependencies between concepts are explicitly
captured (such as loops depends on boolean expressions), permit-
ting students to trace the root causes for their misconceptions. Each
misconception can also be linked to other known misconceptions
used as distractors. Quiz questions covered different levels needed
in problem solving and programming. For example students can
be asked for definitions (knowledge), expected output (compre-
hension), write expressions (application), identify logical errors
(analysis), or to rearrange the order (synthesis). To measure the
effectiveness of our framework in improving problem solving skills
the final exam was designed with three distinct sections, including
a section on development of an object-oriented program based on
specific requirements.
Adaptive Remediation for Novice Programmers through Personalized Prescriptive Quizzes ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus

To encourage engagement and feedback, both standard and pre- • (uuqi ): set of correctly answered questions covering more
scriptive quizzes allowed multiple times. The prescriptive quiz dif- than one concept including misconceptions and are in anal-
fers from a pre-test quiz in that in the former the number and ysis and application levels. They are worth asking again
composition of questions reflect the current ability of the student. because: (1) they are difficult questions, and (2) they cover
The questions can be chosen to meet different objectives such as more than one concept (including misunderstood concepts)
correcting misconceptions or boosting engagement and self-esteem. and it is likely that the student has accidentally answered
The PPQs used in our study were designed to improve the students’ them correctly.
self-esteem. Questions were selected based on how that individual • (allmuq): set of used questions which were answered incor-
and the group as a whole fared, with a small incentive of 2 marks rectly by the majority of students with the following criteria:
for retaking PPQs until one gets 70% or more. The effectiveness (1) the descending order of incorrect answers by all students,
of PPQs in promoting engagement and improving performance (2) prioritizing questions covering more than one concept,
were evaluated by designing anonymous qualitative surveys and and (3) selecting the top 90% percentile questions.
by comparing the pre and post-test performances of students in the
test and control groups.
By week 9 all core course material would have been covered
with rich learning analytics data gathered via responses to class Algorithm 1 PPQ Generator
quizzes. Figure 1 illustrates the intervention window from weeks 1: procedure PersonalizedPrescriptiveQuiz
9 to 12 when any student can get feedback through PPQ as many 2: PPQ = ∅
times as needed. Each PPQ is a collection of questions selected from 3: for each si ∈ S do
the same pool as class quizzes. By analyzing performance in these 4: input: si , mc i , ni , m f qi , muqi , uuqi , allmuq
quizzes, a set of questions are selected from the misunderstood 5: output: ppqi for si
concepts for each student. The number and type of questions in the 6: ppqi = ∅
set (ppqi ) depends on that student’s ability. The PPQ assigned to 7: while ni ≥ 0 do
a stronger student (ppqs ) will be more challenging but smaller in 8: add m f qi to ppqi
number than a weaker student (ppqw ) (that is, |ppqw | > |ppqs |). 9: ni − = |m f qi |
10: add muqi to ppqi
11: ni − = |muqi |
12: add uuqi to ppqi
(PPQ) per student 13: ni − = |uuqi |
14: add allmuq to ppqi
15: ni − = |allmuq|
16: add ppqi to PPQ
17: }
Pre Test(s)
Post 18: return PPQ
Test
19: }

W01 W02 W03 – w08 W09 – W12

Figure 1: The PPQ Approach - The Intervention Process
The PPQ generator algorithm below uses the following symbols:
• students (S): set of all enrolled students, {s 1 ,s 2 ,...,sn }.
• misunderstood concepts (mc i ): set of all concepts where the
student si got at least 50% wrong.
• (ppqi ): subset of dynamically selected questions based on
student si ’s misunderstood concepts to be used as the next
PPQ.
• (ni (== |ppqi |)): number of quiz questions, with default 25;
stronger students might have ni ≤ 25 based on the number
of remaining questions.
• (m f qi ): set of questions covering student si ’s misunderstood
concepts which have not been answered yet; questions with
analysis and application levels and covering more than one
misconception are prioritized.
• (muqi ): set of questions covering student si ’s misunderstood
concepts which have been attempted before.
The input of the PPQ algorithm for each student si is their set of
misconceptions mc i , the number of questions per student ni (ni =
25 by default), fresh questions covering misunderstood concepts
m f qi , used questions covering misconceptions muqi , difficult used
questions uuqi , and difficult used questions covering most students’
misunderstood concepts allmuq. The output of the algorithm is the
personalized prescriptive quiz (ppqi ) per student si .
Initially, the ppqi is null. In the first step, the algorithm adds
the list of fresh questions covering the misconceptions for each
student. The following steps will be performed if there is room to
add further questions (ni ≥ 0): (1) the system adds the set of used
questions covering the misunderstood concepts, (2) adding the set of
analysis and application level questions covering the misunderstood
concepts, and (3) adding the set of top 90% percentile questions that
most of the students have answered incorrectly. Finally, the student
si will be provided with their personalized prescriptive quiz ppqi .
Iterating the algorithm over all students, the system generates the
list of ppqi s for all students, thereby arriving at the entire set of
PPQs={ppq 1 , ppq 2 , ..., ppqn }.

53
ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus Reza Soltanpoor, Charles Thevathayan, and Daryl D’Souza

4 RESULTS Table 1: PPQ’s Impact on Class Test Results

In this section the qualitative and quantitative results are presented.
The responses to the PPQ survey questionnaire are discussed in Pre (10) Post (10) # diff % impr
Section 4.1, followed by the quantitative analysis in Section 4.2. 175 Control Group1 4.6 5.08 96 0.48 10%
(63.8%) of 274 of enrolled students participated in the PPQ approach. Test Group (a)2 6.26 7.42 175 1.16 19%
Test Group (b)3 6.41 7.52 151 1.11 17%
4.1 The PPQ Survey Responses % diff4 36.09% 46.06%
All students enrolled in the course including non-participants were Significance (p-value)5
invited to complete the survey questionnaire: Part A for those who
Control1 vs. Test Group (a)2 0.003516*
completed the PPQ and Part B for those who opted out. Out of the
Control1 vs. Test Group (b)3 0.007592*
274 students 120 (43.7%) completed Part A and 67 (24.5%) Part B.
1 Non-PPQ Takers
There were 120 responses to the key question in Part A “What 2 All PPQ Takers
aspect(s) of the system did you find more beneficial?”, with the top 3 PPQ Takers with ≥ 70 Marks on Their PPQ
five responses being instant feedback and explanation, personaliza- 4 Control Group vs. Test Group (a)
tion, identifying misunderstood concepts, variety of questions, and 5 α = 0.05
* p − value < 0.05
ease of use. Interestingly, even the majority of students in Part B
who opted out (63.5%) also answered “yes” to the question “Do you
think taking the PPQ might have impacted your score in the final Table 2: Control and Test Groups’ Performance in Final Exam
test?”.
Figure 2 summarizes the responses to the five Likert scale ques- MCQ (10) 1 SAQ (10) 2 OOP (10) 3
tions: 99% found instant feedback to their responses was useful,
98% benefited from PPQ intervention, 91% found PPQ helped them Control Group 6.09 7.61 4.33
to correct their misconceptions and made them more confident Test Group 7.08 8.22 5.2
of their progress, 85% acknowledged PPQ’s impacted their perfor- Difference 16.26% 8.02% 20.09%
1 Multiple Choice Questions.
mance positively, and 95% of the novices surveyed wanted the PPQ
2 Short Answer Questions (Code Fragments).
approach extended to other courses having found the personalized 3 Object Oriented Programming (Problem Solving) Questions.
assessment both enjoyable and beneficial.

PPQ Survey Results Table 1 demonstrates the results for the control and the test
groups in pre and post-tests, both of which carried a total of 10
instant feedback on my answers 99
marks. The test group was divided into: (a) all PPQ takers, and (b)
beneficial approach 98 those with PPQ scores ≥ 70, to measure the impact of the (2 marks)
incentive offered. The (#) refers to the number of participants in
made me more confident by removing my misconcep"ons 91 each group. The last two columns show the actual improvements
and the percentage improvements for the control and test groups.
performance was improved 85 The p−value for the improvements in the two test groups computed
with the confidence of α = 0.05 are shown in the last two rows,
should be applied in other courses 95 showing a strong significance on students’ performance from PPQ,
by rejecting the null hypothesis (HO ).
75 80 85 90 95 100
Table 2 presents the comparative performances of the PPQ and
the control group in the three equally weighted sections of the
Figure 2: PPQ Survey Results final exam, which carried 30% of the overall course marks. Note
both groups performed poorly in the object-oriented programming
(OOP) section where students were required to develop a complete
4.2 Quantitative Evaluation Java program based on the requirements specified. However, the
This section evaluates the impact of PPQs on student performance last row shows the difference between test and control groups is
in class tests. Our Test Group consisted of 175 students who took greatest (over 20% difference) in the OOP section suggesting the
the PPQ while the Control Group had 99 students who opted not to PPQ approach can contribute to improvement in problem solving.
take the PPQs. The two-tailed t-test was selected because we had Next, we studied the correlation between the PPQ performance
two groups and one independent variable (the impact of PPQ) to and the overall grade for the 175 students (63.8%), who took part
assess. The p − value was calculated with the standard confidence in the PPQ at least once. We were mainly interested in identify-
of α = 0.05. The null hypothesis was “HO : There is no significant ing the different categories of students in our diverse cohort, and
difference between the control and test groups”. In other words, the how best we could fine-tune the PPQ to meet their learning needs.
improvement from PPQ in the control group is the same as that Specifically, we aimed to study whether promoting engagement
of the test group. The improvement to student performance was through repeated attempts to get bonus marks was an effective
computed based on the difference between pre and post-class tests. strategy. Figure 3 shows a positive correlation of 0.49 between the

54
Adaptive Remediation for Novice Programmers through Personalized Prescriptive Quizzes ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus

Highest
PPQ and the final marks, which reveals engaging in PPQs and 100 (100) 100 (100)
Value

getting student specific feedback has a positive overall impact on

90 90
learning outcomes. Figure 3 also reveals three main clusters: (1) the (84)
red cluster to the right shows that the majority of students who 80 Q3 (80) 80
(76)
got high marks in PPQs also did well overall in the course, (2) the
students in the blue cluster on the left, however, reveals that there 70 Q2 (68)
70
(Median)
(64)
are a number students who performed poorly in the exam despite

Pos t Tes t
Pr eTes t s
60 60
their success in PPQs. These students may be the ones repeating Q1 (56)

the PPQs blindly to get the bonus marks without making any at- 50 50

tempts to clarify their misconceptions. Hence, we may limit the

40 40
permissible number of repeated PPQ in future, and (3) the third (36)
yellow cluster, in the bottom-right of the graph, shows a handful 30 30
of students doing well in the course despite their low PPQ scores. Lowest
(24)
Value
These are probably the more confident students from the high band 20 20

who may have attempted the PPQ once or discontinued midway

10 10
perceiving it to be of little value. Though such students were not
our main target group, it revealed the need to develop more PPQ 0 0

options to challenge our top students.

Figure 4: Pre-Test(s) Marks vs. Post-Test Marks
100

90
feedback and performance improvements have shown adaptive
prescriptive quizzes generated by such a framework can help boost
80 the confidence of stragglers and help narrow the differences in
diverse student cohorts. The use of bonus marks appear to have
70
motivated over 85% of PPQ takers to repeat the prescriptive quizzes
PPQ M ar ks

60
until the required standard was reached. The incentive scheme is
fair as it gives all students the opportunity to earn the 2 marks
50 bonus, with weaker students taking a longer route with increas-
ingly difficult questions. The analysis of exam results suggests a
40 framework explicitly capturing cognitive levels can help novices
improve their program-writing and problem-solving skills. Improv-
30
ing learning outcomes, however, requires delving into pedagogical
20 issues, reasoning about misconceptions, crafting and measuring the
effectiveness of new tasks capable of effecting cognitive changes.
10 Our web-based approach demonstrates such efforts can be reduced
0 by capturing and sharing misconceptions and effective tasks be-
0 10 20 30 40 50 60 70 80 90 100 tween institutions. The courseware, anonymized student data and
E xam M ar ks the tool for generating PPQ will be made available to instructors
on request, thus facilitating a multi-institutional study.
Delay in ethics approval prevented us from offering PPQ early in
Figure 3: Exam Marks vs. PPQ Marks Correlation the first run. In future, PPQs will be offered in the initial, middle and
latter parts of the course offering, which is likely to further improve
We also sought to analyze the impact of PPQ on the lower bands the learning outcomes. However, learning patterns identified in pre-
of our diverse student cohort, our main target. We did this by vious semesters (and not individual student data) will be the main
comparing the distribution of marks in the pre and post-tests. Figure basis for the initial PPQ. Another problem was that PPQ criteria of
4 clearly shows the greatest impact on the two lower quartiles. The improving self-esteem gradually by varying questions from familiar
lowest pre-test value of 24 (left) improved to 36 in the post-test and unfamiliar concepts did not appeal to some students. To better
(right). Also, the lower and the median quartiles in the pre-test (56 cater to our student diversity, we intend to give students greater
and 68) increased to 64 and 76, respectively, in the post-test. Figure control in customizing their own learning pathways, by specifying
4 also reveals changes in the upper bands though not to the same ranges for discrimination and difficulty indexes. One limitation
extent. These results suggest PPQ is an effective instrument for of our study was that PPQ and non-PPQ groups were not formed
dealing with diverse student cohorts. randomly, with self-selecting PPQ students starting with a much
lower base in the pretest. However, PPQs appear to be effective in
5 DISCUSSION AND FUTURE WORK lowering the gap between the two groups, which usually widens
Our main contribution in this paper was a novel framework for or- with time. We also note that the PPQ, and not simply repeating
ganizing tasks based on concepts and cognitive levels. The student the quiz, was the main reason for improvement as both groups

55
ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus
were allowed to repeat the standard quizzes without any limit. In
future, we will analyze aspects of PPQs such as the time spent per
attempt, between attempts and on the number of attempts, as well
as the impact of post-test outcomes. We also plan to improve PPQ
question selection by considering the impact of specific tasks in
bringing about cognitive changes.
6 CONCLUSIONS
Students in introductory programming courses continue to perform
poorly with high dropout and failure rates. The introduction of an
adaptive remediation approach through personalized prescriptive
quizzes improved the learning outcomes of novice programmers
significantly. The framework for organizing common misconcep-
tions into concepts and cognitive levels was shown to be effective in
narrowing the learning outcomes of students in large and diverse
cohorts. In the anonymous student feedback, 95% of the 120 re-
spondents wanted the approach to be extended to other cognitively
demanding courses. The benefit of the proposed approach is that it
allows common misconceptions and tasks facilitating conceptual
changes to be mined, using the rich meta-data captured.
ACKNOWLEDGMENTS
Thanks to Dr. Vu Mai for his invaluable contribution to the code
development.
REFERENCES
[1] S Adams Becker, M Cummins, A Davis, A Freeman, C Hall Giesinger, and V
Ananthanarayanan. 2017. NMC horizon report: 2017 higher education edition.
Austin, Texas: The New Media Consortium (2017).
[2] Lorin W Anderson, David R Krathwohl, P Airasian, K Cruikshank, R Mayer,
P Pintrich, J Raths, and M Wittrock. 2001. A taxonomy for learning, teaching
and assessing: A revision of Bloom’s taxonomy. New York. Longman Publishing.
Artz, AF, & Armour-Thomas, E.(1992). Development of a cognitive-metacognitive
framework for protocol analysis of mathematical problem solving in small groups.
Cognition and Instruction 9, 2 (2001), 137–175.
[3] Department of Education Australian Government and Training. 2017.
Release of the Higher Education Standards - Panel’s Discussion Pa-
per on Improving Completion, Retention and Success in Higher Educa-
tion. https://www . education . gov . au/news/release-higher-education-standards-
panel-s-discussion-paper-improving-completion-retention-and. (2017). [Online;
accessed 31-August-2017].
[4] A Bandura. 1986. Social foundations of thought and action (pp. 5-107). Englewood
Cliffs, NJ: PrenticeHall (1986).
[5] Theresa Beaubouef and John Mason. 2005. Why the high attrition rate for
computer science students: some thoughts and observations. ACM SIGCSE
Bulletin 37, 2 (2005), 103–106. https://doi . org/10 . 1145/1083431 . 1083474
[6] Matthew Berland, Taylor Martin, Tom Benton, Carmen Petrick Smith, and Don
Davis. 2013. Using learning analytics to understand the learning pathways of
novice programmers. Journal of the Learning Sciences 22, 4 (2013), 564–599.
https://doi . org/10 . 1080/10508406 . 2013 . 836655
[7] Maureen Biggers, Anne Brauer, and Tuba Yilmaz. 2008. Student perceptions
of computer science: a retention study comparing graduating seniors with cs
leavers. In ACM SIGCSE Bulletin, Vol. 40. ACM, 402–406. https://doi . org/10 . 1145/
1352322 . 1352274
[8] Paulo Blikstein, Marcelo Worsley, Chris Piech, Mehran Sahami, Steven Cooper,
and Daphne Koller. 2014. Programming pluralism: Using learning analytics to
detect patterns in the learning of computer programming. Journal of the Learning
Sciences 23, 4 (2014), 561–599.
[9] Richard F Burton. 2001. Do Item-discrimination Indices Really Help Us to Improve
Our Tests? Assessment & Evaluation in Higher Education 26, 3 (2001), 213–220.
https://doi . org/10 . 1080/02602930120052378
[10] Ricardo Caceffo, Steve Wolfman, Kellogg S Booth, and Rodolfo Azevedo. 2016.
Developing a computer science concept inventory for introductory program-
ming. In Proceedings of the 47th ACM Technical Symposium on Computing Science
Education. ACM, 364–369.
[11] Peter J Denning and Andrew McGettrick. 2005. Recentering computer science.
Commun. ACM 48, 11 (2005), 15–19. https://doi . org/10 . 1145/1096000 . 1096018
56
Reza Soltanpoor, Charles Thevathayan, and Daryl D’Souza
[12] Andrew Gibson, Kirsty Kitto, and Jill Willis. 2014. A cognitive processing frame-
work for learning analytics. In Proceedings of the Fourth International Conference
on Learning Analytics And Knowledge. ACM, 212–216. https://doi . org/10 . 1145/
2567574 . 2567610
[13] Shuchi Grover and Satabdi Basu. 2017. Measuring student learning in introduc-
tory block-based programming: Examining misconceptions of loops, variables,
and boolean logic. In Proceedings of the 2017 ACM SIGCSE Technical Symposium
on Computer Science Education. ACM, 267–272.
[14] Maya M Khanna. 2015. Ungraded pop quizzes: Test-enhanced learning without
all the anxiety. Teaching of Psychology 42, 2 (2015), 174–178.
[15] Hassan Khosravi and Kendra ML Cooper. 2017. Using Learning Analytics to
Investigate Patterns of Performance and Engagement in Large Classes. In Proceed-
ings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education.
ACM, 309–314.
[16] Marja Kuittinen and Jorma Sajaniemi. 2004. Teaching roles of variables in
elementary programming courses. ACM SIGCSE Bulletin 36, 3 (2004), 57–61.
https://doi . org/10 . 1145/1026487 . 1008014
[17] Catherine Lang, Judy McKay, and Sue Lewis. 2007. Seven factors that influence
ICT student achievement. In ACM SIGCSE Bulletin, Vol. 39. ACM, 221–225. https:
//doi . org/10 . 1145/1269900 . 1268849
[18] Alwyn Vwen Yen Lee and Seng Chee Tan. 2017. Understanding idea flow: applying
learning analytics in discourse. Learning: Research and Practice 3, 1 (2017), 12–29.
[19] Linxiao Ma. 2007. Investigating and improving novice programmers’ mental models
of programming concepts. Ph.D. Dissertation. University of Strathclyde.
[20] Wenting Ma, Olusola O Adesope, John C Nesbit, and Qing Liu. 2014. Intelligent
tutoring systems and learning outcomes: A meta-analysis. (2014).
[21] Michael McCracken, Vicki Almstrum, Danny Diaz, Mark Guzdial, Dianne Hagan,
Yifat Ben-David Kolikant, Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz
Wilusz. 2001. A multi-national, multi-institutional study of assessment of pro-
gramming skills of first-year CS students. ACM SIGCSE Bulletin 33, 4 (2001),
125–180. https://doi . org/10 . 1145/572139 . 572181
[22] Mark A McDaniel, Pooja K Agarwal, Barbie J Huelser, Kathleen B McDermott,
and Henry L Roediger III. 2011. Test-enhanced learning in a middle school science
classroom: The effects of quiz frequency and placement. Journal of Educational
Psychology 103, 2 (2011), 399.
[23] Arnold Pears, Stephen Seidman, Lauri Malmi, Linda Mannila, Elizabeth Adams,
Jens Bennedsen, Marie Devlin, and James Paterson. 2007. A survey of literature
on the teaching of introductory programming. ACM SIGCSE Bulletin 39, 4 (2007),
204–223. https://doi . org/10 . 1145/1345375 . 1345441
[24] Yizhou Qian and James Lehman. 2017. Students’ Misconceptions and Other
Difficulties in Introductory Programming: A Literature Review. ACM Transactions
on Computing Education (TOCE) 18, 1 (2017), 1.
[25] Vennila Ramalingam, Deborah LaBelle, and Susan Wiedenbeck. 2004. Self-efficacy
and mental models in learning to program. In ACM SIGCSE Bulletin, Vol. 36. ACM,
171–175.
[26] Michael C Rodriguez, Ryan J Kettler, and Stephen N Elliott. 2014. Distractor
functioning in modified items for test accessibility. SAGE Open 4, 4 (2014),
2158244014553586.
[27] Henry E Schaffer, Karen R Young, Emily W Ligon, and Diane D Chapman. 2017.
Automating Individualized Formative Feedback in Large Classes Based on a
Directed Concept Graph. Frontiers in psychology 8 (2017). https://doi . org/
10 . 3389/fpsyg . 2017 . 00260
[28] Takayuki Sekiya and Kazunori Yamaguchi. 2013. Tracing quiz set to identify
novices’ programming misconceptions. In Proceedings of the 13th Koli Calling
International Conference on Computing Education Research. ACM, 87–95.
[29] S Mohamed Shuhidan. 2012. Probing the minds of novice programmers through
guided learning. Ph.D. Dissertation. PhD thesis, retrieved July 2013, RMIT Uni-
versity: Australia.
[30] S Mohamed Shuhidan. 2012. Probing the minds of novice programmers through
guided learning. Ph.D. Dissertation. PhD thesis, retrieved July 2013, RMIT Uni-
versity: Australia.
[31] George Siemens and Phil Long. 2011. Penetrating the fog: Analytics in learning
and education. EDUCAUSE review 46, 5 (2011), 30. https://doi . org/10 . 17471/
2499-4324/195
[32] Reza Soltanpoor and Timos Sellis. 2016. Prescriptive analytics for big data. In
Australasian Database Conference. Springer, 245–256. https://doi . org/10 . 1007/
978-3-319-46922-51 9
[33] Maddalena Taras. 2005. ASSESSMENT - SUMMATIVE AND FORMATIVE -
SOME THEORETICAL REFLECTIONS. British Journal of Educational Studies 53,
4 (2005), 466–478. https://doi . org/10 . 1111/j . 1467-8527 . 2005 . 00307 . x
[34] Sven Venema and Andrew Rock. 2014. Improving learning outcomes for first
year introductory programming students. FYHE 2014 (2014).
[35] Susan Wiedenbeck, Deborah Labelle, and Vennila NR Kain. 2004. Factors affecting
course outcomes in introductory programming. In 16th Annual Workshop of the
Psychology of Programming Interest Group. 97–109.