International Journal of Mathematical Education in

Science and Technology, Vol. 38, No. 1, 15 January 2007, 65–83

Introductory statistics, college student attitudes and knowledge – a

qualitative analysis of the impact of technology-based instruction

M. MELETIOU-MAVROTHERIS*y, C. LEEz and R. T. FOULADIx

yCyprus College, Nicosia, Cyprus
zCentral Michigan University, Mt. Pleasant, USA
xSimon Fraser University, Burnaby BC, Canada

(Received 14 July 2005)

This paper presents findings from a qualitative study that compared the learning
experiences of a group of students from a technology-based college-level
introductory statistics course with the learning experiences of a group of students
with non-technology-based instruction. Findings from the study indicate
differences with regards to classroom experiences, student enjoyment of statistics,
and student understanding of the many roles that technology plays in statistics.
However, no significant differences were found between technology-based and
non-technology-based instruction on students’ grasp of fundamental statistical
concepts. In particular, these findings agree with the findings of several other
studies, which indicate that incorporation of statistical software in the
introductory statistics classroom might not always be very effective in building
student intuitions about important statistical ideas related to statistical inference.

1. Introduction

The developments in computing technology during the recent decades have radically
transformed the culture of practicing statisticians. The calculating power of the
computer has relieved the burden of extensive computations underlying many widely
used statistical procedures. Computers have also improved the modelling of complex
systems, and have resulted in an improved ability to generate data simulating the
behaviour of these systems. These advances have opened new and exciting avenues
for increased understanding of stochastic phenomena and the robustness of analytic
data procedures. Similarly, technology has provided the opportunity to create an
entirely new learning environment; it has significantly increased the range and
sophistication of possible classroom activities [1]. While many statistics instructors
still choose to ignore the calls for reform, many others have responded to the calls
and have developed and implemented curricula that take advantage of technology to
expose students to a more data-driven approach to statistics [2–5]. In a recent survey
of introductory statistics courses in the USA, Garfield et al. found that most of the

*Corresponding author. Email: meletiou@cycollege.ac.cy

International Journal of Mathematical Education in Science and Technology

ISSN 0020–739X print/ISSN 1464–5211 online ß 2007 Taylor & Francis
http://www.tandf.co.uk/journals
DOI: 10.1080/00207390601002765
66 M. Meletiou-Mavrotheris et al.

courses used some type of technology, and most of the faculty anticipated increasing
or changing the use of technology in their courses [6].
Despite the wide use of technology in many statistics classrooms, relatively little
published research exists describing its impact on student learning. New curricula are
often developed and implemented without the benefit of research on their effects in
terms of student learning. Even when done, evaluation of technological interventions
is often very superficial and based on the performance in closed-ended tasks.
According to the research literature, there are many examples of alarming situations
in which statistics students get a correct judgment using an incorrect procedure,
or produce a correct solution but do not understand either the solution or even the
question behind it [7–10].
Having a vision of what technology is capable of is not the same as knowing how
to take advantage of these capabilities in a teaching context [1]. The amount of
research done, which investigates in depth whether students’ conceptual under-
standings change because of the use of computers, and if so in what ways, is still
limited. Several statistics educators [1, 11–13] have alluded to the fact that there is
still a lot to be learned about the use of technology as an instructional aid. Konold
warns that the assertion that technology interventions that we design and implement
will be effective without first testing them with students is a naive one [14]. Garfield
argues, ‘without scrutinizing what is really happening to our students and using that
information to make changes, it is unlikely that instruction will improve and we will
be able to achieve our new vision of statistical education’ [15]. More research is
needed in order to gain a better understanding of the source of learners’ difficulties
with probability and statistics, and use this understanding to build improved learning
environments.
The article describes findings from a qualitative study, which compared the
learning experiences of a group of students from a technology-based introductory
statistics course with the learning experiences of a group of students with non-
technology-based instruction. In particular, this paper addresses the following two
research questions in the context of a course on elementary statistics.
(a) How does technology affect students’ motivation and attitudes towards
statistics?
(b) How does technology affect students’ understanding of concepts related to
statistical inference?

2. Methodology

2.1. Context and participants

We conducted individual in-depth interviews with 22 students who had taken a
college-level introductory statistics course within three months after their completion
of the course. Our intent was to investigate students’ statistical reasoning, attitudes,
skills and thought processes, and to make a qualitative evaluation of the potential
impact due to technology usage.
The students interviewed were of varied mathematical backgrounds and had taken
the introductory statistics course in one of five different departments between two
campuses in the USA. They volunteered to participate in the study and were selected
on the basis of their overall performance in the statistics course. This ensured that an
 Introductory statistics, college student attitudes and knowledge 67
Table 1. Interviewees’ background.

No. of Grade received in

Type of course students Gender Major statistics course

Technology-based 7 3 Males 2 Education 3 As

4 Females 1 Computer Science 2 Bs
2 Accounting 2 Cs or lower
1 Meteorology
1 Math
Non-technology-based 15 7 Males 3 Education 5 As
8 Females 2 Sociology 5 Bs
2 Nursing 5 Cs
1 Advertising
4 Buiness
3 Math
1 Geology

approximately equal number of the students interviewed were in As, Bs, and Cs or
lower (see table 1 for some information on interviewees’ background).
Seven of the 22 students had taken a course following the PACE approach,
while the rest of the students experienced courses where non-technology-based
instruction was implemented. PACE (Projects, Activities, and Cooperative learning
using computers and Exercises) [16] is an approach that attempts to provide a
structured framework for integrating projects and hands-on activities conducted
cooperatively in a computer-based classroom environment. The nature of
intellectual activity in the PACE classroom was very different from that in the
non-technology-based classrooms. Whereas in the other courses instruction was
primarily lecture-based and incorporation of technology was minimal or non-
existent, in the PACE course technology was an integral part of instruction.
Each class meeting included laboratory time where students would work
collaboratively on activities designed to help them explore different statistical
concepts. Students had access to calculators and computers and used them to
perform calculations, create graphical displays, analyse real datasets, and conduct
simulations that illustrated statistical concepts such as sampling distributions and
the Central Limit Theorem. Minitab was the main statistical software employed
in the PACE course.

2.2. Instruments, data collection and analysis procedures

We conducted in-depth semi-structured interviews of each of the study participants.
During the interviews, we asked open-ended questions following the interview pro-
tocol in Appendix A as a guide. Despite the explicit design of the interview protocol,
the questions were posed in such a way as to provide direction to respondents rather
than restricting their responses. We were sensitive and responsive to issues identified
by respondents, and often changed the order of specific questions depending on
the interviewees’ responses. Nonetheless, before concluding each of the interviews,
we ensured that we covered all the topics included in the interview protocol.
Each of the 22 interviews, which were audiotaped, lasted for approximately two
hours. During the interviews, students were encouraged to use paper-and-pencil as
needed. In order to make students feel more comfortable, but also in order to get
68 M. Meletiou-Mavrotheris et al.

some information on their background, their attitudes and beliefs about statistics
as well as their experiences from the course, the interviewer started by asking some
general questions. Then, the following task was posed:
Some instructors believe that today’s (name of the institution) students are
not as good in math as (name of the institution) students were 20 years ago.
You are assigned a project to find out if there exists evidence to support
this belief.

Students were then told that the interviewer was interested in their thinking and
reasoning rather than in their getting the ‘correct’ answer. Although the interviews
proceeded depending on students’ responses, the interviewer guided the dialogue so
that the majority of the topics and main ideas encountered in an introductory
statistics course were brought into the conversation. A vast amount of information
regarding the culture of the statistics classroom and its impact on student attitudes
and learning was collected.
All of the interviews were transcribed and then analysed using the inductive data
processing method of constant comparison analysis [17] which involves unitizing,
categorizing, and coding by choosing words or phrases that specifically address the
research questions, and which assisted us in the search for patterns and themes that
were used to develop the study’s interpretation. Using related behaviours and
comments within each topic, we wrote descriptions of the students’ attitudes,
problem-solving strategies, and knowledge. These descriptions formed the findings
of the study. Some of the findings that relate to learning of key concepts regarding
statistical inference are described in the next section. In this section, names have been
changed to protect the identity of the participants. The symbol - P accompanies each
student from the PACE course.
Since the study was a qualitative interview study, and the number of students
in the study not as large as any quantitative study, the findings from the in-depth
interview should not be trivially extended to a general student population.
Nonetheless, they do provide some useful information to statistics educators that
it may not be possible to obtain using quantitative data gathering techniques.

3. Results

The interviews demonstrated differences between technology-based and non-

technology-based instruction with regard to college student introductory statistics
classroom experiences, student enjoyment of statistics, and student understanding
of the many roles that technology plays in statistics. However, no differences were
found between technology-based and non-technology-based instruction on students’
grasp of fundamental statistical concepts related to statistical inference.

3.1. The contrast of technology and non-technology-based instruction – effects on

student motivation and skills
Our findings indicate that students that had completed the non-technology-based
classes experienced a technique-oriented approach to statistics instruction, which
focused on formulas rather than empirical applications. Due to the time necessary
 Introductory statistics, college student attitudes and knowledge 69

for training students in computational methods, classroom examples and activities

were carefully chosen to make manipulation of formulas easier and students never
got to work with datasets that had more than just a few values. Several students
described the learning environment as rigid and most of the activities as contrived.
Others commented that, although recognizing how important statistics is, they felt
that the tools they obtained from this class were inadequate for them to be able to
solve any real world problems since they never had the opportunity to collect,
explore, and present any real data:
A lot of things we learned in the class were the basic technical things but we never
got like here’s a case, here’s all the data to it, now how do you figure through, no
practical application of statistics. (Lee Anna)
It was a little dry. A lot of times it was hard to visualize the concepts we were
talking about in statistics. (Andrew)
The professor lectured the whole time. In such a course you might be able to do
some more interesting things than just lecturing and working out the easy
problems on the board. Which is what he would always do and then we would go
home and have the really hard problems to do on our own. I’m sure there’s
something he could have done to make it more interesting . . . like present a real
problem and have us do something with it. (Kim)
When asked about statistical concepts, students from the non-technology-based
courses showed a strong dependency on formulas, often making several unsuccessful
attempts to remember formulas and rules that they had never understood.
These students also showed unfamiliarity with technology and lack of appreciation
of its role in statistical endeavours. For example, when asked what tool they
would use to get a random sample, only four of the fifteen students in the non-
technology-based group suggested using technology. These students realized that
‘that’s a lot to do physically, so we’d probably do it on a computer somehow’ (Bill).
The majority of the students, however, seemed to be completely unaware of the
fact that one could use a computer to generate random numbers. I’m not sure if
there’s really a tool that would help you do a random selection. There must be such a
tool, but I don’t know of one’ (Ariadne). The response of the following student,
who suggests scrambling up all 50 000 names, is a typical one: ‘I’m going to get
pieces of paper and a pen and write their names on the paper and pick my sample
from that’ (Bob).
Unlike students in non-technology-based courses, use of technology gave
students in the PACE course much more experience with data collection,
manipulation, and representation. Rather than spending most of their time
learning techniques, PACE students were given the opportunity to conduct
investigations and gain experience with the wide applicability and usefulness of
statistical methods. Engagement with activities using technology seems to have led
to an increased enjoyment and appreciation of the subject. Six of the seven
PACE students reported that despite their generally negative attitudes towards
statistics before taking the class, they ended up enjoying the course much more
than they had anticipated. John-P, for example, before taking statistics had the
impression that it was going to be nothing more than ‘number-crunching’,
‘playing with a bunch of statistics and figuring out averages.’ But, unlike his
70 M. Meletiou-Mavrotheris et al.

expectations, he ended up finding the course quite interesting. Computers had a lot
to do with this:
Yeah, it’s not so much as taking average. I mean, I really thought a lot of it was
just averages and stuff like that, but there’s more than just averages. You have
your errors, your skewed to the rights and lefts and all that stuff. Computers,
I really didn’t think much of . . . I mean I knew computers could help you,
but I didn’t realize with the graphs and stuff like that. I didn’t find it that . . .
I mean it wasn’t boring . . . I mean, I was expecting to fall asleep in class. Well,
I didn’t . . . (John-P)
Sara-P, a prospective secondary education mathematics teacher, also changed her
view of statistics after the PACE approach. Whereas before taking the PACE course,
whenever hearing the word statistics she would ‘automatically think of just an abstract
concept . . . numbers to samples, you know’, her experience with the course helped her
realize ‘what lies behind those numbers, how they get them.’ She believed that her
experiences with technology would help her a lot as a teacher: ‘We’re getting
computers more and more in the classroom and since we did it on the computers,
it’s going to be easier to do it that way than it was to try and do it by formulas’.
Ann-P reported her surprise at ‘the amount of computer hands-on stuff’ employed in
the class. She considered computers to be extremely helpful for instruction because
‘they make the class more interesting than if it was just numbers’ and manipulation of
formulas. When asked whether she thinks that the teaching of statistics could
have been as effective without the use of technology, she responded: ‘I don’t think so
because if you don’t use the computers, you’re not going to see the application of
statistics.’
Students in the PACE course came to view technology as an indispensable tool
in statistical problem solving. When, for example, asked what tool they would use to
get a random sample, six out of the seven PACE students mentioned a calculator or
a computer without further prompting. They were well aware of the fact that using
computers or calculators makes the process of generating random data much more
efficient and that ‘trying to do it yourself, that would take a lot of time.’ They realized
that, given the technologies available in modern society, it is silly for one to still do
statistical computations by hand:
Interviewer: So, before you took this class, what was your impression about
statistics based on what you heard from your friends or from others?
Mary-P: Very negative I think. Yeah. Because you hear all these horror stories
about people in statistics and they don’t quite do good or it’s hard or it’s . . . . you
know . . . . I guess I don’t really know. I guess I never really had statistics before
and I was just basing it upon what others said.
Interviewer: Did that affect your attitude towards this class?
Mary-P: I think at first, it probably did.
Interviewer: Were you scared?
Mary-P: Yeah. Maybe just . . . I think at first, I was like, why am I here? This has
nothing to do with what I am studying. I’m just going to be doing all this hard work
and it has no real benefit. I don’t know. I guess that’s what I was kind of thinking
at first. And then I found out it wasn’t as bad as I thought it was going to be.
 Introductory statistics, college student attitudes and knowledge 71

Interviewer: So, after taking the class, you have some different thoughts about
statistics?
Mary-P: Yeah. Well, now that I can see the value that it has, like in meteorology
for example, and seeing how it could benefit me in some of the . . . Like, I do a lot
of personal research, too. And, when I did this file I created for the city of Holland
last summer for record highs and lows, I went though . . . The way that I did it . . . .
I did it all the long way. It took me three months to do. Instead of getting all the
data and letting the computer do it for me, I went through it all one by one.
I started in 1948 and then every time I saw a temperature above a certain level,
I wrote it down for highs and went through every year, and when I saw a record
broken, I would replace it. And I went through every year like this. And it took me
three months to do that. And this was before I took statistics. So, if I had Minitab,
I could have gotten all of this on disc or file or something . . . When I ordered
the data, I ordered it on paper because I didn’t know how to work with a
computer at that time to do this. But if I could have gotten it on disc and
could have gotten the computer do it all for me, and it would have been done just
like that.

3.2. The contrast of technology and non-technology-based instruction – effects

on student learning of concepts related to statistical inference
We observed in this study that most students, regardless of whether they were
coming from the non-technology-based courses or from the PACE course, had
superficial and poorly interconnected knowledge of many statistical concepts.
Especially noticeable was students’ confusion about the nature and purpose of
inferential statistics. Almost all of the 22 students interviewed seemed very confused
about the concepts related to statistical inference.
Though students in the PACE course clearly gained a lot from the implementa-
tion of technology-based instruction compared to students in non-technology based
courses, use of technology did not seem very effective in improving student learning
of key concepts related to statistical inference. Nearly every single student in the
PACE group expressed frustration and failure to see the ideas underlying the
computer activities designed to help them improve their understanding of inferential
statistics. Their opportunity to use computer simulations as a tool for drawing
repeated samples from a specified population and summarizing the results, did not
seem to be very effective in helping them develop the idea of sampling distribution
and to see its connection to confidence intervals, p-values, and hypothesis testing.
Although students knew how to complete the computer activities, they failed to see
the purpose underlying these activities:
Yeah, all that. When we were doing it on the computer, I knew how to do it, but
I felt like I didn’t know why I was doing it. To me, it was just that we were using
too much of the computer and I didn’t understand what was going on because
I think that we didn’t have enough of it in the lecture. (John-P)
Findings from the interviews coincide with those obtained from an earlier study,
in which a survey had been administered to students who had just completed the
PACE course in order to investigate the impact that the course had on their attitudes
72 M. Meletiou-Mavrotheris et al.

towards statistics and on their motivation [18]. Most students reported that they
enjoyed the PACE course and were satisfied with the integration of computer tasks,
projects, and other in-class activities. The survey also required students to provide
a self-assessment of their level of understanding of the most important concepts
encountered during the course. Students reported having a poor understanding of
sampling distributions and the Central Limit Theorem and expressed the opinion
that the computer simulations employed in the course did not contribute towards
improved understanding of these concepts.
In order for students to be able to deal effectively with the statistics that they
encounter, understanding of statistical inference is essential. Students should be able
to make the necessary jump ‘beyond characterizing individual observations and
samples’ towards ‘using the data at hand to make inferences about the general
characteristics or ‘parameters’ of the population from which those data were drawn’
[19, p. 76]. However, since ‘statistical inference is almost by definition imperfect – all
sampling introduces some error’ [20, p. 4], students need to become aware of
potential threats to the validity of statistical inferences. They have to realize that
when sampling, there is always the possibility of ending up with a sample estimate
that differs from the population parameter because of bias or lack of precision.
Furthermore, they should understand that random sampling methods help avoid
biased samples and sample size controls the precision of the statistics obtained from
the sample. In the following section, student comments regarding method of
sampling, relationship between sample estimate and population parameter, and
effect of sample size on sample variability and on variability of the sampling
distribution of the mean are summarized. Because the analysis of the interviews
indicated no major difference in the level of understanding of inferential statistics
between students in the PACE and the non-technology based course, the
presentation of findings proceeds by considering all 22 students as a single group.

3.2.1. Method of sampling. When prompted about the method of sampling they
would use, the majority of students interviewed (18 students; 6 out the 7 students
from the PACE course, 12 (if applicable) out of the 15 students from non-technology
based courses) proposed random sampling because it would help to get a ‘good’
representation of the population. Several students correctly pointed out that random
selection helps get a better representation of the data by avoiding bias. For example,
Helen-P, when asked whether randomization helps to get a representative sample,
responded: ‘Randomization would just try and eliminate bias.’
In the extreme were students who seemed to believe that, given that the sample
selection was done randomly, one is almost bound to get a sample which is
representative of the population. Sara-P, for example, argued that demographic
factors should not affect our choice of sample if we do it randomly because
randomization provides ‘an accurate representation of the population . . . a fair . . . a
fair and accurate representation of the population’. Ann-P said, ‘random sampling
ensures that you have more accurate results’. Ariadne believed that randomness allows
studies with even a small size to be ‘pretty accurate’. Bill argued that randomization
allows you ‘to throw out the extreme highs or lows and you just get like an honest mix
of it’. This group of students perceived random sampling as being synonymous to
‘accuracy, correctness, and lack of error’ [21]; deviation of the sample estimate from
the parameter value did not have the statistical sense of random variation but
implied that something went wrong during the sampling process.
 Introductory statistics, college student attitudes and knowledge 73

However, some of the students did not trust simple random sampling and
preferred stratified sampling, claiming that simple random sampling might lead
to very unrepresentative samples. These students seemed to ‘equate random with
haphazard or without pattern’ [20, p. 4]. Margaret, for example, expressed distrust
towards simple random sampling and preference for a method that takes factors
such as geographic area into account, ‘so you get more of a spread out for people
coming to . . . [name of institution], so you’re taking samples of people from all over,
wherever they come from’. She did not recommend simple random sampling
‘because you could get a cluster of students that were the same or that came from the
same place or had the same teachers or something like that.’

3.2.2. Sample estimate vs. population parameter. During the interviews, students
were asked to describe what they would do if they found the mean of a sample taken
from a certain population to be different from the hypothesized population mean.
Three students (1 from the PACE course, 2 from non-technology based courses)
replied that if sample mean and population mean are any different, one should
conclude that the sample comes from a different population. These three students did
not consider the possibility that the observed difference might be the result of
random variation. The rest of the students realized that finding a difference between
population and sample mean is not adequate to make such a strong inference.
However, they remembered very little about what one should do to assess how likely
it is that an observed difference has occurred by chance. George-P, for example,
knew that ‘just because it’s a lower number, it doesn’t mean that they’re a lot worse’,
but when asked how he would decide, ‘how low is really low enough’, responded:
‘Um . . . I’m not sure. I remember that we did it, but I can’t . . . I don’t know if
I remember how it was set up’. Most students remembered very little about ideas such
as the Central Limit Theorem and hypothesis testing. Responses such as Alan’s were
typical:
Interviewer: Do you remember anything about the Central Limit Theorem?
Alan: Oh that . . . I didn’t like it then.
Interviewer: This is actually related to that (the sampling distribution of the
mean) . . . they go hand in hand . . .
Alan: I’m blocking out because it wasn’t my favourite part.

3.2.3. Effect of sample size on sample variability. All students understood that
different samples from the same population could (and usually do) vary. They
realized that the sample mean is not the same thing as the population mean and
nobody expected the two to be exactly equal. All students also understood that larger
random samples tend to produce better estimates and stressed that the larger the
sample size, the more representative of the population it is going to be. Despite this,
several misunderstandings regarding the effect of sample size on sample variability
were prevalent.
Many of the students in our study were found to have lack of statistical faith and
over-reliance on sample variability. They did not seem to appreciate the fact that,
despite the short-term unpredictability, stochastic events have a long-run regularity
74 M. Meletiou-Mavrotheris et al.

and, since statistical tools do take sampling variation into account, samples of even
a few hundred subjects do have a lot to say about the population. Seven of the
students interviewed (two from the PACE course, five from non-technology-based
courses) felt that a random sample of fewer than a few thousand subjects, even if
carefully chosen, would be too small and would very possibly lead to misleading
conclusions. Additionally, several of the students interviewed did not realize that
taking a small, in proportionate terms, sample of the population could often be
adequate for drawing valid conclusions. In total, nine students (three from the
PACE course, six from non-technology-based courses) said that one ought to
consider population size before deciding what sample size to choose, because sample
size should be at least a certain proportion of the population size. They went ahead
and specified the proportion of the population that they would sample, giving
percentages ranging from 10 to 50%. For one student, even taking half of the student
population did not seem adequate; anything other than a census was not convincing
enough.
On the other extreme were nine interviewees (three from the PACE course,
six from non-technology-based courses) who had more faith than they should have
had in small samples. John-P, for example, stated, ‘probably anything over 50 would
be good. You want to get as many as possible so you have less error, but if you put too
many people in there, you’re going to be sitting there hunting people down forever and
never getting your information.’ For Bob, ‘100 students would be OK’. Sara-P said:
‘You could probably . . . Probably if you went with 100 that would be large enough . . .
I don’t think you would get much more improvement over that.’ When the interviewer
told her that for this study we would typically need to ‘pick like 400 to 600 students’,
her response was: ‘That many?’ These students’ tendency towards deterministic
thinking can be interpreted as a reflection of their lack of awareness of variability
and its relationship to sample size. Similarly to the students who expressed over-
confidence in the accuracy of random samples of even a small size, they also
considered small samples to be more representative of the population than they
really are because they applied characteristics of large-sample variation to them.
It was not apparent to them that sampling variation depends on sample size and
that it takes a reasonably sized sample for random phenomena to display regularity.

3.2.4. Effect of sample size on the sampling distribution of the mean. The
‘recognition that the estimates of a population parameter will vary and that this
variation will conform to a predictable pattern’ [2, p. 137] is a critical step towards
developing the theory of statistical inference. However, when inquired about how
to construct the sampling distribution of the mean for a given sample size, most
students gave responses that revealed their ignorance and confusion regarding the
notion of sampling distribution Several students seemed to confuse sample means
with individual scores. Helen-P, for example, when asked to explain the meaning of
sampling distribution of the mean responded: ‘When I think of sampling distribution,
I think of like range of numbers or something’. Bob argued that the distribution of 200
sample means, each of sample size 100, is ‘going to be similar to the population’s
distribution’. Lee Anna, when asked whether she thinks that the spread of the
sampling distribution of the mean for a sample size of 400 will be as wide as that for
individual scores responded: ‘They are going to be down to 200 (the lowest possible
SAT score)’. In general, the different opinions that the interviewees expressed about
 Introductory statistics, college student attitudes and knowledge 75

the relationship between sample size and spread and shape of the sampling
distribution can be categorized as follows:
(i) correctly identifying the relationship between sample size and spread of the
sampling distribution and pointing out that the shape of the sampling
distribution will be normal (eight students; two of the seven students from
the PACE course, six of the 15 students from non-technology-based courses);
(ii) understanding that the spread of the sampling distribution decreases with
sample size, but not mentioning that the shape of the distribution becomes
more normalized (two students);
(iii) claiming that the spread of the sampling distribution increases with sample size
(six students);
(iv) claiming that the spread of the sampling distribution increases with sample size,
and the shape becomes more normalized (two students);
(v) believing that sample size has no effect on the sampling distribution of the
mean (three students);
(vi) Other responses (one student).

4. Discussion

The purpose of this study was to investigate how technology is changing the learning
of introductory statistics at the college level. The conclusions drawn concerning
the impact of technology should be treated carefully as the number of students
interviewed was a small voluntary sample that might have benefited from technology
differently from the rest of the students in these courses. Also, the effect of
technology cannot be judged in isolation from curricular materials and instructor
characteristics. What made the experience of students in the PACE technology-based
classroom distinctive was not the use of technology alone but the overall nature of
the course with its emphasis on problem solving and working with real-life problems.
However, this study does provide some insights into the effects of technology on
student learning of statistics, and highlights some of the continued difficulties in
helping students grasp fundamental statistical concepts.
In the study, we saw that engagement with technology had a positive impact on
the PACE students’ motivation, on their enjoyment and appreciation of statistics,
and gave them increased exposure and familiarity with the practical aspects of the
subject. At the same time, we observed that not only the non-technology-based, but
also the PACE students’ knowledge of statistical ideas seemed to often be superficial
and not well interconnected. In particular, technology did not adequately contribute
into improving student understanding of key concepts related to inferential statistics,
and particularly the notion of sampling distribution. Most students did not seem
aware of the transformation involved when moving from the distribution of
individual samples to the distribution of sample means; only a few students
had complete understanding of the process of forming a sampling distribution.
The opportunity PACE students had to work on tasks specifically designed to help
them improve their understanding of sampling distribution did not prove very
effective as an aid to achieve of this learning objective. Almost all students in
the PACE course expressed frustration and failure to grasp the ideas underlying the
computer activities; they noted that although knowing how to complete the
computer assignments, they failed to see the purpose underlying these assignments.
76 M. Meletiou-Mavrotheris et al.

Nonetheless, the use of technology had a positive impact on the PACE students’
motivation, helping them acquire a much better grasp of the practical aspect
of statistics, to come to ‘perceive statistics as a valuable tool rather than a bother’
[22, p. 57]. The experience of using technology in the classroom had an impact
on students’ appreciation of the use of technology as a tool to support research
processes and statistical analyses. However, arousing students’ interest is a necessary
but not sufficient condition for learning [23] and statistical competence with
technology is not adequate in its own right. As Starkings stresses, ‘statistics is a
subject that is both theoretical and practical by nature and needs to be addressed in
both of these aspects’ [12, p. 250]. Being statistically literate requires not only
the ability to process data using some software, but also the ability to understand
the assumptions behind the statistical analyses and the constraints imposed by the
underlying theory of the analyses [24, p. 307].
The results from our study agree with the findings of the considerable research
that has been done in the last thirty years, which attests to the many difficulties
people, even ones with substantive formal training, have with understanding
probability and statistics. It has repeatedly been documented that people’s intuitions
are weak and misleading and too often run counter to stochastic reasoning, and that
probabilistic thinking is not usually within the repertoire of the problem-solving
strategies people employ [25]. It has been stated, time and time again, that people
tend to think deterministically and lack awareness or understanding of variation
and its relation to sample size. They tend to (i) believe that any difference in
means between two groups is significant, (ii) have unwarranted confidence in small
samples, (iii) have insufficient respect for small differences in large samples, and
(iv) mistakenly believe that there is no variability in the ‘real world’ [26]. However,
most of this prior research was done with students in non-technological learning
environments. The implication from our study is that most of these misunderstand-
ings might still persist despite the use of technology and empirical activities, that
fluency with technology does not necessarily imply thorough understanding of the
underlying statistical concepts.
Other statistics educators have also questioned the effectiveness of computer
simulations in helping students develop the important ideas of statistics. Del Mas,
Garfield, and Chance, point out that ‘despite the accepted approach used to integrate
simulation software into a statistics class, there is little published research describing
and evaluating such an approach’ [27, p. 4]. In those few cases where formal research
was conducted to study the kinds of understanding that develop as a result of the
use of computer-based simulations, findings have not always been very impressive
[e.g. 2, 27–32]. Lipson for example, after conducting a study to examine the effect
of computer-based strategies on students’ understanding of sampling distributions,
found that ‘many of the propositions that seem paramount in an a priori analysis
to an understanding of sampling distribution do not seem to have been evoked by
the computer sessions, even though the sessions had been specifically designed with
these in mind and students were led to these propositions by the focus questions’
[2, p. 146].
The software typically employed in statistics classrooms, including the PACE
course that some of the students in our study had completed, are not pedagogical
tools but rather professional statistical software designed primarily for statistical
analysis rather than instruction. These professional packages support the conven-
tional technological approach of treating the computer package as a black-box.
 Introductory statistics, college student attitudes and knowledge 77

As the findings from our study, as well as other studies indicate, incorporation of
such professional software in introductory statistics classroom might not be very
effective in building student intuitions about fundamental statistical ideas such as
sampling distribution and statistical inference.
Scheaffer notes that although technological advances have made data exploration
‘fun and quick’, ‘inference is still a black box, whether done by hand or on the
computer’ [33, p. 157] and stresses the need for more thoughtful and constructive use
of technology in the classroom. Ben-Zvi argues that statistics instruction ought
to employ technological tools which support active knowledge construction, provide
opportunities for students to reflect upon observed phenomena, and contribute
to the development of students’ metacognitive capabilities [34]. Computer-based
learning environments should encourage students to: (1) practice data analysis with
an exploratory, interactive, open-ended working style, and combine exploratory and
inferential methods, graphical and numerical methods; (2) extensively use multiple
linked representations and simulations to construct meanings for statistical concepts
and ideas; and (3) construct models for statistical experiments, and use computer
simulations to study them [34].
Advances of technology provide us with new tools and opportunities for
supporting the development of student statistical reasoning. Lately, several newly
developed packages that were explicitly designed to enhance learning and make
statistical thinking accessible to students [35] have appeared. Their design encourages
students to experiment with statistical ideas, to make conjectures and use the
investigative tools provided by the software to test and modify these conjectures.
Meletiou-Mavrotheris [36] describes an instructional experiment that explored the
capabilities of one of these learning environments, the educational statistical package
FathomÕ [37]. Fathom’s designers draw on current constructivist theories of learning
as well on several years of academic research about the way students learn and
process statistical concepts and the main difficulties they face. The software has
affordances towards more constructive pedagogical approaches compared to
professional statistics software. Through attributes like the ability to dynamically
link multiple representations, to provide immediate feedback, and to transform a
whole representation into a manipulable object, it allows students to directly interact
with data and to see the immediate impact of their actions on the different data
representations displayed on the screen. Findings from the study indicate that use
of FathomÕ led students to the construction of a fairly coherent mental model of
sampling distribution and its connection to statistical inference.
Different types of software have different roles to play in statistics courses, and
whatever software is used, it should be appropriate for its educational purpose.
For example, one piece of software might be best to use to help students understand
sampling theory, but this might not be the best piece of software to enable them
to carry out an analysis of variance. Although the use of professional statistical
software in the classroom supports the acquisition of technology-based data analytic
skills, students’ understanding of fundamental concepts is not supported more than
in a non-technology-based course. Since a primary aim of many introductory
statistics courses is to help students of varied backgrounds and aspirations
understand some fundamental concepts related to probability and statistics, the
software used in the introductory statistics classroom should be aligned with
constructive views of learning if it is to support this aim. It should support content
and pedagogy by affording learners with tools they can use to construct their own
78 M. Meletiou-Mavrotheris et al.

conceptual understanding – tools for not only data display and visualizations, but
also thinking and problem solving. Thus, it is preferable for instructors of such
courses to include educational software developed for concept development rather
than solely rely on professional software designed for professional data analysis.
Use of the former type of software in the classroom supports active knowledge
construction by ‘doing’ and ‘seeing’ statistics in a powerful and flexible learning
environment [34]. It encourages students to build, refine, and reorganize their prior
understandings and intuitions about the stochastic, in contrast to black-box
simulations that do not allow direct connections between the formal and the
informal. This interaction between the data and the theoretical model is more
convincing than black-box simulations, and it helps students construct more
powerful meanings. By including both educational and professional software in a
technology-based learning environment, students’ motivation, understanding, and
skills may be simultaneously enhanced.

Appendix A: Interview Protocol

(A) Questions on students’ background, attitudes, and motivation

1. What is your major?
2. Why did you take this course?
3. Before taking this class, what was your impression about statistics?
4. After taking the class, do you have any different thoughts about statistics?
5. Do you consider statistics to be the same thing as mathematics?
6. Do you think statistics will be useful for your future career?
7. What type of learning style do you like the most?
8. What would be the important motivation for you to be willing to spend time to
study for a course?
(B) Questions assessing understanding of concepts taught in the introductory statistics
course Now we will start talking about some of the things that you learned in
your statistics course (explain that purpose is not to test him/her whether he/
she knows or doesn’t know something but to see how he/she thinks about the
different issues; goal of study is to find how students think about statistics and
use the findings to come up with ways to improve instruction):
Some students believe that today’s (name of institution) students are not as good
in math as students 20 years ago. You are assigned a project to find out if there
exists evidence to support this belief.

1. What would be the first thing you would do?

2. Which of the things you have proposed do you consider being the most
reasonable one? Why?
3. Let’s use the SAT score as our measurement. And, also, let’s assume we have
the mean score of (name of institution) students 20 years ago; it was 580 in
math. What is a reasonable target population from which you are going to
draw your sample? (All current students, all freshmen, all students in the state?)
4. So, you are ready to take a sample from the target population. How are you
going to take this sample? What would be a reasonable sample size? Why?
 Introductory statistics, college student attitudes and knowledge 79

5. Why take a random sample? Why not just your friends or just people from
a residence hall?
6. Any other way besides simple random sampling to choose your sample?
7. What demographic factors may affect your choice of sample?
8. How can you take a sample that guarantees that your sample includes all
(demographic factors mentioned above)? (purpose of question is to see what
they know about stratified random sampling)
9. Now, you have collected 400 SAT scores using simple random sampling or
stratified sampling for the purpose of comparing with the mean math
SAT score of 20 years ago. You are ready to summarize these 400 scores.
What information do you think will be important for describing these scores?
10. Do you know how the mean of the SAT scores is computed?
11. Let’s say that the mean score is 540. Explain what the value 540 means.
12. What other summary would you suggest?
13. Do you know how the median is computed?
14. Let’s say the median score is 535. What does this median score of 535
tell you?
15. Any other summary you think would be useful? (Variance, standard deviation)
16. Here is the formula for computing standard deviation (s.d), you do not need
to remember it (give formula). Can you explain what sample s.d. means?
What does it measure?
17. Let’s say the sample s.d. is 40. Can you explain what this value means for
our study?
18. What other type of information is important?
19. How would you graphically display the distribution of these 400 scores?
(Histogram, stem-leaf plot)
20. Verbally describe how a histogram is constructed.
21. For this sample of 400 scores, we have a mean of 540, a median of 535, and a
s.d. of 40. How do you think the histogram for this sample would look like
(symmetric, skewed-to-right, skewed-to-left)?
22. Could you draw a symmetric histogram? Could you give an example of
a population that might have a symmetric distribution? How about a skewed-
to-right histogram? How about a skewed-to-left histogram?
23. What about the scores in a test? Can you give a test situation where you would
expect the distribution of scores to be symmetric/skewed-to-right/skewed-
to-left?
24. If the histogram of the SAT scores ends up being symmetric/skewed, what does
this shape tell you about the population of SAT scores?
25. Suppose one of the scores in the sample is 590. You want to find out where this
score stands among all 400 scores. What do you need to find out? (Percentile)
26. Verbally explain what percentile means.
27. In this case, can 590 be in the 25 th percentile? Why or why not?
28. What would be a reasonable percentile for a score of 590? Why?
29. Given our sample characteristics, do you think it is reasonable to assume that
all (name of institution) scores follow a normal distribution? Why?
30. Suppose that the SAT scores of all (name of institution) students followed
a normal distribution with a mean of 520 and a s.d. of 50. Can you draw
a normal curve of the distribution?
31. Describe the difference between 520 (population mean) and 540 (sample mean).
80 M. Meletiou-Mavrotheris et al.

32. Describe the difference between 50 (population s.d.) and 40 (sample s.d.).
33. Can you estimate, given the population parameters, the chance of having
SAT scores higher than 520?
34. Let’s consider the area that is within two standard deviations of the mean
(420 to 620). What approximate percentage of scores do you expect to fall
between 420 and 620? Why?
35. Is it reasonable to say around 5%, 50%, 95%, 100%?
36. Let’s go back to our sample. We have obtained a sample of 400 scores and
calculated the following statistics: mean ¼ 540, median ¼ 35, s.d ¼ 40. Do you
expect the population parameters for all (name of institution) students to be the
same as these sample statistics?
37. Based on your response, if we take a different sample, but with the same
sample size, then we will probably obtain a different estimate of the overall
mean. How would you decide which one is a more accurate estimate of the
population mean?
38. Can you suggest any way that would have increased the precision of using the
sample mean to estimate the entire (name of institution) population mean
score? (Increase sample size)
39. Say we take another sample of n ¼ 3000 students and obtain an average score
of 512. Which one do you expect to be a more precise estimate of the entire
population mean? 540 or 515? Explain why.
40. Here we are talking about the precision of using the sample mean to
estimate the entire population mean. How can we measure the degree of
precision?
41. What would you expect the sample mean to be, if you kept on increasing the
sample size?
42. Can you verbally explain how the sampling distribution of the sample mean is
formed using the example of a sample size of n ¼ 400 drawn from the entire
school population?
43. What do you think the expected mean of all possible means, each of n ¼ 400,
would be?
44. What do you think the shape of this sampling distribution of the mean will look
like? Why?
45. The C.L.T. says: ‘the shape of the sampling distribution of the sample mean
will become normal, regardless of the original population from which we drew
the sample, as long as the sample size is large enough.’ Can you elaborate your
understanding of this statement? What does this mean to you?
46. If you increase the sample size, what will you expect to happen to the sampling
distribution of the mean as far as the shape is concerned?
47. Can you draw two relative distribution curves for the sample mean, one for
n ¼ 400 and the other for n ¼ 10?
48. Now, back to our original purpose: ‘To decide if there is strong enough
evidence that current (name of institution) students’ mean SAT score is
significantly lower than that of 20 years ago, when the mean was 580.’ What
kind of problem is this? What type of test is it? (Right sided? Two-sided?
Left-sided?) Why?
49. Can you give the hypothesis that we are testing? Ho: H1:
50. How are you going to decide if you should choose Ho or H1? (describe the
general idea)
 Introductory statistics, college student attitudes and knowledge 81

51. Remember that we obtained a sample of 400 scores with a mean of 540.
Is it reasonable, since 540<580, to claim that today’s SAT mean score is
significantly lower than that of 20 years ago? Why? Why not?
52. Will knowing the sampling distribution of the sample mean help us in making
our decision? How? (You may draw a picture to explain your thinking).
53. In testing this hypothesis, we often use a level of significance of, let’s say,
 ¼ 5%. What does this value mean? Why not take  ¼ 30%?
54. Can you give an analogy to hypothesis testing using a court case?

References

[1] Hawkins, A., 1997, Myth-conceptions. In: J.B. Garfield and G. Burrill (Eds) Research on
the Role of Technology in Teaching and Learning Statistics (Voorburg, The Netherlands:
International Statistical Institute), pp. vii–viii.
[2] Lipson, K., 1997, What do students gain from simulation exercises? An evaluation of
activities designed to develop an understanding of the sampling distribution of a
proportion. In: J. Garfield and G. Burrill (Eds) Research on the Role of Technology in
Teaching and Learning Statistics (Voorburg, The Netherlands: International Statistical
Institute), pp. 137–150.
[3] delMas, R., Garfield, J. and Chance, B., 1999, A model of classroom research in action:
developing simulation activities to improve students’ statistical reasoning. Journal of
Statistics Education, 7, 3. Online: http://www.amstat.org/publications/jse/secure/v7n3/
delmas.cfm
[4] Saldanha, L., 2001, ‘Is this sample unusual?’: An investigation of students exploring
connections between sampling distributions and statistical inference. Paper presented at
the Research Forum on Statistical Reasoning, Teaching and Literacy. Armidale,
Australia.
[5] Doane, D.P., 2004, Using simulation to teach distributions. Journal of Statistics
Education, 12, 1. Online: www.amstat.org/publications/jse/v12n1/doane.html
[6] Garfield, J., Hogg, B., Schau, C. and Whittinghill, D., 2002, First courses in statistical
science: the status of educational reform efforts. Journal of Statistics Education [Online],
10, 2: http://www.amstat.org/publications/jse/v10n2/garfield.html
[7] Estepa, A. and Batanero, M.C., 1994, Judgments of association in scatter plots: an
empirical study of students’ Strategies and Preconceptions. In: J.B. Garfield (Ed.)
Research Papers from the Fourth International Conference on Teaching Statistics
(Minneapolis: The International Study Group for Research on Learning Probability
and Statistics).
[8] Garfield, J. and del Mas, R., 1990, Exploring the stability of students’ conceptions of
probability. Research Papers from the Third International Conference on Teaching
Statistics (Dunedin, New Zealand: University of Otago).
[9] delMas, R. and Garfield, J., 1990, The use of multiple items to identify misconceptions in
probabilistic reasoning. Research Papers from the Third International Conference on
Teaching Statistics (Dunedin, New Zealand: University of Otago).
[10] Meletiou-Mavrotheris, M. and Lee, C., 2002, Student understanding of histograms:
a stumbling stone to the development of intuitions about variation. Proceedings of the
Sixth International Conference on Teaching Statistics [CD ROM]. Durban, South Africa.
[11] Burrill, G., 1997, Discussion: how technology is changing the teaching and learning of
statistics in secondary school. In: J.B. Garfield and G. Burrill (Eds) Research on the Role
of Technology in Teaching and Learning Statistics (Voorburg, The Netherlands:
International Statistical Institute), pp. 71–74.
[12] Starkings, S., 1997, How technological introduction changes the teaching of statistics and
probability at the college level. In: J. Garfield and G. Burrill (Eds) Research on the Role of
Technology in Teaching and Learning Statistics (Voorburg, The Netherlands:
International Statistical Institute), pp. 233–254.
82 M. Meletiou-Mavrotheris et al.

[13] Chance, B., 2001, Stats education: current and future directions. MSOR Connections
[Online] 1, 3 http://www.mathstore.ac.uk/newsletter/aug2001/pdf/statseddirections.pdf
[14] Konold, C., 1995, Issues in assessing conceptual understanding in probability and
statistics. Journal of Statistics Education, 3, 1, http://www.amstat.org/publications/jse/
v3n1/konold.html.
[15] Garfield, J.B., 1994, Beyond testing and grading: using assessment to improve student
learning. Journal of Statistics Education, 2, 1, http://www.amstat.org/publications/
jse.v2n1/garfield.html.
[16] Lee, C., 1999, A computer-assisted approach for teaching statistical concepts. Journal of
Computers at Schools, 16, 193–208.
[17] Glaser, B. and Straus, A., 1967, The Discovery of Grounded Theory: Strategies for
Qualitative Research (New York: Aldine).
[18] Lee, C., 1998, An assessment of the PACE strategy for an introductory statistics course.
Proceedings 5th International Conference on Teaching Statistics, pp. 1215–1221,
Singapore, July.
[19] Nisbett, R.E. and Ross, L., 1980, Human Inference: Strategies and Shortcomings of Social
Judgment (Englewood Cliffs, NJ: Prentice Hall).
[20] Jacobs, V., 1997, Missed opportunities on the teaching and learning of data and chance.
In: J. Garfield and J. Truran (Eds) Research Papers on Stochastics Education, pp. 3–37.
[21] Ballman, K., 1997, Greater emphasis on variation in an introductory statistics course.
Journal of Statistics Education, 5, 2, http://www.amstat.org/publications/jse/v5n2/
ballman.html.
[22] Lajoie, S., 1997, The use of technology for modelling performance standards in statistics.
In: J.B. Garfield and G. Burrill (Eds) Research on the Role of Technology in Teaching and
Learning Statistics (Voorburg, The Netherlands: International Statistical Institute),
pp. 70–75.
[23] Hawkins, A., 1997, Children’s understanding of sampling in surveys. In: J.B. Garfield and
G. Burrill (Eds) Research on the Role of Technology in Teaching and Learning Statistics
(Voorburg, The Netherlands: International Statistical Institute), pp. 1–14.
[24] Glencross, M.J. and Binyavanga, K.W., 1997, The role of technology in statistics
education: A view from a developing region. In: J. Garfield and G. Burrill (Eds) Research
on the Role of Technology in Teaching and Learning Statistics (Voorburg,
The Netherlands: International Statistical Institute), pp. 301–308.
[25] Pfannkuch, M. and Brown, C.M., 1996, Building on and challenging students’ intuitions
about probability: can we improve undergraduate learning? Journal of Statistics
Education, 4, 1, http://www.amstat.org/publications/jse/v4n1/pcannkuch.html.
[26] Shaughnessy, J.M., 1992, Research in probability and statistics: reflections and directions.
In: D. Grows (Ed.) Handbook of Research on Mathematics Teaching and Learning
(New York: Macmillan), pp. 465–494.
[27] delMas, R., Garfield, J. and Chance, B., 1999, A model of classroom research in action:
developing simulation activities to improve students’ statistical reasoning. Journal of
Statistics Education [Online], 7, 3. http://www.amstat.org/publications/jse/secure/v7n3/
delmas.cfm
[28] Lipson K., Kokonis S. and Francis, G., 2003, Investigation of statistics students’
experiences with a web-based computer simulation. Proceedings of the International
Association of Statistical Education Satellite Conference on Statistics Education: Statistics
education and the internet, Berlin. [CD-ROM] (Voorburg: International Statistical
Institute).
[29] Konold, C., Pollatsek, A., Well, A. and Hendrickson, J., 1990, The origin of
inconsistencies in probabilistic reasoning of novices. In: J.B. Garfield (Ed.) Research
Papers from the Fourth International Conference on Teaching Statistics (Minneapolis:
The International Study Group for Research on Learning Probability and Statistics).
[30] Hodgson, T.R., 1996, The effects of hands-on activities on students’ understanding of
selected statistical concepts. In: E. Jakubowski, D. Watkins, and H. Biske (Eds)
Proceedings of the Eighteenth Annual Meeting of the North American Chapter of the
International Group for the Psychology of Mathematics Education, pp. 241–246
(Columbus, OH: ERIC Clearinghouse for Science, Mathematics, and Environmental
Education).
 Introductory statistics, college student attitudes and knowledge 83

[31] Green, D., 1990, Using computer simulation to develop statistical concepts. Teaching
Mathematics and its Applications, 9, 58–62.
[32] Behrens, J.T., 1997, Toward a theory and practice of using interactive graphics in
statistical education. In: J.B. Garfield and G. Burrill (Eds) Research on the Role of
Technology in Teaching and Learning Statistics (Voorburg, The Netherlands:
International Statistical Institute), pp. 111–121.
[33] Scheaffer, R.L., 1997, Discussion. International Statistical Review, 65, 156–158.
[34] Ben-Zvi, D., 2000, Toward understanding the role of technological tools in statistical
learning. Mathematical Thinking and Learning, 2, 127–155.
[35] Rubin, A., 2002, Interactive visualizations of statistical relationships: what do we gain?
Proceedings of the Sixth International Conference on Teaching Statistics, Durban,
South Africa.
[36] Meletiou-Mavrotheris, M., 2003, Technological tools in the introductory statistics
classroom: effects on student understanding of inferential statistics. International Journal
of Computers for Mathematical Learning, 8, 265–297.
[37] Finzer, W., 1999, Fathom. [Software] 1993 (Berkeley, CA: Key Curriculum Press).