Session M4J

Critical Reading-an Evaluation of a Teaching

Approach
David F Dalton
Petroleum Institute, Abu Dhabi. ddalton@pi.ac.ae

Abstract - Reading for research purposes obviously goes
beyond basic comprehension and requires the reader to
have reasonable confidence in and mastery of a range of
critical reading skills. There is a generalized concern
about student proficiency in this area. This is a difficult
challenge in one’s own language and even more so for
foreign language students. The students involved in this
study are enrolled in a two-part Communications course
at the Petroleum Institute in Abu Dhabi, UAE. Reading
comprises an important part of their endeavours in the
programme and developing effective critical reading
skills is essential to their success. The purpose of the
study is to evaluate an approach to teaching critical
reading, employed by the researcher to determine its
effectiveness in developing such skills. A control and
study group with similar academic profiles were used
and a series of reading tests was administered to each
group. In addition, interviews and focus group were
conducted for qualitative data. Results across groups
were then compared. Test results indicate a higher level
of successful performance in the class exposed to the
reading approach.
Index Terms - communications, critical reading foreign
language, research
Approach
The approach employs a series of matrices and taxonomies
which capture some of the central concept-defining lexis.
These are used since the concept of matrix and taxonomy is
something students are familiar with from their studies in
science and engineering. Wherever possible, constructivist
thinking is referred to as this allows the learner to employ The concepts captured by the lexis in the glossary also
and build on existing schema and thus begin from a basis of
a least some familiarity. [1] Also, frameworks which are
familiar from other disciplines permit cognitive cross-
referencing and bridge building, thereby reinforcing the
integrated nature of learning and transfer of skill. The eight
stages outlined below are preparatory in nature and
demonstrate a process which students are encouraged to
internalize and activate during other reading work. As such
it may cover a number of learning periods as student
assimilation of the knowledge influences the pace. Explicit
reference will be made to the approach during further
learning (reading) periods. It also provides learners with a
glossary with which to discuss critical reading and thinking.
The concepts captured by the lexis in the glossary are also
978-1-4244-4714-5/09/$25.00 ©2009 IEEE
39th ASEE/IEEE Frontiers in Education Conference
M4J-1
encountered in generalized academic contexts and specific
areas of science content.
Stage 1 Context setting:
Discussion begins with what students understand reading to
be and what purposes they read for. Areas of difficulty in
understanding and factors which impede comprehension are
identified. Prior reading experience is described. [2]
Stage 2
Brainstorming on what different kinds of readers do. To
answer the question: “What is a critical reader?”
Stage 3
What skills do critical readers need? Discussion and report
back. Personal skills audit with exemplification to answer
the question: “Am I a critical reader?”
Stage 4
Comparison with critical skills taxonomy. Vocabulary focus
on unknown terms and concepts. Location of vocabulary in
other academic contexts. Discussion on the meaning and
usefulness of skills. (See TABLE 1)
Stage 5
Application of taxonomy to knowledge matrix. (See P5)
Stage 6
Application of skills and knowledge to text. Students self -
select difficulty of text according to Stage 3. (Texts have
different levels of ‘scaffolding’ added. One text is unedited).
Work on text and report back.
Stage 7
Connection of the above concepts to current need and
context-research project with particular reference to the
literature search/review.
Stage 8
Review. Further discussion on usefulness of the approach,
problems encountered, modifications and reflection.
provide the basis for exemplification and in-class discussion.
TABLE 1
Skills matrix
Determining advantages and disadvantages
Determining the accuracy of presented information
Determining relevance
Determining overgeneralization
Identifying exaggeration
Identifying points of view
Identifying missing information
Identifying implication
Distinguishing between fact and opinion
October 18 - 21, 2009, San Antonio, TX
 Session M4J
Method
Recognizing assumptions
Detecting inconsistencies in an argument
Judging essential and incidental evidence
Student Profile. Two classes of thirty seven Emirati males
Judging the credibility of a source from government and private schools currently in the
freshman year at the PI. All have a 500+ in TOEFL (Test of
TABLE 2 English as a Foreign Language) and all have successfully
Knowledge Matrix completed the first part of the two-part communication
programme. Their average age is 19. The two classes have a
Within text Beyond text similar GPA (grade point average) profile and the grade
Focus Action
range from the previous semester fell within a similar curve
Recognize Connect, The students are in two different classes on the same
evaluate, be
K influenced by,
programme following the same syllabus.
N apply
TABLE 3
Read O Select prioritize
Understand Specify use Grade breakdown.
W /purpose
L Group Grade A B C D F
E Test 7 9 2 0 0
D Control 8 8 3 0 0
G
E While homogeneity between classes would be almost
Contextualize impossible, these two groups at least display similar profiles
S use/purpose with respect to certain criteria relevant to the research.
Words The researcher’s class was the test group and the other the
K
Structure Describe
Concepts I Exemplify
control group. The test group was exposed to the approach
Meaning L Link described; the control group was not. Neither group was
L Develop aware that they were part of a research project.
Purpose Conclude Sample size might be considered an issue. However, as this
Opinion Evaluation, is a causal-comparative study and the sample in each group
implication,
assumption is not less than fifteen, [3] it can be considered reasonable
Fact Completeness for the research purposes.

Materials. These were as described above for the approach.

This approach is worked with over a period of weeks, but
always as a part of a range of reading activities directly
related to the students’ research, in order to give it purpose
and context [3]. The focus of the knowledge matrix
(knowledge is not being used here as a synonym for content,
but rather relates to concepts and language) is to stimulate
discussion that understanding within a text leads to
application beyond, it thereby reinforcing the idea of reading
purpose. This can then be explicitly related to the content
and purpose of the research report the students will later
produce. For example, the concepts of selection and
evaluation are linked (albeit not exclusively) to selection of
sources (literature review); the notion of implication can be
located within the context of the discussion section;
description related to results and so on, since students also
need to be able to read their own texts critically. The concept
of implication is linked to discussion and results sections of
the report.
These themes are returned to periodically during the
preparation period and worked with explicitly. Test texts are
then administered (twice for the purposes of this study) and
differences between groups noted. The texts are not
identified as ‘research’ or ‘test’ documents and were
therefore seen as part of the normal reading load.
The test readings were selected according to the following
criteria.
Length: text length (420 and 435 words respectively)
Interest: Test texts were selected according to area of
interest. [4] One of the texts focused on literacy and reading
which is a relevant topic for these learners. Much of the text
focused on reading in India, where a high proportion of UAE
residents come from. The second text focused on nuclear
power, an energy source currently of great interest here as
the government has recently signed agreements with France
and the USA for constructing nuclear power stations.
Level of difficulty: Level of difficulty can relate both to
subject matter and vocabulary [5]. Nation stated at a STETS
workshop in 2001 that for academic texts a reader would
need to be able to understand 98% of the running vocabulary
for full comprehension [6]. In this study students were
permitted to ask questions about vocabulary as they would in
any class. No such questions were asked, supporting the text
selection as having a high known vocabulary level.
Teachers from a language/communications background were
given texts to read and identify text opportunities related to
concept recognition to be tested. Ten examples of
distinguishing between fact and opinion were agreed and
five text opportunities lending themselves to implication.

978-1-4244-4714-5/09/$25.00 ©2009 IEEE October 18 - 21, 2009, San Antonio, TX

39th ASEE/IEEE Frontiers in Education Conference
M4J-2

The researcher then reviewed these and selected the areas of
commonality as those which would provide the test focus;
that is, these would be the text content students would be
expected to be able to identify and operate from.
The fact and opinion text was given first as it was seen as
less cognitively complex than the concept of implication as
the implication text went beyond recognition into
explanation and interpretation.
For the concept of implication, it is difficult to be totally
prescriptive since a range of implications may be possible
derived from text content/evidence. For each implication,
some notion of what is realistic had to be agreed, based on
text evidence. Interestingly teachers did not initially agree.
For example, the excerpt below had some colleagues
agreeing that 1 is possible but not 2 (see P 7). Since
‘alternative’ in this context can be taken to be synonymous
with ‘renewable’, it is difficult to see how both could not be
permitted. After group discussion, the researcher was the
final arbiter.
“In the quest for renewable sources of energy questions have
been raised about nuclear power since uranium is not
renewable and uranium resources are calculated by some to
last for between thirty and sixty years depending on levels of
production and use.”
Which of the following could reasonably be implied from
the above statement?
1 We should not invest so heavily in nuclear power as the
life-span may not be long. yes/no
2 We should put more money into researching and putting
on line alternative sources of energy. yes/no
3 Uranium is dangerous. yes/no
4 Uranium will run out. yes/no
While 3 may be true, it can in no way be stated as an
implication derived from the information in the text. Since 4
is actually stated in the text, this implication could not be
derived either. The readers in the test group within the
context of the approach have been taught to analyze how
evidence/statement in a text makes certain implications
possible and others not and also to be able to comment on
when such interpretations become untenable. However, the
researcher accepts this is a rather more ‘fluid’ area that fact
and opinion differentiation. Five agreed (clear) opportunities
for implication to be derived were agreed. These provided
the focus for testing.
978-1-4244-4714-5/09/$25.00 ©2009 IEEE
39th ASEE/IEEE Frontiers in Education Conference
M4J-3
Session M4J
Procedure
A pre –test survey was given to the test group. This elicits
evaluation of prior reading teaching and asks them to score
its effectiveness on a 5-point Likert scale. Respondents are
further asked to define and state the purpose of critical
reading. They are then asked to rate their understanding of
key elements of critical reading such as distinguishing
between fact and opinion, recognizing what may be implied
by the text, identifying points of view, recognizing
assumptions and so on. The test group is exposed to the
approach over a period of three weeks. Explicit reference is
made to the purpose, concept and skills of critical reading. In
the third week the first test text is issued to both groups in a
normal 50-minute class session. Neither group was aware
that this was research material. Results were analyzed and
collated. Further work was developed through the approach
for the following three weeks and then the second text was
issued in week six. Results were analyzed and collated.
A post-test survey was given to the test group. It is the same
as the pre test apart from asking respondents to evaluate the
teaching of reading in the class (same Likert scale) and
requesting qualitative data on if/how they feel their reading
may have improved. Pre and post surveys were compared.
Ad hoc individual sessions were conducted with 10 of the
test group (55%) about their experiences of reading after
having followed the approach. Specifically we wanted to
determine if they noticed any and (what) differences in
levels of confidence and understanding and allow some
correlation with the survey results.
As a gate- keeping procedure, a selection of responses was
sent to the original text readers and they were asked to
comment on whether these were reasonable implications.
A triangulation approach was used since by examining
information collected by different methods, findings can be
corroborated across data sets, reducing the impact of
potential biases that can exist in a single study. Cross-group
results were compared initially with a simple frequency
count of successful answers. For the implications text,
qualitative analysis of answers was applied focusing on how
realistic the implications were based on text evidence. There
is, of course a degree of subjectivity here. This was
minimized by giving a random selection of answers to a
group of language and communications teachers and
comparing their comments on the qualitative nature of the
student responses. Responses from the surveys and
interviews were then built into the overall analysis.
Results
The results from the fact and opinion text displayed in Fig 1
are immediately interesting. The results for the test group
show that 7of the respondents had 100% correct whereas
none of the control group achieved this target. Results for
nine correct answers were 4 and 3 respectively. Clear
differences can be observed for scores of five and under.
October 18 - 21, 2009, San Antonio, TX
 Session M4J
Seven of the control group scored four or less while none of Again, results are interesting, though analysis in this case is
the test group fell into this bracket. a little more complex for some of the reasons outlined on P
6. The same five text opportunities for implication were
TABLE 4 FACT/OPINION TEXT identified by 3 of the test group while none of the control
Test group: Control group:
group was able to do so. Exactly the same result is there for
4 same-opportunity choices. The bottom-end results are also
10 7 10 0 notable with 7of the control group having only one correct
9 4 9 3 choice and 2 with zero while the test group had 2 with one
8 3 8 3 correct choice and no respondents with zero.
7 1 7 1 However, going beyond the simple numerical count, issues
6 2 6 1 of accuracy must be considered. Of the two respondents in
5 1 5 3
the control group who identified five text opportunities, one
4 0 4 3
3 0
identified two of the pre-selected opportunities and one
3 2
2 0 2 1
identified none. This issue will be taken up in the discussion
1 0 1 1 section.
0 0 0 0 The quality of the implications derived from the text also
need to be compared. There were qualitative differences in
TABLE 5 IMPLICATIONS TEXT
many of the responses and this will be further addressed in
Test group Control group the discussion section.

9 0 9 1 Survey results
8 0 8 6
7 3 7 2 Comparison of pre and post surveys generally show a more
6 5 6 7 developed ability in the test group to define critical reading
5 8 5 2 and state purposes for which it is used. Descriptions were on
4 2 4 1 the whole more detailed and thoughtful showing less
3 0 3 0 superficiality than in the pre survey. A thorough reading of
2 0 2 0 the responses shows a clear (and sometimes impressive)
1 0 1 0
qualitative improvement in terms of thinking and
understanding.
Table 5 simply shows the numerical range of the number of Below is a selection of typical responses.
text opportunities for implication that were seen by the
respondents. No respondents indicated that there were less Student 1.
than four opportunities. Q What is critical reading?
None of the test group identified more than seven text Pre –survey: The ability to read quickly and fully,
opportunities, while 7 of the control group did. This included understanding the content and analyzing it.
areas of text where no implication was possible as all givens Q What is its purpose?
were stated. The range of error here is therefore noticeably To read articles for research and study.
less for the test group. Of the six to seven range, the spread Post survey: The ability to read beyond the text to compare
was very similar. Of the optimum five, the difference was 8 your own opinion with the writer’s opinion. To identify
to 2 critical areas and important information in the text that you
Next, results were looked at to see the numbers in each will use. To not just agree with what the writer says.
group who selected all or some of the same (teacher) five To produce more effective writing in the context of research
pre-selected opportunities within their range. The results are and study.
displayed in Fig 6. Student 2
Pre –survey: Critical reading is a way to read and
TABLE 6 TEACHER/STUDENT SELECTION understand.
Prepare for test and quizzes.
Test group Control group Post-survey: Critical reading is a skill that is applied to get
5 3 5 0 answers to specific questions when you read. Understand all
4 3 4 0 the concepts the writer uses.
3 4 3 3
Developing research and getting points of view.
2 6 2 7
1 2 1 7 Student 3
0 0 0 2 Pre survey: Don’t know.
No information on purpose
Post survey: A way of reading to analyze the facts, writer’s
point of view and hidden meaning of the topic.

978-1-4244-4714-5/09/$25.00 ©2009 IEEE October 18 - 21, 2009, San Antonio, TX

39th ASEE/IEEE Frontiers in Education Conference
M4J-4

Reading for research and report writing. To take information
from sources effectively.
Student 4
Pre survey: It is to read something and understand it
carefully.
It is used for understanding texts and to say what is it talking
about.
Post survey: Critical reading is to understand every bit of the
text and its meaning and also to understand the text from
different perspectives.
I use it to understand any text I am going to use for a
purpose.
Student 5
Pre survey: Critical reading is reading that gives a clear
image about the ideas of the text….
To identify the quality of a text.
Post survey: The ability to analyze the words and concepts
of a text in order to identify things like advantage and
disadvantage, facts and opinions, reliable information and
the quality of the text in general.
To communicate with the text in order to judge reliability,
key ideas etc.
Student 6
Pre survey: To read an article and understand it.
To study, to understand the texts
Post-survey: Understanding what, how and why the text says
what it say, to understand the purpose
To review sources, for study purposes.
Sudent 7
Pre survey: To read quickly
To study
Post survey: It helps me to understand the texts clearly and
effectively. Also it helps me to decide which text will be
helpful or not for my research.
To choose resources for example.
All respondents indicated that their reading had improved.
While this response is also high in our post- course survey
given each semester; here it was unanimous. Students were
asked to evaluate the effectiveness of the approach to
reading on a scale of 1 to 5, with 1 being not effective and 5
being very effective .The results are displayed in table seven
below.
TABLE 7 SURVEY RESULTS
5 4 3 2 1
8 7 3 0 0 notable. This is bourne out through the individual interviews
Results of student discussions
The focus here was on the idea of improvement. One basic
question was given about whether the students felt their
reading had improved or not and if so in what ways.
Information derived from the focus group shows the
following. Overall perception of improvement was
specifically stated. Below are a typical set of responses.
978-1-4244-4714-5/09/$25.00 ©2009 IEEE
39th ASEE/IEEE Frontiers in Education Conference
M4J-5
Session M4J
Student A I improved at recognizing overgeneralizations
and differentiating between facts and opinions. I also came
to recognize implications in the texts which I didn’t know
about before.
Student B Well my reading before was just reading the
words of the text and trying to know the surface meaning,
but now I know there is much more than meets the eye in the
text, where a text is a way of communicating what the writer
uses to deliver information that may not always be right and
may be exaggerated and maybe not credible.
Student C I can identify things like facts and opinions,
implications, relevance and overgeneralization.
Student D I can read better now...in a way I can understand
what the writer is really trying to say. I can differentiate
between facts and opinions better than before and my
vocabulary has increased from determining what new words
mean to me…
Student E I can pick out relevant and important information
and know the purpose of a piece of writing.
Student F This semester more than ever I got to read a lot of
interesting and fairly complex texts. This developed my
reading in a way that I can read with more focus and
understand what the writer is trying to convey and I can
identify the writer’s opinion about the topic.
Student G Well I can extract important information for my
purpose, differentiate between facts and opinions, know the
strengths of a text and overall if it is well written or not
Student H I thought the training to read critically was very
effective and it’s an important skill not only for the course,
but also for life.
Student I The reading was especially effective and mainly
in finding articles and proofreading my own work. I am
better at this now.
Student J Yes it has helped me understand the text clearly
and also to decide what I would use it for and whether it
could be helpful in my research.
Discussion
While the researcher accepts that the sample size is small,
triangulation of data permits the following qualified claims
to be made [7]. In summary, the results on text one are
immediately interesting for the high end and low end
grouping of effective responses: the test group simply had a
much higher level of success with the task. The fact that
only three student in the control group fell into the 9/10
range on this text, while 11 of the test group did, is very
where the test- group students indicated they had not had
difficulty with the task. In discussion with the colleague
teaching the control group, it was clear that completion time
on average was also faster than in the control group
Analysis of results from test two is more complex given
some of the issues identified in the procedure section (see
page 6) However, the number of choices at the 5 and 4 and
the 1 to 0 level again shows a clear difference in successful
performance. Further there are important differences
evident, with respect to the more wide ranging content of
October 18 - 21, 2009, San Antonio, TX

implications given and their quality. For example, in the
control group there was a noticeable tendency to paraphrase,
summarize and repeat information given in the text which
seems to indicate a lack of surety on behalf of the students as
to the nature of implication. Such tendencies were not nearly
so noticeable in the test group. These kinds of qualitative
differences were evident throughout the responses, with the
implications expressed by the test group being rather more
focused and going beyond the paraphrasing of a high
number of responses from the control group.
Further, there was an element of randomness in the approach
to the task taken in the control group. For example, four
respondents did not number the text as directed, five had no
numerical order and two repeated more than the same
number twice. Some respondents identified 8 and only wrote
about three. One respondent who identified 8 wrote about 7.
Two respondents Identified 7 but only wrote about 2. One
identified 7 and wrote about five. The respondent who
identified 4 only wrote about two. One respondent wrote
about six but with no reference to text location.
This problem did not occur in the test group. A number of
conclusions might be drawn from these differences in
approach to the task, performance and quality.
However, given that the groups had such a similar profile
and are in the same programme, following the same
syllabus, it is reasonable to assume that a major affective
factor in the success of the test group was the explicit
approach to critical reading they had been exposed to. This
was the only significant element of difference in classroom
practice in the context of reading.
The post-test surveys indicated an enhanced understanding
of critical reading measured by generally more sophisticated,
comprehensive and detailed definitions, as well as a wider
range of responses as to what these skills can be used for. It
is reasonable to conclude that the approach used was an
affective factor in the development of this understanding.
The student discussion indicated a similar development in
thinking and understanding in which students could clearly
describe what they were able to do as readers and make
reference to past performance to demonstrate qualitative
difference. Use of time adverbials such as ‘now’, ‘before’,
this semester’ are common in the responses. More
importantly, the approach is describes as ‘helpful’, ‘very
effective’ and ‘especially effective’ and specific reference
and exemplification is given to qualify these statements.
Recommendations
The writer does not seek to legislate on the classroom
practice of teachers. The choices we make as educators to
reach learning objectives by the most appropriate means our
experience and knowledge indicates, is one of the great
freedoms of teaching and the notion of prescription is often
(and perhaps rightly) seen as anathema.
However, there is obviously value in looking at the
effectiveness of different approaches used by others to
978-1-4244-4714-5/09/$25.00 ©2009 IEEE
39th ASEE/IEEE Frontiers in Education Conference
M4J-6
Session M4J
inform our own behaviours. This is in fact an integral a part
of informal discussion between teachers everywhere. What
is often lacking though is some kind of evidence to support
our notion that things ‘work’ and one issue that is clearly
important is the ability to go beyond the anecdotal. Often in
our profession we are influenced by the ‘feeling’ that
something works well but perhaps have little measure of
‘how’, and trade techniques accordingly.
The classroom is a rich environment for carrying out action
research into teaching and learning and particularly in terms
of testing our own beliefs as educators as to what constitutes
effective approaches to learning.
Conclusion
This short study has sought to give some basic analysis and
data that go beyond the subjective in order to show that a
particular approach to teaching the skills identified, has
measurable value for the learner and can be demonstrated to
be effective. Such studies are significant examples of action
research and clearly have a role to play in stimulating
discussion of pedagogy in a variety of contexts.
REFERENCES
1 Piaget, J., and Garcia, R. (1991). Towards a Logic of Meanings. Hillsdale,
N.J.: Lawrence Erlbaum Associates
2 Linderholm T, Van der Broek P The Effects of Reading Purpose and
Working Memory Capacity on the Processing of Expository Text. (2002)
Journal of Educational Psychology v94 p778-8
3 Cohen L, Manion L, Morrison K (2003) Research Methods in Education,
5th Ed. Routledge UK, 93.
4 Day R and Bamford J (2002) Reading in a Foreign Language,Volume 14,
Number 2, October 2002 Pp146 to 147
5 Nation P, (2001) Teaching and learning vocabulary, STETS Public
Lecture and Workshop, 28 November 2001 - 1 December 2001
6 Luke A (2000) Critical literacy in Australia: a matter of context and
standpoint. Journal of adolescent and adult Literacy, 45(5) P 448 – 461
7 Massey A (1999) Methodological Triangulation,
Or How To Get Lost Without Being Found Out
Reproduced by kind permission of JAI Press
from Massey, A. and Walford, G. (Eds.) (1999) Explorations in
methodology,
Studies in Educational Ethnography, Vol.2, Stamford, JAI Press, 183
October 18 - 21, 2009, San Antonio, TX