Effects of Assessment Literacy on ESL Speaking Performance

EFFECTS OF ASSESSMENT LITERACY ON ESL SPEAKING PERFORMANCE

EMY HARLINA BINTI IBRAHIM Kolej Matrikulasi Selangor emyharlina@kms.matrik.edu.my

ABSTRACT
This study was conducted to examine the effects of assessment literacy on ESL speaking performance. It intends to help students to get better insights of how they are being evaluated in an oral presentation. Thus, it is hoped to improve their speaking performance and make them feel more confident in their own ability. This study involves twenty-two ESL matriculation students from Kolej Matrikulasi Selangor. The samples were given a speaking task which they have to present after a duration of three weeks. Subjects were exposed to assessment literacy after their first oral presentation. The presentation scores of before and after the intervention was recorded. Interview sessions were also conducted on the subjects to obtain their opinion regarding the approach. The effects of assessment literacy provided to the subjects on their speaking performance have revealed the result favoring the use of the approach. The outcome suggests that it could be integrated into traditional teaching sequence and instructional strategies. INTRODUCTION

Effects of Assessment Literacy on ESL Speaking Performance

Selangor Matriculation College (SeMaC) is the 13th college opened by the Matriculation Division under the Ministry of Education. Announced as the Centre of Excellence among all Ministry of Education matriculation colleges, SeMaC formed a Research Unit at the college level to improve the teaching quality through involvement in action research. Members of the academic staffs are encouraged to participate in writing and presenting their research papers in upgrading the quality of teaching and learning in their respective content areas. The concept of teacher as a researcher was taken up with the adoption of action research to improve teaching. It was argued that the practitioners should be involved in their own problem instead of becoming the subjects to be studied by an external researcher. Malaysia has adopted this idea and action research culture is being propagated in the education institutions nationwide. Zuber Skerrite (1982) proposed; through systematic, controlled action research, higher education teachers can become more professional, more interested in pedagogical aspects of higher education and more motivated to integrate their research and teaching interests in a holistic way. This, in turn, can lead to greater job satisfaction, better academic programmes development of new educational theories, improvement of student learning and practitioners insight and contributions to the advancement of knowledge in higher education. In the Matriculation colleges, the English Unit is in charge of teaching two English syllabuses. The first syllabus is the Malaysian University English Test (MUET) for students to prepare for the MUET examination which is the English entry requirement for our local universities. The other syllabus is the Matriculation English Program for students of the One-Year or Two-Year Matriculation Program prescribed by the Matriculation Division of the Ministry of Education. In both syllabuses, each component of reading, writing, listening and speaking, is tested separately unlike the tests that they are used to in primary or secondary school which comprise of more written tests. Students at the matriculation level face a transition from traditionally assessment method of paper and pencil type (Gronlund, 1998), terminal end year type examination to alternative assessment which looks at students performance in a continuous manner throughout the whole learning process. This includes oral presentations, portfolios, projects, article review, role-play etc. Even though they were taught all the four major skills; reading, writing, listening and speaking, but all throughout their primary and secondary education, the skills were not tested separately, like how it is done at the matriculation level. Reading, writing, listening and speaking is treated as an individual component which students are not very well-found of. Thus, this study aims to see if providing the matriculation students with assessment literacy will help improve their speaking performance. DEFINITION OF TERMS The notion of "literacy" has been interpreted in many ways (Wagner and Kozma, 2003). For example, not so long time ago, only a hand full of people regard computer skills as indispensable for assessment literacy (Sror, 2005). Those who are hoping for a single, cogent definition of "assessment literacy" that can fit in and works for all groups will be disappointed because the construct represents a wide matrix of skills which vary significantly from population to population. "Assessment literacy" from the viewpoint of a university student, a high school teacher, and a professional test developer probably involve vastly different skills and varies in its definition. Assessment literacy: What it means for teachers 2

Effects of Assessment Literacy on ESL Speaking Performance

The importance of assessment literacy for different groups also ranges extensively. Newfields (2006) suggested that there are three compelling reasons for teachers to be assessment literate. First, assessment is a widespread (if not intrinsic) feature of most educational systems. Teachers are estimated to spend from 10% - 50% of their work time on assessment-related activities (MacBeath & Galton, 2004, p. 31; Gunn, cited in Brindley, 1997) In many schools in the United States, a good portion of the budget also goes into formal testing. With so much time and money devoted to assessment, it's worth critically understanding how assessment decisions are made. He further explained the second reason assessment literacy is essential because it's necessary to understand much of the educational literature. A familiarity with basic statistical terms is needed not only to critically read specialized journals, but even many general articles in academic publications. A final reason assessment literacy is needed is that it allows teachers to communicate their own classroom results with others. As Hopkins (1985, p. 58-60) suggests, teachers should share their research with peers and develop a community that fosters learning. Mastering the conventions of qualitative and quantitative inquiry are essential to make classroom research comprehensible to a wide audience. Newfields (2006) also mentioned that assessment literate researchers should be committed to an ongoing self-critique their own research and sharing the findings in ways that are technically convincing. All too often articles with interesting insights lack sufficient analysis and/or evidence to allow readers to critically interpret the ideas. Assessment literacy: What it means for university undergraduates Instead of describing assessment literacy as a single phenomena with some sort of unitary meaning, this paper adopts a grounded theory perspective by suggesting it means different things to different populations. For students, it largely means knowing how to perform well in classroom evaluations, graded presentations and in exams. For teachers, it is associated with the ability to grade students ethically and accurately. And for professional test developers, every aspect of their work revolves around assessment literacy. A study conducted by Ohata (2005) suggests that test anxiety is far from rare in Japan. Many Japanese have what Baker (2006) describes as a victim mentality when it comes to testing. At the same time, Japan can be described as an edumetric society in which test results significantly impact life outcomes. One way of operationalizing assessment literacy in terms of what it might mean for university undergraduates can be list down as in Table 1: Table 1: A suggested operationalization of "assessment literacy" for one first year college student population (Newfields, 2006) 1 2 3 4 5 An ability to interpret statistical raw data in terms of common measures of centrality (mean, mode, median) and deviation (SD, quartiles). A basic understanding of the concept of measurement error and confidence intervals. An ability to discern whether or not the difference between two or more data sets is significant. A capacity to logically distinguish between correlation and causation. An understanding of what constitutes ethical assessment and what should be done if encountering unethical testing practices 3

Effects of Assessment Literacy on ESL Speaking Performance

These points represent many of the competencies outlined in the American Association for the Advancement of Science's Benchmarks for Science Literacy (1993). In other words, they denote what some educators think that university students should know. Chances are, this is far from what most students themselves actually want to know. Often, the main dispute in promoting assessment literacy seems to be convincing end-users that the topic is actually worth learning: when many people encounter the esoteric jargon and complex statistical formulas sometimes used in assessment, perplexity is a frequent response. (Newfields, 2006) There's so much to know and most learners (and perhaps teachers as well) are frankly unprepared. Generally speaking, little time is given in promoting assessment literacy skills in most curriculum in Japan: people take tests without considering why such tests are being conducted or whether they ethically assess a given skill. This I believe is a common scenario in many other countries as well, Malaysia unexceptional. PROBLEM DESCRIPTION Assessment is a fundamental part of the teaching and learning process. According to Belk and Calais (1993) assessment enables the teacher to gather information about the students progress, program goals and objectives as well as the extent to which methods of instruction deployed in the classroom are helping the students achieve these goals. In accordance to that, being an examiner at national level examinations has improved my teaching skills tremendously. My fellow friends who are assessment literate have shown a higher level of confidence in their teaching too. Their selection of methods and materials are tailored to meet students need for the assessment. Knowing how a task will be assessed guides our organization skills, time management skills, task fulfillment skills, creative thinking skills etc. It has given me a clearer guide on what is needed for my students to excel in their assessments. Assessment literacy helps to improve both teaching and learning. Personal Reflection After teaching for some time I noticed that my students written assignments or oral presentations do not meet the assessment criteria of the course. Their response was either too shallow, too general or somewhat superficial leaving me to wonder whether they actually have mastered the skills for each component. I reflect back on my own experience during examination coordination meetings, every examiner was clear with the standard set, with the evaluation criteria that was to be met. If a nationwide group of examiners can be standardize to evaluating the same sample script through assessment literacy, by having assessment literacy examiners have a strong content mastery for that particular test. I anticipated that it might work on my students too. If students have to develop tools and procedures for monitoring, recording, and assessing their friends progress and achievement in the classroom on a more systematic and formal basis (Brindley, 1997),in other words providing them with assessment literacy (Fullan, 1999), maybe it could give them some useful insights on what they are expected to do to improve their own performance. I decided to embark on action research to find out if providing assessment literacy will help my students improve their speaking performance.

Effects of Assessment Literacy on ESL Speaking Performance

RESEARCH OBJECTIVES The intention of providing assessment literacy is to help the students to meet the assessment criteria of each test better. Thus, this study aims to see if providing the matriculation students with assessment literacy will help improve their understanding of the different skills needed to score for each component. RESEARCH METHODOLOGY I believe that action research approach can contribute to our understanding of teaching and learning problems at tertiary level and it can contribute in finding solutions to some of the teaching problems. This methodology also offers a systematic approach of introducing innovations in the teaching and learning in the classroom. It would serve or encourage our lecturer to play a dual role of being a producer of educational theory as well as a user of that theory. (Like what I am attempting now) The approach provides one way of increasing our understanding about learning and teaching practices. Thus the approach is adopted for this research. RESEARCH PROCEDURE The sample for this study is a small number of students from the same class. On average we have 10-22 students in a class at the matriculation college. The Two-Year Matriculation Program was selected. This is because they have presentations in their syllabus. The instrument used in this study is the presentation assignment for the course. It is prescribed in the Matriculation syllabus. As a tool to elicit the results for assessment literacy, the scoring guide will be used. Another vital instrument in this study is the researcher herself. The researcher will be involved in the administrating of the study. The researcher will observe the subjects during the process to write down her reflection as a form of triangulation. A semi structured interview questions will also be used illicit further understanding about the effect of the assessment literacy treatment on the subjects. The interviews were conducted on the subjects and it is very practical to obtain useful insights and obtain accurate information. (Mohd Majid Konting, 1998) Some of the interview questions involve gathering information on subjects perception and experiences, it has an entity of a survey research method as well. Kerlinger (1986) suggest that survey research has strength in terms of realism, significance, theory-oriented and have a high quality of heuristic value. The qualitative method of data collection is used in this study to further support the quantitative findings. It is hoped the combination of these two methods will result in a strong outcome of the research. The students were informed at the beginning of the semester the format of the assessment. The students were to prepare a 10 minute Story Telling presentation after a duration of 3 weeks. The presentations were video recorded so that it can be use later in the coordination meeting. They 5

Effects of Assessment Literacy on ESL Speaking Performance

did the presentation in pairs but be graded individually. This was to reduce the anxiety and give them more confidence in doing the task. After 3 weeks they presented their Story Telling. During this time the researcher assessed them using the scoring guide. The researcher took down notes about each presentation and justification for the marks. The scores were recorded as Score 1. After all the subjects have presented, the researcher asked the subjects to make a scoring guide. The subjects were put in groups to discuss What are the criteria that they think an examiner should assess for the presentation. How many marks should be allocated for each criteria. Should all criteria carry equal mark.

All the response of the subjects will be written on the board for them to see. After the discussion the researcher will give the real scoring guides to the students. Then the students viewed their presentation recording. Each student wais to give their own marks for each of the presenters using the scoring guide. After that, the researcher compared the students marks (Score 2) and marks given by the researcher. (Score 1). Here is where the coordination meeting between the researcher and the students take place. Justifications of marks awarded and scoring points was discussed. Each student gave their own justification for the marks that they give for each presentation. The researcher provided assessment literacy to the subjects. The subjects learned how they were assessed. After the coordination meeting the subjects are given 1 week to do the presentation again. This time the scores (Score 3) was given again by the researcher and the subjects (Score 4). The scores between Score 1 and Score 3 was compared. This is to see if there is a significant improvement in their performance. Score 2 and Score 4 will be compared to see if the subjects have indeed gain assessment literacy from the treatment. RESEARCH FINDINGS Table 1: To find out if the subjects find the approach effective in developing their speaking performance. Response n 22 YES % 100 n 0 NO % 0

Table 1 indicates that the approach was perceived as effective in developing their speaking performance. Table 2: Reasons for Finding It Effective Reasons It makes them think like an examiner n 16 % 72.7

Effects of Assessment Literacy on ESL Speaking Performance

It gives them more confidence It gives them a clear guideline of what is expected of them during a presentation. It makes them see their mistakes It develops their speaking skills It is easier to see when others make mistakes

19 9

86.4 40.9

19 17 13

86.4 77.2 59

From Table 2, it was indicated that majority of the respondents feels that the approach helps them to see their mistakes and make them look at the assignment from a different perspective. Almost all of the respondents admitted to feeling more confident to do the presentation again after the treatment. Table 3 : Main Problem When Doing Oral Presentation No 1 2 3 4 5 Interview Items Not fluent with the language Do not know the best way to present Nervous in front of an audience Lack of self confidence Not enough time to practice n 6 14 5 9 4 % 27.3 63.6 22.7 40.9 18.1

From the interview conducted, it can be seen that majority of the respondents admitted that not knowing the best way to present is the main problem when doing oral presentation. Others admitted that they lack of confidence and experience nervousness when standing in front of an audience. Table 4: Perceptions Towards Assessment Literacy in Developing Students Speaking Performance 5= totally agree 4=agree 3=partially agree 2=disagree 1=totally disagree Interview Items 1 .
The approach helps me a lot in understanding the requirement of the task. The approach increases my 5 n 16 9 % 72. 7 40. 9 n 5 11 4 % 22. 7 50 n 1 2 3 % 4.5 9.1 n 2 % n 1 %

2 .

Effects of Assessment Literacy on ESL Speaking Performance

3 . 4 . 5 . 6 .

confidence in doing the 2nd presentation. The approach helps me to identify my weaknesses. I can help to teach my friends with this approach I can see other peoples strength in their presentation The approach gives me a sample of a good presenter for me to follow

12 8 8

54. 5 36. 4 36. 4 72. 7

8 14 8

36. 4 63. 3 36. 4 27. 3

4.5

4.5

27. 3

16

Majority of the respondents agreed that the approach used was very helpful in giving them a sample of a good presenter among themselves for them to follow. Through an understanding of what they were being assessed of during a presentation makes them feel more confident to do the 2nd presentation better. More than 70 percent respondents strongly agreed that the approach used helped them understand the requirement of the task better. TESTS RESULT Table 5 : Scores on Presentation Score 1 & Score 2DIFFERENCE Subj ects Score 3 & Score 4DIFFERENCE Presentation 1 (Total marks = 30) Score 1 Score 2 Presentation 2 (Total marks =30) Score 3 Score 4 Score 1 & Score 3DIFFERENCE +6 +6 +3 +5 +3 +3

1 2 3 4 5 6 7 8 9

19 17 17 20 25 26 22 22 17

25 26 26 26 27 28 27 27 26

+6 +9 +9 +6 +2 +2 +5 +5 +9 8

25 23 20 25 25 26 25 22 20

25 24 18 23 23 24 25 22 22

+1 -2 -2 -2 -2 +2

Effects of Assessment Literacy on ESL Speaking Performance

10 11 12 13 14 15 16 17 18 19 20 21 22

24 24 24 24 24 16 16 17 20 16 20 23 23 Total Difference

26 26 30 30 27 26 26 25 27 27 30 30 26

+2 +2 +6 +6 +3 +10 +10 +8 +7 +11 +10 +7 +3 138

26 25 24 25 24 24 24 26 26 24 22 25 20 24 22 25 25 24 20 20 26 26 26 25 23 27 Total Difference

+1 +1 +2 -2 +3 +4 +3 -1 -1 +4 2112=9

+2 +2 +8 +4 +5 +5 +4 +6 +3 -

STATISTIC OF SUBJECT PERFORMANCE To see if there are any significant changes in the scores of the first and second presentation after the provision of Assessment Literacy on the subjects, the marks are compared in table 6. Table 6 : Difference of Score 1 & Score 3 Total INCREASE(+) 15 ((68.2 %) DECREASE(-) 0 (0 %) UNCHANGE 7 (31.8 %)

REFLECTION OF RESEARCH Discussion of Findings Classrooms assessment that are effectively planned and systematically carried out can promote active learning and self-confidence amongst students. The treatment of assessment literacy given to the students has a positive significance and would open new possibilities and new approaches towards improving teaching and learning. The study has increased our understanding about learning and teaching practices. Thus the significance of this study is very much highlighted. My findings show a marked difference between score 2 and score 4 from the subjects. The subjects speaking performance has shown an increase of 68.2% from their 1st presentation while the difference between Score 3 and Score 4 had shown a significant similarity where the subjects scores and the lecturers score have a smaller gap. The results of this study support the notion that the exposure of assessment literacy on the students will indeed have a positive effect on their speaking performance. This study not only increased my students performance for their presentation, but it also rekindled my excitement for teaching because now my students are seeing what I am seeing. CONCLUSION 9

Effects of Assessment Literacy on ESL Speaking Performance

Although some may not agree with the use and methods in action research as compared to the real traditional empirical researches, I do believe that somewhere along the line there are some significant if not useful outcomes that contribute to the development of teaching and learning in the classroom. The next thing that interests me would be to see if Assessment Literacy will also have a similar impact on other samples of population, like lecturers and exam paper setters. Hopefully it too will produce the same if not more positive values that will help advance education in Matriculation colleges nationwide.

Bibliography Baehr, M. (2004) Overview of Assessment. In S.W. Beyerlein and Daniel K. Apple A Comprehensive Tool For Improving Faculty Performance. Lisle, IL: Pacific Crest. Bailey, K.M. (1998) Learning about Language Assessment. Dilemmas, decision and directions. London: Hainie and Hainie. Baker, B. (n.d.) Victim mentality. Accessed December 12, 2006 from http://www.selfgrowth.com/articles/Baker4.html. Belk, J.A and Calais, G.J (1993) Portfolio Assessment in Reading and Writing: Linking Assessment and Instruction to Learning. Paper presented at the annual meeting of Mid South Educational Research Association, New Orleans, November 10-12 (ERIC Document Reproduction Service No ED 365732) Berry, R. (2003) Alternative Assessment and Assessment For Learning. Paper presented at the 29th International Association for Educational Assessment (IAEA) Annual Conference, Manchester, United Kingdom. Chan SweeHeng, Sharon GohSeng Pang, Lee GeokIm, Ong Wei Siang, UPM, (2006) Exploring the Social Responsibility of Tests: Views From Univerity Lecturers and Students. Kuala Lumpur International Conference on Assessment. 10

Effects of Assessment Literacy on ESL Speaking Performance

Gunn, M. (1995). Criterion-based assessment: A classroom teacher's perspective. In G. Brindley (Ed.), Language assessment in action. Sydney: National Centre for English Language Teaching and Research, Macquarie University. Cited in Brindley, G. (1997, September). Assessment and the Language Teacher: Trends and Transitions. The Language Teacher Online. 21 (9). Accessed on January 28, 2007 from http://jaltpublications.org/tlt/files/97/sep/brindley.html. Hopkins, D. (1985). A teacher's guide to classroom research. Philadelphia: Open. University Press Newfields Tims (2006) (Toyo University) Authentic Communication: Proceedings of the 5th Annual JALT Pan-SIG Conference.May 13-14, 2006. Shizuoka, Japan: Tokai University College of Marine Science. (p. 48 - 73). Ohata, K. (2005, December). Potential Sources of Anxiety for Japanese Learners of English: Preliminary Case Interviews with Five Japanese College Students in the U.S. TESL-EJ, 9 (3) 1-21. Accessed on Dec 31, 2006 from http://www.kyoto-su.ac.jp/information/teslej/ej35/a3.pdf. W. James Popham Is Assessment Literacy the "Magic Bullet"? June 4, 2009 http://www.hepg.org/blog/19 Sror, J. (2005). Computers and qualitative data analysis: Papers, pens, and highlighters vs. screen, mouse, and keyboard. TESOL Quarterly, 39 (2), 321- 328. Shohamy, E. (1998) Critical language testing and beyond. Studies in Educational Evaluation, 24 (4), 331-45. Shohamy, E. (2001) The power of tests: A critical perspective on the uses of language tests. London: Longman. Subahan T. Mohd Meerah. (2006) An Action Research Approach to Alternative Assessment. Kuala Lumpur International Conference on Assessment. Suzieleez Syrene Abdul Rahim, Tajularipin bin Sulaiman (2006) Classroom Assessment : Paper-Pencil Vs Alternative. Kuala Lumpur International Conference on Assessment. Wagner, D. A., & Kozma, R. (2003, Oct. 1). New technologies for literacy and adult education: A global perspective. University of Pennsylvania International Literacy Institute. Accessed on June 12, 2006 from http://ncal.literacy.upenn.edu/products/wagner_kozma.pdf. White, Eddy Assessment literacy for effective classroom-based assessment Jan 11 2009 http://www.slideshare.net/ewhite/assessment-literacy-effective-classroombased-assessmentpresentation-907377

11