1

Language Testing: Process Evaluation

Name Class No. Pokok

: Muhammad Rum, S. Pd :C : 12B0112

GRADUATE PROGRAM STATE UNIVERSITY OF MAKASSAR

Abstract Evaluation is defined as judging worth. What is to be evaluated is educational program which is an on-going activity designed to modify peoples behavior in desirable ways. The fundamental purpose of evaluation is to produce information and use it to make decisions about an educational program. Since evaluation is about decision making, therefore the decision might be to continue an existing program, to terminate it, or perhaps to modify it. Or it might be to develop a new program with a view to a possible adoption. There are two ways in evaluating, formative evaluation and summative evaluation. The previous one is conducted during the program and the latter is conducted to draw the conclusion based on the program that has been conducted. In evaluating the evaluation that is on progress, there are four stages that can be conducted. Before starting off the program, the teacher should prepare all the materials needed, namely, curriculum, syllabus and teaching material. During the program, both the teacher and the students can observe each others behavior in the classroom. The teacher can evaluate the learning style of the student in the classroom or the student can observe the teaching techniques used by the teacher. After gathering all information, the data should be studied just to eliminate the thing that makes the program runs unsuccessfully.

1. Introduction Evaluation is an effective means of measuring teaching and learning performances in a language program and of improving the teaching process. It can be used to trace both teaching procedures and learning progress. By comparing their teaching to the syllabi and any other documents, teachers can evaluate their teaching and adjust their teaching strategies to meet the learners needs and the requirements of the program. Evaluation is a process to judge or measure the value of a finished or ongoing program, plan, or even a policy (Gasper 1995). In the language teaching field, especially in ESL/EFL programs, there are numerous reports on how to apply evaluations to class activities and program assessment (McDonough and McDonough 1990; Marrow and Scholker 1993; Williams and Burnden 1994; Carry and Dauber 1995; Mackay et al., 1995). An approach based on the evaluation theory is proposed by Jack C. Richards and Charles Lockhart (1995). This article will focus on the ongoing evaluation of a language program and the roles of the teachers and learners in the process. A. Definition of Evaluation To measure the students progress or the students understanding concerning with the subject that they have been taught is by conducting a certain test. Each subject has its own test which is taken place by the educators. A test is one of the components of the evaluation. According to Rea-Dickinsins and Germaine (3: 1992), evaluation is an intrinsic part of teaching and learning. By evaluating the students is considered to be beneficial because it can provide a wealth of information to use for the future direction of classroom practice, for the planning of courses, and for the management of learning tasks and the students. To emphasis the statement, they, then, add additional definition of evaluation that is evaluation is not restricted to the context of education; it is a part of our everyday lives. Based on the previous definition of evaluation that is mentioned above, Pilliner (97: 1968) shares the similar definition of evaluation by stating that evaluation is to make a judgment of the worth or value of something.

To get the in-depth definition of evaluation, Pilliner (97:1968) provide some questions that leads the function of evaluation, as follows. 1. What are we to evaluate? What are we to evaluate to judge the value or worth of is an educational programme or project defined, with Austin and Panos, as any ongoing educational activity which is designed to produce specified changes in the bahaviour of the individuals exposed to it 2. How do we evaluate? Starting with the premise that evaluation involves the collection of information about the impact of the educational programme. There is no doubt that for many educators and researchers the task of evaluating programmes is associated mainly with the construction and administration of achievement tests. It goes without saying that the assessment of student achievement or progress and that well-contructed achievement test can contribute importantly to such evaluation. In this context, evaluators need to consider which of the two test styles; norm referenced or criterion-referenced, is the more suitable for their purpose. Unless that purpose is to rank students (which is scarcely an educational objective), the evaluator will normally opt for criterion-referencing procedures. 3. When we are to evaluate? According to Scriven in Pilliner (97: 1968), there are two types of evaluation. The first is formative evaluation which is taken place during the educational programmes. Formative evaluation refers to data emerging on taking stock at some intermediate stage, leading probably to slight modification or possibly even to substantial design of sequent procedures. Following to the first type of evaluation is summative evaluation. On the other hand, summative evaluation refers to an evaluation of effectiveness or success of the programme as a whole after it has been completed. Particularly with the extensive programmes, both formative and summative styles are essential. It would be unrealistic to suppose that no

change need ever be made from initial plans. The programme would be pointless if no-one were concerned to establish overall and final effectiveness. The roles of the formative and summative evaluator are in strong contrast. Though both are concerned in making judgment, their standpoints are very different. The essential thing in formative evaluation is close cooperation between evaluator and program developer, interplay and involvement in smoothing out difficulties as they occur and in maintain momentum. The essential thing in summative evaluation is total independence on the part of the evaluator, disinterest and uninvolvement and commitment only to dispassionate analysis and reporting. To get the in-depth distinction between formative and summative evaluation, the following is table taken from Language Curriculum Design book written by Notion and Macalister (126: 2010).

B. Steps in Evaluation All of the early steps in evaluation aim at deciding why the evaluation is being done and if it is possible to do it. According to Nation and Macalistor (124: 2010), there are several steps in conducting evaluation, as follows. 1. Find who the evaluation is for and what kind of information they need. 2. Find what the results of the evaluation will be used for to improve the course, to decide whether to keep or get rid of the course. 3. Decide if the evaluation is necessary or if the needed information is already available.

4. Find how much time and money are available to do the evaluation. 5. Decide what kinds of information will be gathered. a. b. c. d. e. f. g. h. i. j. k. Amount of learning Quality of learning Quality of teaching Quality of curriculum design Quality of course administration Quality of support services library, language lab, etc. Teacher satisfaction Learner satisfaction Sponsor satisfaction Later success of graduates of the course Financial profitability of the course.

6. Try to gain the support of the people involved in the evaluation. 7. Decide how to gather the information and who will be involved in the gathering of information. 8. Decide how to present the findings. 9. Decide if a follow-up evaluation is planned to check the implementation of the findings. C. Definition of Evaluation Process/Ongoing Process Ongoing evaluation, which occurs during the progression of an action, is a widely used means of judging the progress and achievements of language programs. Developers and supervisors often use the results of their evaluations to rate programs. Evaluations may help directors decide such matters as ranking teachers, funding the programs, and promoting teachers. In some cases, evaluations include input from colleagues and supervisors, with teachers and learners playing only a passive role in the evaluation process. The ongoing evaluation of language programs can assist in improving teaching and learning practices and help teachers and learners. First, teachers and learners can objectively view their work and performances during the course. Second, they can better understand the course progression and its

relation to the programs goals. Third, teachers and learners can benefit from the evaluation by modifying their teaching or learning strategies, thus improving their performances. By participating in the ongoing evaluation program, teacher and learners become more involved in the program. Finally, the evaluation project can help to create good rapport between teachers and learners. Ongoing evaluation is a systematic and reflective process to measure the program. Unlike other evaluations, this project usually involves only the teacher and learners. It can be considered a self-evaluation process. Administrators or supervisors of the program may provide background information or other materials. They also can be asked to inspect the project at its conclusion. D. Stages in Evaluating The teacher and learners should perform ongoing evaluations periodically. An initial ongoing evaluation may be carried out in the first week. The tone here will probably have a greater overall effect on the success of the course than what occurs later, since initial impressions are very often more enduring than later ones (Hutchinson and Waters 1987). Formal ongoing program evaluations may be at one month intervals or at each stage of the program as a check on the progress of the courses. Informal evaluations may be carried out whenever the teacher and learners think it is necessary. It is important at all stages that data be stored in a database for analysis. The ongoing evaluation process is divided into four stages. (See Figure 1) 1) Stage 1 In the preparation period of the program, the teacher will need to have complete program documents including the curriculum, syllabus, and any other related materials. The teaching materials should be evaluated to see that they correspond with the programs requirements and teaching methodology. The learners data, like personal information, language proficiencies, language learning strategies, and other psychological factors,

should be collected. From the data collected and studied, the teacher should have a thorough knowledge of the program and the learners. Then a detailed teaching plan can be written. 2) Stage 2 After the language program has started; the teacher and learners may use informal evaluations to measure their performances. Both the teacher and learners should be actively involved in collecting data and analyzing them periodically during the course. Teachers and learners will need to observe each others performance. Besides teaching, the teacher should conduct studies on the learners styles and their learning strategies. This may include recording students responses to the teachers questions and the roles learners play in the discussions. The learners will need to observe the teachers performances during classroom activities and the teaching techniques used. Both the teacher and learners can share views on their observations and make comments on questionnaires and during periodical interviews and group discussions. Learners should receive feedback from the teacher on corrections and personal conversations with the teacher. The teacher should become familiar with the learners learning styles in order to compare the learners classroom behavior to their learning strategies, which were collected during the first stage. Also, the teacher can assist learners in adapting their learning strategies to suit the present program. Conversely, the learners may study the teaching strategy of the teacher and make comments, so adjustments can be made to suit their own needs and the goals of the program. 3) Stage 3 After a certain period, a formal or periodic evaluation needs to be carried out. During this period, the data previously collected are studied to see if the goals have been achieved. At this point, the teacher and supervisor can study all the data such as scores, comments, and other feedback from

the learners. Periodic discussions may be needed so teacher and learners can exchange views on each others styles. Documents such as syllabi, informal evaluation results and materials of other similar programs can be studied and compared to the evaluation results, so as to obtain an objective view of the programs progress. The learners should also write brief summaries of their learning progress during this stage. The teacher can then revise the plan for the coming sessions. 4) Stage 4 When the program is concluded, another evaluation should be undertaken. The teacher and the supervisor of the program need to study all the data collected during the course of the program. The result of this stage should be an objective comment and/or statistical conclusion that summarizes the progress of the program. This is later used as the fundamental data for the institutional evaluation and as a reference for similar future programs. During all the stages, feedback should be readily available to the teacher and learners, so they can understand what adjustments they must make throughout the course. D. Ways in Collecting Data Within a language program, there are numerous factors which need to be considered and studied in order to get an objective view of the teaching process (Braskamp et al., 1984). These factors may be classified into four fields and are self-explanatory (See Figure 2). Ways of collecting data are varied. In our own teaching practice, we used questionnaires and classroom discussions. During the classroom discussions, the teacher sat among the learners with a class monitor chairing the sessions. Both teacher and learners made comments about the classroom activities during a certain period. Frank criticisms and friendly suggestions not only provided direct and objective views on the teaching/learning process, they also created closer relations between the learners and the teacher. Both received

10

immediate feedback from these sessions, allowing them to adjust their strategies. The feedback on how the teacher teaches and how learners learn could be shared with the supervisor and the learners. Besides direct communication, the teacher can be also use questionnaires for some sensitive questions. Figure 3 is an example of a questionnaire adopted from Grammar Practice Activities (Ur 1988). Learners completed these questionnaires anonymously. After the questionnaires were collected, the teacher statistically rated the value of each question in order to get the learners opinions on the teaching. The results of the questionnaire showed that the learners wanted the teacher to use the blackboard more often, wanted more opportunities to speak in English, and wanted the teachers help in their oral and listening practices. All these data helped improve the teaching and as a result, benefited the learners. In addition to these two methods of collecting data, we found program record-keeping, class observations, and personal interviews effective means of data collection. All these results should then be entered into a database for study and evaluation. By comparing these data, the teacher is able to monitor the progress of the program, and the learners also know where the program is going. E. Feedback and Benefits Ongoing evaluations should never be a means of punishment for either the teacher or learners. Instead, it should help both parties in their practices. Teachers and learners should have positive attitudes and consider the evaluation process as part of the program. Teachers may not only pay attention to fulfilling the text objectives, but they may also learn about the students learning styles and strategies. On the other hand, the learners become aware of how the teacher teaches and why certain methods are used. By mutual observations and the exchange of ideas, both help improve the learning process.

11

By using the computer in the evaluation process, the teacher and learners get instant feedback, thereby allowing them to adjust the teaching and learning to suit the requirements of the program. The learners can also benefit by learning about other learning strategies. Thus, because teachers become aware of the learners styles, they are able to adjust their teaching methods to better serve the learners needs. F. Conclusion An ongoing evaluation is a systematic review of a language program. By changing the roles of teacher and learners in a program, the teaching and learning can improve. The process should include teachers and learners and various kinds of activities at the different stages of evaluating. The evaluation also benefits the teacher who becomes a researcher and makes research a part of the classroom activity. Finally, supervisors of the program can learn more about the program, making it easier for them to make decisions when setting up future programs. Most importantly, an ongoing evaluation benefits everyone involved by helping to maximize communication between the teacher and learners.

REFERENCES

12

Braskamp, L., D. Branderbug and J. Ory. 1984. Evaluating teaching effectiveness: A practical guide. Saga Publications Inc.

Carry, J. and M. Dauber. 1995. Management education: An approach to improved English language teaching. ELT Journal, 49, 2, pp. 3743.

Gasper, D. 1995. The social science encyclopedia. eds. A. Kuper and J. Kuper. Routledge and Kegan Paul. Chinese Translation published by Shanghai Translation Publishing House.

Hutchinson, T. and A. Waters. 1987. English for specific purposes: A learningcentered approach. Cambridge: Cambridge University Press.

Mackay, R., S. Wellesley, and E. Bazergan. 1995. Participatory evaluation. ELT Journal, 49, 4, pp. 308317.

Marrow, K. and M. Scholker. 1993. Process evaluation in an INSET course. ELT Journal, 47, 1, pp. 4755.

McDonough, J. and S. McDonough. 1990. What is the use of research? ELT Journal, 44, 2, pp. 102109.

Nation, I.S.P. and Macalister, John. 2010. Language Curriculum Design. New York: Routledge.

Pilliner, A.E.G. 1968. Evaluation. Article Language Testing. Rea-Dickins, P & Germaine, K. 1992. Evaluation. Cambridge University Press.

13

Richards, J. and C. Lockhart. 1995. Reflective teaching in second language classrooms. Cambridge Language Education Series. Cambridge University Press.

Williams, M. and R. Burnden. 1994. The role of evaluation in ELT project design. ELT Journal 48, 1, pp. 2227.

Ur, P. 1988. Grammar Practice Activities. Cambridge: Cambridge University Press.

14

Appendices

15

Figure 1
Stages in Evaluating The process is divided into four stages with four levels of evaluation Evaluation and Function and doer Explanation 1. Collect basic data 2. analyze Pre-evaluation: syllabus and and Preparation (done by the textbooks teacher) 3. study and plan for the reference materials program Stage 1. study procedure Informal 2. compare the evaluation goal and (conducted by achievements the teachers and 3. analyze the learners) strategies of both teaching and learning 1. measure achievements 2. analyze the results and strategies 3. plan for the remainder of the program 4. analyze and compare the data and other reference documents 1. review program 2. study the process of the program 3. prepare for Data and Result Document 1. learners' basic data 2. program documents teaching plan 3. teaching in detail materials 4. reference materials from similar programs 1. notes and comments from both teacher and learners reports and 2. exercises and immediate quizzes both feedback teaching and learning 3. interviews and discussions

Ongoing Program

Formal (or periodical) evaluation (done by the teacher and supervisor)

1. data of informal evaluation with reports and statistical teaching plan analysis for next stage 2. tests and comments 3. teaching notes

Postprogram

Summary (evaluation done by the teacher and supervisor)

1. test scores or other results 2. class papers or summary and learners' report comments 3. teacher

16

authoritative evaluation

reports, plans and teaching notes 4. analyzed data

17

Figure 2
Factor fields Items 1. program plan and syllabus 2. staff and specific course teaching plan 3. evaluation results of prior programs and teaching materials 1. basic data such as name, age, sex, etc. 2. language tests and other records on the proficiencies 3. learning strategies and other psychological factors involved 4. attitudes toward the program 1. teaching notes and class activity records 2. exercises and comments 3. comments on class activities by both teacher and learners 1. scores and other achievement records 2. summaries and evaluation reports

of organization

of learner

of teaching and learning

of the result

18

Figure 3