573 1466 1 SM

Adair-Hauck, Willingham-McLain, and Youngs
Evaluating the Integration of Technology and Second Language Learning

Bonnie Adair-Hauck, Ph.D.
University of Pittsburgh
Laurel Willingham-McLain, Ph.D.

Duquesne University
Bonnie Earnest Youngs, Ph.D.

Carnegie Mellon University
ABSTRACT This article reports the findings of a program evaluation project that assessed the integration of technology-enhanced language learning (TELL) into a second semester, college-level French course. Thirty-three French II students participated in this study. Students in the treatment group met with the instructor three days per week and, for the fourth class, they participated in TELL activities. The control group met with the instructor four times per week. Both groups had the same instructor, textbook, and ancillary materials. The article reports on student performance in French for listening, speaking, reading, and writing skills, and on their cultural knowledge. Findings indicate that the students in the treatment group performed equally well as the control group in listening and speaking and better on reading and writing achievement measures. The study also includes findings regarding student motivation, anxiety, and perceptions on meeting the language learning goals students set for themselves. The results may be interpreted that it is both feasible and desirable to integrate, in principled ways, TELL activities into the language learning curriculum. KEYWORDS Technology-Enhanced Language Learning, Multimedia Curriculum, Integration of Technology, Program Evaluation, College-Level French Teaching, Computer-Assisted Language Learning, Computer-Mediated Writing
1999 CALICO Journal
Volume 17 Number 2
269
Evaluating the Integration of Technology

INTRODUCTION
Background
Although the use of personal computers has proliferated into the workplace as well as in academe, there is a paucity of published research on computer-assisted language learning (CALL). Conrad (1996) reviewed several professional journals (Modern Language Journal; Foreign Language Annals; Unterrichtproxis; French Review; Hispania; and System) published from 1992-1995 and reports that only 1.4% of the articles published addressed empirical CALL studies. The majority of CALL articles are practical in nature and showcase a software program or suggest guidelines for the implementation of software. The few empirical CALL studies published have focused on topics such as feedback (Brandl, 1995; Nagata, 1993; Nagata & Swisher, 1995; Robinson, 1989, 1991); pronunciation and intonation training (Stenson et al., 1992); interactive reading (Svenconis & Kerst, 1995); computer-facilitated student interaction and talk (Beauvois, 1992; Chun, 1994; Cononelos and Olivia, 1993; Meunier, 1994); writing (Armstrong & Yetter-Vassot, 1994; Barnett, 1989; Greenia, 1992); and cognitive style (Liu & Reed, 1994; Raschio, 1990). Previous studies on technology and language learning have compared the effectiveness of a new technology with more traditional modes of learning. In general, these studies have examined learning over a short time period such as a few days. For example, Raschio (1990) explored the cognitive style (field dependent or field independent) of 62 first semester Spanish students. The goal of both the control and experimental groups was to learn how to form Spanish direct and indirect object pronouns over a two day period; day three was reserved for testing. The results did not reveal any statistically significant relationships between the level of field dependence, mode of instruction (printed vs. CALL), and student achievement. Although the study did unveil some interesting findings regarding student attitudes, Raschio suggests that the profession needs to modify its traditional research framework to include process variables, not just outcomes in second language learning. Besides a lack of research-oriented CALL articles, few published evaluations of second language (L2) programs have integrated CALL into the curriculum. Johnson (1985) conducted a study on computer-assisted learning to promote L2 acquisition, but her report focused primarily on such issues as equal access, software development, computers in composition, typical practices, and model programs. There appears to be no published article on language program evaluation intended to assess language skill development and the integration of technology into the curriculum.1 At the beginning of this decade, Garrett (1991) articulated a number of research questions regarding program development and technology, What
270 CALICO Journal

kind of software, integrated how and into what kind of syllabus, at what level of language learning, for what kind of language learners, is likely to be effective for what specific purposes? Herron and Moos (1993) reiterated the aforementioned concerns by stating that One of the major concerns facing the foreign language teaching and literature profession is how to integrate new technological advances into instruction. This article reports the findings of a longitudinal program evaluation project that focused on L2 skill development (listening, speaking, reading, writing, and culture) and the integration of technology-enhanced language learning (TELL). These findings can serve to inform foreign language departments as they make important decisions concerning the use of technology in their programs.
Innovations in Technology Application
In the 1980s, CALL software featured a behavioral or stimulus-response approach to language learning by stressing mastery of grammatical properties or discrete-point learning (Conrad, 1996; Johnson, 1992; Pusack & Otto, 1997). Much of the software was grounded in a transmission approach to learning. For example, Johnson (1992) reviewed the ESL software for language minority children and found that in general these programs focused on grammatical forms, many of which were not that particularly useful. Recently, tremendous progress has been made in technology, and the profession has broadened its interest in the role of technology to encompass a multimedia approach by including computers with CD-ROM drives, videodisc players, video players, and hypermedia capabilities. In this respect, the preferable use of multimedia has evolved from an add on component (CALL) to a fully integrated feature of the foreign language curriculum (TELL). Pusack and Otto (1997) advocate that the strength of multimedia is the synergy derived from presenting content using a variety of modalities (listening, reading, writing, and speaking) that are linked together in meaningful ways [italics added] to provide an in-depth experience. Multimedia also has the advantage of providing students with longer stretches of oral and written discourse embedded within a rich visual cultural context for communication (Garrett, 1991). Technological advances, such as the World Wide Web, connect our students to up-to-date authentic realia and to native speakers. Multimedia thus has the capability to stretch our curriculum beyond the traditional walls of the classroom and to integrate much needed sociolinguistic authenticity into our programs (Meunier, 1994). Besides expanding its orbit of interest to include a multimedia approach, the profession has also redefined the role of technology in L2 learning.
Volume 17 Number 2 271

Similar to the reoriented role of the teacher from central to supportive in the language learning process, we have witnessed a metamorphosis in the role of the computer from transmitter of knowledge and skills to a tool that supports and assists learners to complete tasks (DeVillar & Faltis, 1991). The profession now realizes that a computer is a medium for learning and not a method for L2 instruction. Although there are numerous benefits of integrating technology into the curriculum, we will address three specific areas within the context of our program evaluation: (a) computer-mediated writing, (b) the integration of culture, and (c) the sociocultural benefits of technology.
Computer-Mediated Writing
The research concerning computer-assisted composing and the development of L2 writing has produced diverse results. While some studies indicate that students spend more time and write more at the computer (Hawisher, 1989; Roblyer et al., 1988), some writers tend to plan less and give less attention to their ideas or content (Haas, 1989; Theismeyer, 1989). Costanzo (1994) stresses that newer word-processing software incorporates a process-oriented approach to writing as a recursive process of discovery, elaboration, and revision. Bernhardt et al. (1989), Phinney (1988) and Rodrigues (1985) have stressed that many of the studies that report on computer-mediated writing have been short-term studies. Phinney (1991) argues that at least one semester is needed for positive improvement in students writing to occur. Phinney (1989) also notes the important role of the teacher to serve as a guide in assisting learners on how to use computers in the writing process, Without specific instruction in using the computer to facilitate the writing process, from prewriting to revision, the computer alone, appears to have little effect in changing the writing behavior of naive writers. Studies of student motivation and attitude toward writing have produced more positive results (Daiute 1984; Neu & Scarcella; 1991). For example, Phinney (1991) found that computer composing reduced blocking problems and improved the students overall attitude toward writing in English. Phinney noted, however, the importance of the teachers role in providing writing activities during regular class time which stressed brainstorming, drafting exercises, peer-commenting, and revising of drafts. Studies investigating computer networks used in L1 and L2 classes underscore that these networks encourage more written discussion and therefore more involvement on the part of the learner (Bump, 1990; DiMatteo, 1990; Kelm, 1992).
272
CALICO Journal

Integrating Culture through Technology
Multimedia capabilities such as audio/videotapes, CD-ROMs, videodiscs, the World Wide Web, E-mail exchanges, and electronic conferencing, enable the profession to incorporate much needed sociolinguistic authenticity into the L2 classroom (Meunier, 1994). Lafford and Lafford (1997) explain that access to multimedia capabilities facilitates the learners understanding of the various social and psychological forces at work today in the target culture and provides a context in which students can interpret the behavior of the target culture inhabitants. These authentic materials can serve as the tools to frame meaningful, communicative classroom-based discussions. Furthermore, the use of video exposes students to native speakers using appropriate interactional communication strategies including nonverbal or kinesthetic behaviors.
Language Learning as a Social Phenomenon
Since language is a social phenomenon, language learning occurs through social interaction involving teachers and more capable peers (Tharp & Gallimore, 1988; Vygotsky, 1978; Wertsch, 1979). Similarly, multimedia is now seen as a tool to enhance communication and discovery-oriented learning, much of which can be accomplished by working cooperatively in small groups. In this way, multimedia is a tool which assists the learners as they work in their zones of proximal development (Vygotsky, 1978). Cooperation with a teacher, more capable peers, or with multimedia allows students to perform tasks that they would be incapable of performing on their own. Generally, these shared activities need to be followed by individual practice on a similar task until learners can perform the task independently (Cummins, 1991). Multimedia can thus assist learners to move from the interpsychological to the intrapsychological level (Vygotsky, 1978). Johnson (1991) found that integrating computers into the classroom can promote cooperative learning. Rather than isolating and promoting asocial behavior, as many had feared, there is a growing body of evidence that computer use can promote new ways of working together, productive peer teaching, as well as high quality social and academic task-based interaction, and that these kinds of interaction are related to higher levels of interest, motivation and achievement. The teacher, of course, plays a critical role in selecting computer software that will influence the sociocultural dynamic of cooperative group work (Johnson, 1992; Mydlarski, 1987; Piper, 1986).

Foreign Language Program Evaluation
Brown (1989, 1995) defines language program evaluation as the systematic collection and analysis of information necessary to improve a curriculum, assess its effectiveness and efficiency, and determine participants attitudes within the context of a particular institution. Brown (1989) continues by saying that formative evaluation occurs during the development of a program; the purpose of gathering information is to improve the program. Summative evaluation occurs at the end of the program, and the purpose of gathering information is to determine whether the program was successful. Various evaluation models exist as well. A participatory model of evaluation can be preferable to an outside expert model because the latter often encourages adversarial roles between the expert and the teacher. Consequently, Alderson and Scott (1992) recommend a collaborative or participatory model which includes a consultant who provides expertise and guidance along with the teacher(s) of the program. A participatory evaluation model centers on insiders though benefitting from the advice of outsiders. Ross (1992) concurs with the notion of a collaborative or participatory model for it decreases the teachers anxiety of being watched and allows direct participation by the teacher as both practitioner and observer. In addition, Beretta (1986) summarizes the strengths of two frameworks, field research and laboratory research, and argues for more field research in language program evaluation. Beretta notes that field research is longterm, classroom-based inquiry into the effects of a program and that laboratory research is short-term and only involves the testing of individual components of a theory in an environment in which extraneous variables are artificially held constant. A further consideration is the timing of the evaluation: during or after the program or both and for how long? Brown (1995) examined language evaluation programs between 1988 and 1994. Thirteen of the 16 language evaluations were longitudinal in nature and ranged from two months to three and a half years. These evaluations were conducted during the programs. Evaluations can also occur at the end of a program. For example, Snow and Brintons (1988) evaluation used a retrospective format after students had completed a content-based program. Pitiyanuwat (1986) notes that the profession needs to gather data from students after they have finished a program and actually have to use the language. Brown (1995) suggests that the best framework would include evaluation during the program, immediately after, and later in a follow-up phase. With regard to data collection, Alderson and Scott (1992) point out that quantitative data are both easier to gather, and more amenable to analysis and summary. However, a number of researchers (Beretta, 1986;
274 CALICO Journal

Donato, Antonek, & Tucker, 1996; Lynch, 1990; Ross, 1992) stress the need to provide multiple perspectives by triangulating data. As Ross (1992) argues, both qualitative and quantitative types of evaluation can be utilized with a view to defining a program, and both may provide alternative views of the same classroom phenomena. Finally, Brown (1995) defines the evaluation of product as any evaluation which is focused on whether the goals (product) of the program were achieved, as contrasted to process evaluation which centers on what is going on in the program (process) that helps to arrive at those goals (product). Long (1984) argues for doing both when he states using process and product evaluations in combination, one can then determine not only whether a program really works, or works better, but if so, why, and if not, why not?
THE PROJECT
Project Rationale and History
We conducted a semester-long evaluation of the integration of TELL into a second semester French class. We chose to examine the impact of replacing one class per week with multimedia activities for four reasons: (a) the paucity of empirical research on the effects of TELL over time, (b) the mandate from our university to make learning remote in time and space, (c) the difficulty students had enrolling in four-day-a-week courses, and (d) the success we had had with computerized grammar and vocabulary exercises in our self-paced courses. Informal observation indicated that even though the self-paced students had minimal contact with a professor and other students, many of them excelled in their writing. Some students mentioned using Dasher2 exercises as a way to prepare for written portions of the course exams. Endeavoring to be thorough in our program evaluation, we drew on the strengths of multiple evaluative procedures. Although our evaluation was primarily summative in that we sought to establish whether the TELL curriculum was successful in meeting our goals, we also conducted formative evaluations in order to improve continually the TELL components. According to a participatory model, we did field research in a classroom context throughout the entire semester. We collected both quantitative and qualitative data in order to assess both the process and product of language learning.
Volume 17 Number 2
275

THE PILOT STUDY
During our spring 1996 pilot study, students in one section of secondsemester French at Carnegie Mellon University (CMU) participated in the following multimedia activities in lieu of the fourth contact hour: computerized grammar and vocabulary exercises using Dasher, instructional videos, and optional use of an on-line spell-checker and French-English glossary. A control group followed the traditional syllabus, meeting four times per week. Qualitative and quantitative data on student performance, anxiety, and motivation were gathered to determine whether segments of language courses could be reasonably expected to become remote in space and time and still ensure that students develop an appropriate level of proficiency in all skill areas. We found parity on the measures of anxiety, motivation, listening, reading, and writing. Although both groups showed significant improvement on the posttest of cultural understanding, the treatment group showed significantly more improvement than the control group (p = .019). Furthermore, there was a strong trend toward significance (p = .066) of the treatment group scoring better on the semester-end speaking test. Given these encouraging results, we revised the course and the study design to answer more research questions. Specifically, whereas Dasher exercises had been the primary focus of the pilot study, the study currently being reported incorporated a rotation of multimedia-based reading, grammar, vocabulary, and listening exercises with systematic speaking and writing follow-up activities.
Research Questions
The primary purpose of our study was to investigate the feasibility and desirability of replacing the fourth contact hour with multimedia activities outside the classroom. Our evaluation of this program was intended to answer the following questions: 1. Would the TELL experimental group students perform as well as the control group in listening, speaking, reading, writing, and cultural understanding? 2. How would TELL affect student foreign language motivation and anxiety? 3. How would TELL affect students perception of meeting the language learning goals they set? 4. In what ways would TELL modify the roles of the teacher and students? 5. More generally, would the TELL course be an effective curricular design for achieving second language development in this context?
276 CALICO Journal

Design of the Study DESCRIPTION OF STUDENTS
We began the study by administering a demographic and language background questionnaire in order to determine whether the treatment group (n = 17) and control group (n = 16) were similar. There were no significant differences between the groups in terms of sex, age, grade point average and year in college. There were 5 women and 12 men in the treatment group, and 7 women and 9 men in the control group. The students ranged in age from 19 to 33, but most were of traditional college age. Students had declared majors in five of the seven colleges at CMU, representing the humanities, social sciences, natural sciences, technology, fine arts, and business. Despite CMUs reputation for technology, 21 of these 33 students were majoring in the humanities, social sciences, or fine arts. Every student was familiar with computers; 21 owned a computer, and only one student in the treatment group reported not being comfortable using a computer. Although the majority of the students in both groups were American by birth and native speakers of English, many students reported knowing other languages. Out of the total number of students, only two from the control group reported that most of their experience in French had been outside the classroom. Half of the students in each group had taken the first semester French course with the researcher/instructor the previous semester. This facilitated conversation among the students, since many already knew each other and were familiar with the teaching style and general course requirements.
COURSE DESIGN
We conducted the research over the course of the spring 1997 semester in a second semester French course, which used chapters 6-10 of Allonsy (Bragger & Rice, 1996). The treatment group met three days a week from 10:30 to 11:20, and, in lieu of the fourth class period, students chose when and where to do the multimedia components. The control group met four days a week, from 12:30 to 1:20, and prepared the same multimedia components that were assigned to the treatment group, but in class, on paper, or by watching the video as a group. Both groups were assigned identical writing homework and follow-up speaking tasks. The control group completed these tasks in class and for homework. The treatment group prepared the assignments on their own time. For both groups, the speaking tasks were evaluated in class, and the writing tasks were collected and graded. The challenge was to incorporate systematically the four language skills and cultural understanding into the students work on a weekly basis. We developed a 12-week rotation of three TELL components that were familVolume 17 Number 2 277

iar and had been successful in past courses: the CMU French Reader, Dasher exercises, and the Allons-y video.3 In addition, we had already tested the materials sufficiently to know that after a brief orientation, students were unlikely to encounter technical difficulties. We also chose these components because they are accessible to many instructors interested in implementing them in a traditional classroom setting. The CMU French Reader, a computerized reading program featuring authentic journalistic texts, pre- and postreading questions, a glossary (French-English, English-French), culture notes, grammar notes, and a writing pad was created by Bonnie Youngs, the course instructor, and Christopher M. Jones (1996). This HyperCard-based program is housed on the Modern Languages server and is accessible on any Macintosh computer on campus. We used the Reader to introduce each Allons-y chapter through a text relevant to the chapter theme (e.g., food, lodging). The second multimedia component in the cycle consisted of Dasher exercises developed by CMU faculty. Dasher (Pusack & Otto, 1992) is an authoring program which permits an instructor to write exercises of various types (e.g., multiple-choice, fill in the blank, sentence completion, listening comprehension, written and visual-based text). For this project, we assigned Dasher exercises that emphasized the grammar and vocabulary of each chapter. The third TELL component was the Allons-y videocassette (Heinle & Heinle, 1996). The video presents brief scenarios for each chapter performed by native speakers in France and Guadeloupe speaking at natural speeds. Students had previewing questions and listening comprehension exercises to do while watching the video. We offered the video last in the cycle because it provided a synthesis of the chapter content. For each of the 12 weeks, students in the treatment group received a lab assignment sheet outlining (a) their reading, grammar/vocabulary, or listening comprehension task and (b) their weekly writing homework and follow-up speaking task (to be prepared for the next class period). (See the Lab Assignment Sheet in Appendix A.) The writing and speaking activities provided a way for students to apply and develop what they had learned through their own personal expression in simulated authentic tasks. The lab assignment sheet also contained a checklist of the tools students used while doing the lab, for example, their textbook, paper dictionary, electronic glossary, and spellchecker. The English-French/French-English electronic glossary (Carnegie Mellon University, 1996) is integrated into the CMU French Reader, but also can be accessed as a stand-alone program. The Spellchecker is a foreign language spell checking program. In addition, for each weeks lab, we asked students to indicate the amount of time they had spent, to rate the lab on a scale of 1 poor to 5 great, to explain their rating, and to describe in a few words how they felt while doing the assignment.
278 CALICO Journal

RESEARCH
DESIGN
In order to test our primary hypothesis that students using the TELL components in lieu of one class period per week would perform French tasks in all the skills as well as students meeting four times a week, we collected a variety of qualitative and quantitative data early, throughout, and near the end of the semester (see Table 1). Table 1 Overview of Data Collected from Treatment and Control Groups
Foreign Language Motivationa Foreign Language Anxietyb Goal Setting/Attainment & Affective Questionnairec Cultural Knowledge Listening Speaking Reading Writing Early-semester data 24-item measure 20-item measure open-ended questionnaire Quiz 5 achievement tests Paired Test 5 achievement tests Writing on first lab 5 achievement tests and on first weekly lab followachievement test d up activities 12 weekly labs (treatment group only) 12 weekly labs (treatment group only) Paired Test Writing on last lab and last achievement test During-semester data End-of-semester data 24-item measure 20-item measure open-ended questionnaire Quiz
Time on Task
Students Perspectives
Open-ended questionnairee Follow-up interviews (selected students, treatment group only)
The Foreign Language Motivation measure was adapted from Tremblay and Gardner (1995; personal communication, November 1995). b The Foreign Language Classroom Anxiety Scale was adapted from Horwitz and Young (1991). c The Goal Setting and Affective Questionnaire was adapted from Chamot (personal communication, November 1995). We also collected information about learning styles using Reids Perceptual Learning Style Preferences Questionnaire (1987), but these data were not conclusive. d Both groups had the same weekly homework writing assignments and test items. An independent rater used an adapted form of the ESL Composition Profile (Glisan, 1981) to rate early and late semester writing samples. Note that the first and last writing test items are also reported as part of the MANOVA analysis of the achievement tests from throughout the semester. e Students perceptions of improvement in listening, speaking, reading, writing, cultural knowledge and self-confidence were gathered on an end-of-semester open-ended questionnaire. (See end-of-semester questionnaire in Appendix D.)
Volume 17 Number 2
279

As noted earlier, we first established that there were no significant demographic differences between the two groups and could thus compare the language performance of the two groups using t-tests and MANOVA without having to adjust for initial differences between them. Recognizing that the instructor and students were there first and foremost for doing a second semester French course, and only secondarily for carrying out our study, we balanced the number of data collection instruments with the need for time-on-task learning French. Instead of viewing the classroom context as a contaminated research environment, we chose to describe, in the ethnographic tradition, as many of the variables as possible so that readers could themselves interpret our data and determine the relevance (or generalizability) to their own instructional contexts. In order to support each finding with evidence from different sources, we collected scores on pre- and posttests, grades on classroom chapter tests, self-reports from the students and instructor, interview data from a sample of students, student evaluations of each lab assignment, check-lists of materials used, and records of time-on-task. Our research team consisted of the instructor/researcher who was largely responsible for developing the curriculum and the TELL components and two researchers who designed the study and conducted most of the data analysis. Two student researchers observed the classes, summarized much of the qualitative data, and coded the quantitative data. An independent rater analyzed the writing samples.
Findings EVALUATING LEARNER MOTIVATION AND

ANXIETY
In addition to demographic background, we examined whether the groups differed on measures of foreign language motivation and anxiety. Both measures consisted of statements answered using a Likert scale ranging from 1 strongly agree to 5 strongly disagree. We adapted 18 items about students motivation to learn French from Tremblay and Gardners questionnaire designed for students of French in Canada (1995; personal communication, November 1995). These items reflected various aspects of their expanded construct of language learning motivation (e.g., desire to learn French, motivational intensity, persistence, and instrumental and integrative orientations). We then added six technology related questions such as The thought of using various types of technology to learn French sounds interesting, and I would rather spend my time on activities that do not require the use of technology, to determine whether there were differences between the two groups in their motivation to use technology as a means of learning French.
280 CALICO Journal

To examine language-learning anxiety we chose items from Horwitz and Youngs (1991) Foreign Language Classroom Anxiety Scale and once again added questions about technology and language learning such as Doing French exercises on the computer sounds like fun, and The thought of using various types of educational technology to learn French scares me. There proved to be no significant difference in motivation or anxiety between the groups either on the entire set of items, or on the technology clusters alone, at the beginning or end of the semester. Furthermore, there was no significant change over the semester in motivation or anxiety. On the final questionnaire, one student explained how TELL helped to lower her level of anxiety, Last semester, when we did the video as a whole class, I didnt have a clue what was going on. But this semester, I could do it by myself, or with a friend, and it became so much easier. Now, Im not afraid of the listening sections for our tests. I used to be so scared of those.
EVALUATING LEARNERS ATTAINMENT OF GOALS
Very much linked to students motivation to learn French and their language learning performance is the setting of specific goals (Tremblay & Gardner, 1995).4 Therefore, using a simple open-ended questionnaire, we asked students to state their personal goals for this course: Why are you taking elementary French II? Please state your projected goals for this course. Be as specific as possible. Near the end of the semester, we asked, This is the end of the semester in elementary French II. Do you believe that you have made progress toward your projected goals for the course? The goals students reported were similar for both groups. The majority (treatment group, 10 of 17 and control, 11 of 16) had linguistic goals such as gaining fluency, especially in spoken French. Only one or two in each group stated that they were taking French because it fulfilled a general education requirement. (There is no foreign language requirement at CMU.) Other goals included learning about French/Francophone culture, traveling, using French in their careers, and living abroad. At the end of the semester, 14 of the 17 students in the treatment group reported that they had met their goals, two said they almost had, and one said he or she had not met the projected goals. In the control group, 11 of the 15 students who filled out the final goals questionnaire stated they had made clear progress, and four students had made some progress, but not as much as they had hoped for. Treatment group students reported, I am more confident and not afraid to speak, Im on my way to being fluent, I have sharpened my French skills, I am more confident in my reading, writing, speaking skills, I have almost met my goals. Control

group students reported, Yes, because my vocabulary has increased, Im on my way to being fluent, I built a foundation for learning the language, I have made progress but would have liked to have made more, Not as much as I would have liked. More students in the treatment group were positive about having made satisfactory progress toward their projected goals. On the final questionnaire, students reported that having the flexibility to individualize the use of the multimedia activities played an important role in their progress and development of certain language skills: One of the great things about the multimedia activities is that you can spend as much time on them as you want, and I liked the flexibility of the multimedia activities, so that if you feel that you need more practice, you can do it again and again. The multimedia activities challenged students to take greater responsibility in working toward their goals for learning French.
EVALUATING KNOWLEDGE OF CULTURE
We designed a culture quiz to diagnose knowledge of both general facts about the Francophone world, and specific facts about topics treated that semester (e.g., university studies, shopping, lodging). (See the culture quiz in Appendix B.) The treatment group significantly outperformed the control group on both the prequiz and postquiz, with average scores of 68.4% and 76.0% versus 57.0% and 61.9% respectively (see Table 2).
282
CALICO Journal

Table 2 Cultural Knowledge, Speaking, and Writinga Early-semester data Mean SD Cultural Knowledge Treatment (n = 17) Control (n = 16) Speaking Test Treatment (n = 17) Control (n = 16) Test Writing Treatment (n = 17) Control (n = 16) Homework Writing Treatment (n = 14) Control (n = 12c)
a
End-of-semester data Mean SD 76.0 61.9 p < .005b 86.1 91.1 p < .05b 18.94 18.38 n.s. 17.93 16.92 n.s.d 12.9 13.3
68.4 57.0 p < .05 80.8 78.9 n.s. 17.41 17.31 n.s. 15.21 18.25 p < .05
13.5 11.0
7.8 13.8
6.2 7.2
2.06 2.36
2.03 2.09
3.29 2.74
2.30 4.10
We used t-tests to examine differences in mean scores between the treatment and control groups. b When change over time was taken into consideration, there was no significant difference between the treatment and control groups in cultural understanding and speaking. c Whereas all the students took the exams, several students did not turn in their homework. d There was a significant difference between the groups in the way they changed over time; the control groups homework writing scores decreased, and the treatment groups writing scores increased.
However, there was no significant difference between the two groups in average gain made over the semester (computed as postquiz score minus prequiz score).
EVALUATING SPEAKING
SKILLS
Early and late in the semester, students took an audio-recorded, paired speaking test administered by the instructor and then rated blindly by another member of our research team. The speaking test focused on functional use of language and meaning-making and consisted of three parts: picture description, a role play, and past narration cued by a series of pictures. (See the oral test in Appendix C.) The early and late tests were similar in format, but the tasks were somewhat altered such that students

had to use the same skills in new contexts. On the pretest, there were no significant differences between the average scores of the treatment and control groups (80.8% vs. 78.9%). Although the control group performed significantly higher on the posttest (treatment group 86.1% and control group 91.1%), there was no significant difference between the groups in gain made over the semester, computed as posttest score minus pretest score (see Table 2).
EVALUATING LISTENING, READING, AND WRITING
Throughout the semester, students took five achievement tests, each one testing listening, reading and writing, all at a paragraph length discourse level. These tests were designed and graded by the instructor.5 We used a Multivariate Analysis of Variance (MANOVA) to examine the difference between the groups performance on these tests in each skill. In listening, there was no significant difference between the two groups (see Table 3).
284
CALICO Journal

Table 3 Listening, Reading, and Writing Testsa Listening Reading Mean SD Mean SD Test I Treatment 83.7 6.9 91.7 7.7 Control 74.3 15.0 78.4 11.1 Test II Treatment Control Test III Treatment Control Test IV Treatment Control Test V Treatment Control 90.2 90.6 59.6 65.0 88.5 75.3 80.9 72.4 18.5 12.9 11.3 11.5 17.5 28.4 16.1 17.3 91.7 78.4 99.3 98.3 99.8 99.7 84.6 86.2 5.9 8.1 3.0 4.4 1.0 1.0 16.2 15.4
Writingb Mean SD 80.5 88.6 80.0 88.4 79.9 77.3 73.5 62.3 82.6 79.4 7.8 5.1 6.7 5.8 8.0 12.1 10.0 10.0 7.3 8.3
Note: For all tests, Treatment Group (n = 17) and Control Group (n = 16).
a
We used MANOVA to examine differences between mean scores for the treatment and control groups in listening, reading and writing. For listening, there was no significant difference between the groups. The treatment group scored significantly better than the control group on reading (Wilks Lambda = .41; F(5,26) = 7.62, p < .001) and writing (Wilks Lambda = .24; F(5,26) = 16.18, p < .001). b The writing on the first and last test was also analyzed by an independent rater using a different scale. See Table 2.
In reading, the treatment group performed significantly better than the control group (Wilks Lambda = .41; F(5,26) = 7.62, p. < .001). Similarly, in writing, the treatment group outperformed the control group (Wilks Lambda = .24; F(5,26) = 16.18, p. < .001). In order to further investigate the difference between the groups in writing, we had an independent rater analyze the writing on the first and last achievement tests and the first and last homework writing assignment (follow-ups to labs 1 and 12 for the treatment group and the analogous homework done by the control group).6 There was no significant difference between the group means on the first test, on the final test, nor in the amount of improvement shown by each group over time. In the writing homework, however, the control groups mean on the first assignment was significantly higher than that of the treatment group (treatment group mean = 15.21; control group mean = 18.25, p < .05).

The pattern of change from the first to the final homework assignment differed significantly for the two groups (p < .05). The treatment group scored higher on the final assignment than on the first, and the control groups scores decreased (experimental group mean = 17.93; control group mean = 16.92). Given that both groups performed equally well on the tests, clearly one group did not learn to write better than the other. Instead, it appears that at the end of the semester, treatment group students were more motivated than control group students to do quality writing on their homework, even though it counted for less of a grade than the tests. Indeed, the instructor had noted that the treatment group wrote longer and more complex homework compositions than the control group.
TIME ON TASK DOING THE TELL COMPONENTS
For each of the twelve weeks of TELL labs, the treatment group filled out a lab assignment sheet. (See the Lab Assignment Sheet in Appendix A.) Table 4 portrays the systematic cycling of the Reader , Dasher exercises and Allons-y video for each of the five chapters.
286
CALICO Journal
Volume 17 Number 2
287

In three instances, two of these components were combined into one week in order to fit the 15 labs into 12 weeks. Because the treatment group students were working outside of class, we asked them to report the amount of time they spent on the lab assignment so that we could compare their time on task with that of the control group, who used the majority of 50 minute class periods doing the in-class activities described earlier.7 With the exception of the second chapter (Chapter 7), on which the average amount of time spent was 44 minutes, the average amount of time spent on each of the three tasks per chapter did not vary more than 7 minutes from the overall mean of 34.6 minutes. The control group spent about 40 minutes per week on the in-class activities, analogous to what the treatment group did during their lab time. The control groups actual time on task per week is therefore very similar to the treatment groups averages. As the semester progressed, students were able to do increasingly complex tasks in approximately the same amount of time, that is, 28-37 minutes (see Table 5). Table 5 Lab Assignment: Average Time on Task and Ranking Average time on task Average ranking of task in minutes (range) on 1-5 scale (range) By task Reader 43 (97) 3.5 (3.2) Dasher 36 (55) 3.4 (3.4) Video 24 (41) 4.2 (2.0) By chapter Chapter 6 Chapter 7 Chapter 8 Chapter 9 Chapter 10 Overall 37 (62) 44 (123) 34 (63) 27 (35) 30 (38) 34 (64) 3.8 3.6 3.5 3.8 3.9 3.7 (3.3) (3.0) (3.0) (2.3) (2.7) (2.9)
Note that the range of time spent decreased from over an hour in the first three chapters to a little over a half hour in the final two chapters. The reason for this trend is unclear; students possibly learned to use the technology more efficiently or simply felt they had less time to spend on the labs as the semester progressed. In addition, the dramatic decrease in time spent doing Dasher exercises between lab assignments 7 and 9 no doubt resulted from our decision to turn off Dashers triple repetition of items answered wrong before giving students the correct answer. This repetition had proved frustrating to students and thus seemed to hamper rather than enhance learning.
288 CALICO Journal

STUDENTS EVALUATIONS OF THE TELL COMPONENTS
Over the semester, the average student ranking of the lab assignments by chapter was fairly constant, from 3.5 to 3.9 on a Likert scale of 1 poor to 5 great (see Table 5). The range in students assessments of lab assignments, however, decreased slightly. The waning novelty of the TELL components as well as the general semester-end anxiety college students experience was perhaps counterbalanced by their familiarity with and enjoyment of the lab components. Of the three component types, students preferred the video (4.2 ranking) to the Reader and Dasher (3.5 and 3.4 respectively). This preference was evident in the weekly lab reports, the semester-end questionnaire (see the final questionnaire in Appendix D), as well as the five follow-up interviews we conducted with treatment group students. In addition to spending the least amount of time on the video, students reported that they often viewed the tapes with a classmate. The students explained how their collaboration was beneficial, Yes, we collaborated, because we are both strong in different areas. Working together increased how much we got out of the lab assignments. Another student stated, With the videos, if I couldnt understand a part, then I was able to ask the other student if she knew. Between the both of us, we could work it out together. This spontaneous pair work increased their ability to understand these videos. Students also found the videos to be the richest source of cultural information. In spite of some students being annoyed by the precision required by Dasher, seven students reported that they liked Dasher. On the final questionnaire, they said, for example, that Dasher gave an opportunity to focus on the finer details of grammar, and that Dasher gave good practice by example. A few students critiqued Dashers identical treatment of all errors (e.g., grammar, spelling, accents). On the final questionnaire, students were most critical of the Reader. One or two students praised the readings for giving them first-hand experience in French and for being informative, but others found them tedious. The most substantial critique made by a few students on the semester-end questionnaire and in the follow-up interviews was that the current design of the Reader did not need to be electronic. It worked equally well with pencil and paper. If, however, the students chose to print out the Reader assignment, they were unable to access the glossary and the cultural and grammatical notes intended to inform their reading and help them develop their reading strategies. Worthy of note is the fact that the Reader was the one multimedia component that had not been piloted in second semester French, so it follows that changes in its design and implementation would be required.8 We asked the treatment group students to indicate whether they felt the

multimedia activities were effective in helping them develop the four language skills. Similarly, we asked the control group to assess the Tuesday in-class activities.9 More students in the treatment group indicated they thought they had developed their reading and writing skills than in the control group (15 vs. 12, and 16 vs. 11, respectively). The treatment groups perceived development of writing skills is corroborated by their higher test writing scores overall (see Table 3) and their improvement on homework writing (see Table 2). In both groups, 15 students reported that the lab or analogous in-class activities had helped them develop their listening skills. In ranking the helpfulness of language learning resources, the treatment students placed the video higher than did the control group, perhaps because they were able to listen to it as often as they liked in the lab, whereas the control group viewed the tape no more than twice in class and had no control over its use. When asked whether the multimedia activities were effective in helping them develop their speaking skills, 4 of the treatment group students said yes; 10 of the control group students said the Tuesday in-class activities had helped them do so. In their comments, treatment group students noted that although the three multimedia components did not require speaking, their speaking improved through vocabulary building and listening to the videos; It didnt really help my speech or accent but it did help my vocabulary, I dont know, I think it [speaking] ties in with listening a lot, and Videos were valuable for listening and speaking. In addition, when asked whether the follow-up speaking activities prepared for Wednesdays class were effective ways to synthesize and apply what they had learned in the lab, 9 of the treatment group said yes, and 5 somewhat. They commented, Yes, they reinforced the ideas of the lab well, and Yes, we were forced to use what we had just learned, making it stick. So, whereas they were not required to speak French during the three TELL components, students recognized that the components did help prepare them for the simulated authentic speaking classroom activities. Finally, the treatment group gave overwhelming support for the TELL 3-day-per-week French course as an option. Fifteen of 17 students said it should be offered, only two of whom suggested some modifications.
THE CHANGING
ROLES OF THE TEACHER AND STUDENTS
Given the results of this study and its supporting academic research base, it would seem appropriate to institute some form of TELL-based curriculum for foreign language courses. However, implementing a TELLbased curriculum may be a challenging task for teachers, administrators,
290 CALICO Journal

and students. Instructors may expect that using multimedia will decrease their workload and that multimedia curricula are more innovative, thereby ensuring the attention of students. Although the research appears promising, some fallacies regarding TELL contribute to its largely uncontrolled and unguided use. Its informed implementation requires constant evaluation of using technology to learn foreign language. Designing a TELL curriculum is a new task for most teachers but, like traditional curricula, requires a sound general pedagogy. In addition, it demands a sound instructional technology methodology. If an instructional context combines a modified traditional classroom setting with lab time, then the instructor must not only develop materials for learning a foreign language in the classroom but also implement technology to enhance the learning already gained in the classroom. This combination is difficult to manage and does not save the teacher any time in materials development and implementation. It is a new methodology that demands practice, planning, and versatility. Moreover, not meeting a class on a fourth day may appear to be a labor saver. On the contrary, in addition to the work of teaching in the traditional classroom setting, there is an extensive list of steps for the teacher who uses a lab or computers. These steps include integrating technologybased materials with traditional materials, mastering the implementation of technology-based materials, learning how to use the hardware, troubleshooting both hardware and software, orienting students to using the hardware and software, being available to students who have technological anxieties and difficulties, adapting technology materials that fail into a spur of the moment traditionally oriented lesson plan, and constantly assessing student learning and the classroom/lab environments to ensure continued student success. Nonetheless, some educators misuse TELL and view it as a panacea for current administrative issues. For example, at a large midwestern university, there have been significantly increasing enrollments permitted in beginning level foreign language classes and a drastically decreased amount of contact time with the instructor (Musumeci, 1998). Research has clearly shown that smaller class size is a critical factor in language development (Wiggins, 1997).10 Therefore, at every level, caution is required when implementing TELL. Like teachers, students also make personal adjustments in a TELL-based curriculum. In the early stages, they learn to use new hardware and software and to reschedule their homework periods around available lab or network time. Although students may be intrigued by the idea of missing class time in favor of lab time, they are now responsible for working outside the traditional classroom setting and for keeping pace with the requirements of their new curriculum.

Having adapted successfully to a TELL curriculum, however, students do find interesting ways to use their time away from the classroom. In this study, as noted above, students spontaneously engaged in collaborative learning situations, especially when using the video. Although they may have had some difficulties adjusting to a TELL-based curriculum, our students proved that given the tools and the time needed, they will take the initiative to become active language learners. Program evaluation, both formative/summative and process-/productoriented, is a vital aspect of building a TELL curriculum. The question is not should we use TELL? but how can we most effectively use it to enhance student learning? We must weigh all aspects of this curricular type before deciding on its implementation. Students must learn how to use it, and teachers must know their students and be aware that student input focuses the instructor on the best use of technology. In order to achieve a balanced TELL curriculum, teachers must assess the materials from the students perspective and understand how such a curriculum will influence student learning, not only cognitively but affectively.
Discussion
Let us summarize the findings by answering our research questions. 1. Will the TELL experimental group students perform as well as the control group in listening, speaking, reading, writing, and cultural understanding? Treatment group students performed as well as control group students in listening, and they outperformed them in reading and writing overall on the achievement tests (see Table 3). In writing homework, there was a significant difference between the groups in the way they changed over time; the control groups homework writing scores decreased, and the treatment groups writing scores increased (see Table 2). More treatment students than control group students indicated that they felt the TELL components (or analogous in-class activities) had helped them develop their writing skills. Fewer treatment group students perceived the labs as having helped them in their speaking, and they performed somewhat more poorly than the control group on the final speaking test. There was, however, no statistically significant difference between the groups when change over time on the pre- and postspeaking test was taken into consideration. On both the pre- and postquiz of cultural knowledge, the experimental group did better than the control group; they continued to score significantly higher, even though both groups improved considerably on this measure.
292 CALICO Journal

2. How does TELL affect student motivation and foreign language anxiety? Neither group changed significantly over the semester in their motivation to learn French or in their foreign language anxiety. It is possible that 15 weeks is insufficient time to effect a positive change in either of these language learning factors. 3. How does TELL affect students perception of meeting the goals they set? In the treatment group, 14 of 17, and in the control group 11 of 15 students reported having clearly made progress toward the goals (82% & 73%, respectively). 4. In what ways does TELL modify the roles of the teacher and students? The model of three class periods plus a lab assignment does not decrease the teachers work. Instead, the instructors energies are channeled in different directions such as evaluating, choosing, designing, adapting software, serving as consultant to students, assuring that the overall course learning objectives are being met, and that the course is an integrated whole. Students also take on a new role as they gain the freedom to work when and where they choose but also face the responsibility of doing considerably more work outside of class. 5. Is the TELL course an effective curricular design for achieving second language development? After carefully monitoring and assessing language skills in listening, speaking, reading, writing, and cultural knowledge, we recognize that the TELL course was indeed effective in helping our students learn French. Its strength in this particular instructional context appeared to be in writing, and its weakness in speaking. We are pleased that students were able to apply skills from the three componentsthe Reader, Dasher, and the videoto the writing of paragraphs and longer discourse. A few students mentioned that they improved in their speaking skills thanks to increased vocabulary and listening to the videos. We are confident that with a few modifications, such transfer to speaking in semi-authentic contexts can be increased. We need, however, to help students make the connection more clearly and to incorporate speaking opportunities directly into the multimedia components.
Volume 17 Number 2
293

CONCLUSION The following are the highlights of what we learned in this study: 1. The answer to our primary research question is that students in the TELL group performed as well as those in the control group on every skill and on cultural understanding according to our statistical measures. Students reported that they felt the multimedia tasks helped them in writing, reading, and listening, but less so in speaking. 2. It is indeed feasible and, for the majority of the TELL students, desirable to replace one day in four with TELL. Fifteen of 17 students favored offering the three class period plus TELL format as an option, and, despite tremendous semester-end pressures, students maintained their interest in the TELL componentsactivities which they had to complete on their own time. In addition, the instructor and researchers were satisfied with student performance as well as their attitude toward the curriculum. 3. Treatment group writing scores on the five achievements tests as a whole were significantly higher than those of the control group. The independently scored early and late-semester writing showed that treatment students improved in their homework writing and were motivated to write well even on the final homework assignment. 4. An unexpected consequence was that the TELL components promoted positive and spontaneously occurring collaboration among students outside the classroom. Let us conclude by noting some limitations of this study, followed by recommendations for continuing the study of TELL. 1. We are generally satisfied with the revisions we made to the program evaluation design after carrying out the pilot. It would, of course, be advantageous to have larger numbers of students or to collect data over even longer periods. Indeed, the Department of Modern Languages at CMU has extended the project to first and second semester classes. 2. The multimedia components that we used could be enhanced, a process constantly being done in the French program at CMU. a. The TELL component needs to integrate even more authentic, contextualized discourse. For example, there is room for considerable improvement in the grammar/vocabulary exercises which might entail using software that is more flexible than Dasher. In general, interaction among students and between students and authentic cultural documents in the target language could be increased.11
294 CALICO Journal

b. A further revision to the TELL treatment would be to improve the Reader; some students saw no need for the Reader, as it is currently designed, to be electronic. The Reader could, for example, include links to web sites, more linguistic support (the use of which would be required in follow-up assignments) and the removal of access to the glossary during the initial reading in order to promote sight-reading strategies. c. As described above, the speaking skill needs to be integrated more directly with the TELL component. We envisage numerous directions that this research of multimedia enhanced instruction could take. With respect to students and learning issues, the following questions need to be addressed: What is the role of students locus of control in where and when they use technology? (To be optimal in the college setting, it seems that students need access to all TELL components in their rooms at any time of the day.) What is the value of collaborative work among students using TELL outside the classroom? How could TELL be adapted to enhance learning by targeting the various multiple intelligences? How could TELL components be individualized for remediation of students with particularly weak language skills in language learning? And finally, can TELL be developed to track effectively the linguistic development and cultural understanding of students? Clearly, the examination of the integration of technology into second language teaching and learning is fertile ground for future research.
Volume 17 Number 2
295

APPENDIX A Lab Assignment Sheet le franais 102A/Youngs/Printemps 97 Ltudiant(e) Exercice de labo # 1 Date:
Fill out this checklist for each assignment you do using multimedia or language software.
Task Where? How long? Assessment Scale: 1 (poor) to 5 (great)
Check tools used Fre Glossary (elec.) Paper Dictionary Netscape Dasher ModLang Server LLRC computers Spellchecker Grammar reference Allons-y textbook Verb book Other? (please list)
Reader Chapt. 6 (server)
1-2-3-4-5
Additional information (you may be brief): 1. Please comment on why you chose the assessment (1-5) above.
2. How did you feel during this assignment? (upset, nervous, enthusiastic, interested, bored...)
3. Les Devoirs (essayez dutiliser le glossaire franais):
296
CALICO Journal

a. A lcrit: Quest-ce que vous faites pour vous prparer pour la rentre?
b. A loral: Prparez un dialogue entre vous et le vendeur ou la vendeuse chez la librairie. Achetez tout ce quil vous faut pour la rentre.
APPENDIX B Francophone Cultural Knowledge Quiz
Answer the questions in English, or give the French expression where asked. 1. Define Francophone. 2. Name five Francophone countries (not France). 3. Name five cities in France. 4. Name denominations of French and Canadian currency. 5. Name as many forms of public transportation in Paris as you can. 6. What can you use, besides coins, to make a phone call in France, and where would you go to get one? 7. a. Name a French political figure. b. Name a French film star. c. Name a French scientist.
Volume 17 Number 2
297
8. a.
What do you say if someone says merci to you?
b. How do you say youre feeling fine? 9. What are some things that French people typically have for breakfast? 10. What is a croque-monsieur? a. a drink b. an opera c. a sandwich d. a type of building 11. Label the order in which the following items are typically served during a French dinner: salad cheese coffee main dish dessert 12. What would you purchase at a charcuterie? 13. Which best describes a traditional French university. a. a complex of classroom buildings and sports facilities b. a sprawling suburban campus c. an urban campus, no green areas 14. Which field of study does lettres refer to? a. education b. fine arts c. humanities d. sciences 15. Where do most French university students live? a. in rented rooms in town b. in university dorms c. with their families 16. How are floors of a building numbered in France? 17. Whats a deux-pices? a. a coin b. an apartment c. a theater d. a vehicle 18. What does a green cross refer to? a. a church b. a pharmacy c. a political party 19. What are the following (city, region, country) and where are they located? What do you associate with each one? What? Where? Association? a. Guadeloupe b. Morocco c. Louisiana d. Normandy e. Bordeaux 20. How would you describe French people to someone from your country? What stereotypes do you have of French people? Why?
298
CALICO Journal

APPENDIX C Oral Test Grading Grid Picture description/comparison Vocabulary/content Structures Accomplishes the task Role play Vocabulary/content Structures Level of formality (polite) Accomplishes the task Telling a story in the past Vocabulary/content Structures Accomplishes the task Overall Communication strategies (asking for clarity, getting point across) Pronunciation/fluency (generally accurate pronunciation, not groping for words) TOTAL
5 5 5
4 4 4
3 3 3
2 2 2
1 1 1
0 0 0
/15
5 5 3 5
4 4
3 3 2 3
2 2
1 1 1 1
0 0 0 0 /18
5 5 5
4 4 4
3 3 3
2 2 2
1 1 1
0 0 0
/15
/6
/54
Volume 17 Number 2
299

APPENDIX D Final Questionnaire (for treatment group) 1. What did you like most about the multimedia activities (Dasher, reader, video)? 2. What did you like least about the multimedia activities (Dasher, reader, video)? 3. Were they effective in helping you develop the following skills in French? Yes or no? Please explain your answers. Reading Writing Listening Speaking 4. Were they effective in helping you to increase your knowledge of francophone culture? Why or why not? 5. At times were you frustrated with the multimedia activities? If so, please tell which aspects frustrated you. 6. Did the benefits of the multimedia activities outweigh your frustration? Please explain. 7. Did the multimedia activities help your self-confidence to speak, read, write and listen to French? Please explain. 8. Did you ever do parts of the lab in collaboration with another student? Yes or no (circle one)? If yes, do you feel that working together facilitated your learning of French? 9. Were the speaking and writing follow-up activities done on/for Wednesday effective ways to synthesize and apply what you had learned in the lab? Please comment: Speaking activities: Written work: 10. Rank these language learning resources from 1 (most helpful) to 7 (least helpful). Feel free to comment. Allons-y textbook Allons-y video Dasher exercises French glossary (electronic) paper dictionary Reader (electronic) Spellchecker
300
CALICO Journal

NOTES
1
Nieves (1994) did a pilot study exploring the use of a multimedia program in Spanish entitled Exito. The project included the development of a first-semester Spanish course using multimedia, classroom-based activities based on the multimedia materials and assessment of the students language development. The research project highlighted that students could develop a beginning level of proficiency in Spanish by interacting with the multimedia program and by meeting with a professor in small groups one period per week.
2
We used the Dasher authoring system to create vocabulary and grammar exercises (Pusack & Otto, 1992).
3
It is not our intent to promote any particular software, video, or textbook materials. Since we were working at Carnegie Mellon University, we chose materials already adopted and tested by the Department of Modern Languages.
4
Following Chamot, (personal communication, November 1995), we also collected information about students affect at the beginning and end of the semester. It is beyond the scope of this article to report those findings.
5
We would have preferred to administer externally rated pre- and posttests on all skills but this would have taken too much class time and would thus have been unfair to the students. Therefore, we chose to use data from classroom tests.
6
The writing was analyzed using an adapted form of the ESL Composition Profile (Glisan, 1981).
7
Given the wide range of time students reported, we wonder if some students may have mistakenly included the time they spent preparing the follow-up writing and speaking assignments in the early weeks.
8
We chose to use the Reader because the courseware had been used successfully and repeatedly in fourth semester French classes at Carnegie Mellon University. However, neither the courseware nor the readings themselves had been piloted in second semester classes.
9
Appendix D contains the final open-ended questionnaire administered to the treatment group. Control group students filled out an adapted questionnaire.
10
At Carnegie Mellon University, the Department of Modern Languages offers instructors the option of capping classes at 18 students.
11
In 1998-1999, CMU does not use the Reader in the first year French curriculum but instead requires students to work with documents on the web. These documents afford even greater authenticity and provide completely up-to-date information (e.g., weather reports, news).
Volume 17 Number 2
301

REFERENCES
Alderson, J. C., & Scott, M. (1992). Insiders, outsiders and participatory evaluation. In J. C. Alderson (Ed.), Evaluating second language education (pp. 25-58). Cambridge, UK: Cambridge University Press. Armstrong, K. M., & Yetter-Vassot, C. (1994). Transforming teaching through technology. Foreign Language Annals, 27, 475-486. Barnett, M. A. (1989). Writing as a process. French Review, 63 (1), 34-44. Beauvois, M. H. (1992). Computer-assisted classroom instruction in the foreign language classroom: Conversation in slow motion. Foreign Language Annals, 25, 455-463. Beretta, A. (1986). A case for field-experimentation in program evaluation. Language Learning, 36, 295-309. Bernhardt, S. A., Edwards, P., & Wojahn, P. (1989). Teaching college composition with computers. Written Communication, 6 (1), 108-133. Braegger, J. D., & Rice, D. B. (1996). Allons-y: Le Franais par tapes (4th ed.). Boston, MA: Heinle & Heinle. Brandl, K. (1995). Strong and weak student preferences for error feedback options and responses. The Modern Language Journal, 79, 194-211. Brown, J. D. (1989). Language program evaluation: A synthesis of existing possibilities. In R. K. Johnson (Ed.), The second language curriculum (pp. 222-241). Cambridge, UK: Cambridge University Press. Brown, J. D. (1995). Language program evaluation: Decision, problems and solution. Annual Review of Applied Linguistics, 15, 227-248. Bump, J. (1990). Radical changes in class discussion using networked computers. Computers and the Humanities, 24, 49-65. Chun, D. M. (1994). Using computer networking to facilitate the acquisition of interactive competence. System, 22, 17-31. Cononelos, T., & Olivia, M. (1993). Using computer networks to enhance foreign language/culture education. Foreign Language Annals, 26, 527-534. Conrad, B. K. (1996). CALL: non-English L2 instruction. Annual Review of Applied Linguistics, 16, 158-181. Costanzo, W. (1994). Reading, writing and thinking in an age of electronic literacy. In C. L. Selfe & H. Susan (Eds.), Literacy and computers (pp. 54-73). New York: The Modern Language Association of America. Cummins, J. (1991). Introductory remarks. In R. DeVillar & C. Faltis (Eds.), Computers and cultural diversity: Restructuring for school success (pp. viiix). Albany, NY: State University of New York Press. Daiute, C. (1984). Can the computer stimulate writers inner dialogues? In W. Wrench (Ed.), The computer in composition instruction. Urbana, IL: NTCE. DeVillar, R., & Faltis, C. (1991). Computers and cultural diversity: Restructuring for school success. Albany, NY: State University of New York Press. 302 CALICO Journal

DiMatteo, A. (1990). Under erasure: A theory of interactive writing in real life. Computer and Composition, 7, 71-89. Donato, R., Antonek, J., & Tucker, G. R. (1996). Monitoring and assessing a Japanese FLES program: Ambiance and achievement. Language Learning, 46 (3), 497-528. Garrett, N. (1991). Technology in the service of language learning: Trends and issues. The Modern Language Journal, 75 (1), 74-101. Glisan, E. (1981). ESL Composition profile. In H. L. Jacobs, S. Zingraf, D. Wormuth, V. Hartfield, & J. Hughey (Eds.), Testing ESL composition: A practical approach (pp. 323-324). Rowley, MA: Newbury House. Greenia, G. D. (1992). Computers and teaching composition in a foreign language. Foreign Language Annals, 25, 33-47. Hass, C. (1989). How the writing medium shapes the writing process: Effects of word processing on planning. Research in the Teaching of English, 23, 181-207. Hawisher, G. E. (1989). Research and recommendations for computers and composition. In G. E. Hawser, & C. L. Selfe (Eds.), Critical perspectives on computers and composition instruction (pp. 44-69). New York: Teachers College Press. Herron, C., & Moos, M. (1993). Electronics media in the foreign language and literature classroom: A fusion between science and the humanities. Foreign Language Annals, 26 (4), 478-490. Horwitz, E. K., & Young, D. J. (1991). Language anxiety: From theory and research to classroom implications. Englewood Cliffs, NJ: Prentice Hall. Johnson, D. M. (1985). Using computers to promote the development of English as a second language: A report to the Carnegie Corporation. Tuscon: University of Arizona. (EDRS:ED 278 211). Johnson, D. M. (1991). Second language and content learning with computers: Research in the role of social factors. In P. Dunkel (Ed.), Computer-assisted language learning and testing: Research issues and practice (pp. 61-83). New York, NY: Newbury House. Johnson, D. M. (1992). Approaches in second language learning. White Plains, NY: Longman Publishing Company. Kelm, O. R. (1992). The use of synchronous computer networks in second language instruction: A preliminary report. Foreign Language Annals, 25 (5), 441454. Lafford, P. A., & Lafford, B. A. (1997). Learning language and culture with internet technologies. In M. D. Bush, & R. M. Terry (Eds.), Technology-enhanced language learning (pp. 215-262). Illinois: National Textbook Company. Liu, M., & Reed, W. M. (1994). The relationship between learning strategies and learning styles in a hypermedia environment. Computers in Human Behavior, 10 (4), 419-434. Long, M. H. (1984). Process and product in ESL program evaluation. TESOL Quarterly, 18, 409-425. Volume 17 Number 2 303

Lynch, B. (1990). A context-adaptive model of program evaluation. TESOL Quarterly, 24, 23-42. Meunier, L. E. (1994). Computer-assisted language instruction and cooperative learning. Applied Language Learning, 5 (2), 31-56. Musumeci, D. (1998). Mallard, a web-based application in use at the University of Illinois (technology demonstration). Pittsburgh, PA: University of Pittsburgh. Mydlarski, D. (1987). Cooperative computer-assisted language learning: Is it living up to its promise? Journal of Educational Techniques and Technologies, 20 (1), 26-29. Nagata, N. (1993). Intelligent computer feedback for second language instruction. The Modern Language Journal, 77, 330-339. Nagata, N., & Swisher, V. M. (1995). A study of consciousness-raising by computer: The effect of metalinguistic feedback on second language learning. Foreign Language Annals, 28, 337-347. Neu, J., & Scarcella, R. (1991). Word processing in the ESL writing classroom. In P. Dunkel (Ed.), Computer-assisted language learning and testing: Research issues and practices (pp. 169-187) New York: Newbury House. Nieves, K. A. (1994). The development of a technology-based class in beginning level Spanish: Experiences with Exito. (Doctoral dissertation, George Mason University). Phinney, M. (1988). Computers, composition and second language teaching. In M. C. Pennington (Ed.), Teaching languages with computers: The state of the art. La Jolla, CA: Althesan. Phinney, M. (1991). Computer-assisted writing and writing apprehension in ESL students. In P. Dunkel (Ed.), Computer-assisted language learning and testing: Research issues and practices (pp. 189-204). New York: Newbury House. Piper, A. (1986). Conversation and the computer: A study of the conversational spinoff generated among learners of English as a foreign language working in groups. System, 14, 187-198. Pitiyanuwat, S. (1986). On-the-job behavior of trainees as a function of their reaction to the training program and learning achievement. In A. Wangsotorn, A. Maurice, K. Prapphal, & B. Kenny (Eds.), Trends in language program evaluation (pp. 142-154). Bangkok: Chulalongkorn University. Pusack, J. P., & Otto, S. (1997). Taking control of multimedia. In M. D. Bush & R. M. Terry (Eds.), Technology-enhanced language learning (pp.1-46). Lincolnwood, IL: National Textbook Company. Raschio, R. (1990). The role of cognitive style in improving computer-assisted language learning. Hispania, 73 (2), 535-541.
304
CALICO Journal

Reid, J. M. (1987). The learning style preferences of ESL students. TESOL Quarterly, 21, 87-111. Robinson, G. L. (1989). The CLCCS CALL study: Methods, error feedback, attitudes, and achievement. In W. F. Smith (Ed.), Modern technology in foreign language education: Application and projects (pp. 119-134). Lincolnwood, IL: National Textbook Company. Robinson, G. L. (1991). Effective feedback strategies in CALL: Learning theory and empirical research. In P. Dunkel (Ed.), Computer-assisted language learning and testing: Research issues and practice (pp. 155-167). New York: Newbury House. Roblyer, M., Castine, W., & King, F. (1988). Assessing the impact of computerbased instruction. New York: Haworth Press. Rodrigues, D. (1985). Computers and basic writers. College Composition and Communication, 36 (3), 336-339. Ross, S. (1992). Program-defining evaluation in a decade of eclecticism. In J. C. Alderson & A. Beretta (Eds.), Evaluation in second language education (pp. 167-195). Cambridge, UK: Cambridge University Press. Snow, M. A., & Brinton, D. M. (1988). Content-based instruction: Investigating the effectiveness of the adjunct model. TESOL Quarterly, 22, 553574. Stenson, N., Downing, B., Smith, J., & Smith, K. (1992). The effectiveness of computer-assisted pronunciation training. CALICO Journal, 9 (4), 5-18. Svenconis, D. J., & Kerst, S. (1995). Investigating the teaching of second language vocabulary through semantic mapping in a hypertext environment. CALICO Journal, 12 (2/3), 33-57. Tharp, R., & Gallimore, R. (1988). Rousing minds to life: Teaching, learning, and schooling in social context. Cambridge, UK: Cambridge University Press. Thiesmeyer, J. (1989). Should we do what we can? In G. E. Hawser, & C. L. Selfe (Eds.), Critical perspectives on computers and composition instruction. New York: Teachers College Press. Tremblay, P. F., & Gardner, R. C. (1995). Expanding the motivation construct in language learning. The Modern Language Journal, 79 (6), 505-520. Youngs, B. E., & Jones, C. M. (1996). CMU French Reader [Computer software]. Pittsburgh, PA: Carnegie Mellon University. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Wertsch, J. (1979). From social interaction to higher psychological processes: A clarification and application of Vygotskys theory. Human Development, 22, 3-22. Wiggins, G. (1997). Educative assessment: Designing assessments to inform and improve student performance. San Francisco, CA: Jossey-Bass.
Volume 17 Number 2
305

ACKNOWLEDGMENTS We are grateful to G. Richard Tucker, Barbara Freed, Christopher M. Jones, Keiko Koda and Brian MacWhinney for their feedback on the pilot study and their encouragement throughout this project. We are indebted to Elaine Rubinstein in the Office of Measurement and Evaluation of Teaching at the University of Pittsburgh for conducting the statistical analysis, to Michael West for his help in drafting the culture quiz, and to Mark Sanford for rating the writing samples. We also want to acknowledge the work of our two undergraduate research assistants during the spring of 1997: Ena Khan and Emily Spencer.
AUTHORS ADDRESSES Bonnie Adair-Hauck University of Pittsburgh Department of Instruction and Learning 4M20 Forbes Quad Pittsburgh, Pa 15260 Phone: 724/935-8275 Fax: 724/934-8832 E-mail: adairhauck@mindspring.com Laurel Willingham-McLain, Associate Director Center for Teaching Excellence 312 Administration Bldg Duquesne University Pittsburgh, PA 15282 Phone: 412/396-1760 Fax: 412/396-6577 E-mail: willingham@duq.edu Bonnie Earnest Youngs Department of Modern Languages Carnegie Mellon University Baker Hall 160 Forbes Avenue Pittsburgh, PA 15213-3890 Phone: 412/268-8050 (office); 412/621-2517 (home) Fax: 412/268-1328 E-mail: byoungs@andrew.cmu.edu
306
CALICO Journal

573 1466 1 SM

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

573 1466 1 SM

Uploaded by

Copyright:

Available Formats

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology and Second Language Learning

Laurel Willingham-McLain, Ph.D.

Bonnie Earnest Youngs, Ph.D.

Evaluating the Integration of Technology

Adair-Hauck, Willingham-McLain, and Youngs

Innovations in Technology Application

Evaluating the Integration of Technology

Adair-Hauck, Willingham-McLain, and Youngs

Language Learning as a Social Phenomenon

Evaluating the Integration of Technology

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

Adair-Hauck, Willingham-McLain, and Youngs

Open-ended questionnairee Follow-up interviews (selected students, treatment group only)

Evaluating the Integration of Technology

Findings EVALUATING LEARNER MOTIVATION AND

Adair-Hauck, Willingham-McLain, and Youngs

EVALUATING LEARNERS ATTAINMENT OF GOALS

Evaluating the Integration of Technology

EVALUATING KNOWLEDGE OF CULTURE

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

EVALUATING LISTENING, READING, AND WRITING

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

TIME ON TASK DOING THE TELL COMPONENTS

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

ROLES OF THE TEACHER AND STUDENTS

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

Reader Chapt. 6 (server)

3. Les Devoirs (essayez dutiliser le glossaire franais):

Adair-Hauck, Willingham-McLain, and Youngs

APPENDIX B Francophone Cultural Knowledge Quiz

Evaluating the Integration of Technology

What do you say if someone says merci to you?

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration of Technology

You might also like