What a Formative Experiment Reveals About Internet Reciprocal Teaching David Reinking Jacquelynn Malloy Angela Rogers Katherine

Robbins

Clemson University Paper presented at the National Reading Conference Austin, TX December, 2007 This paper provides an overview of and preliminary findings from a formative experiment conducted during Year 2 of a three-year project funding by the Institute of Education Sciences, U.S. Department of Education. The project, currently in its third year and referred to as Teaching Internet Comprehension to Adolescents (TICA), addresses the following: (a) profiling levels and patterns of online activity among middle grade students, specifically students in the seventh-grade who are at risk of dropping out of school; (b) identifying students at a high end of that populations distribution of use and skill in regard to the Internet; (c) determining the skills and strategies of that sub-population in locating, evaluating, synthesizing, and communicating information on the Internet; (d) developing, based on those data, a taxonomy of Internet skills and strategies to guide instructional activities aimed at enhancing Internet reading comprehension; (e) and investigating how an instructional framework referred to as Internet Reciprocal Teaching (IRT) might be used effectively to increase Internet reading comprehension. Year 1 of the project employed a survey to address the first two of these purposes, which was followed by the collection of verbal protocol data for 50 savvy users of the Internet

Formative Experiment 2 to address the third and fourth purposes. Years 2 and 3 used these data and an evolving taxonomy of Internet skills to refine and investigate the effectiveness of IRT as an instructional framework to incorporate Internet reading comprehension skills, strategies, and dispositions into seventh-grade. Year 2 used the methodology of a formative experiment, a relatively new, but increasingly used, approach to investigating interventions. A brief introduction to this approach and its purposes is provided in a subsequent section of this paper. Year 3, which is the current year of the project, is a conventional experiment designed as a randomized field trial and using HLM statistical procedures. Readers interested in more detail concerning the overall project, the integrated goals and purposes across each year of the project, and specifics of Internet Reciprocal Teaching are referred to the other papers in the symposium of which the present paper is a part. In the remainder of this paper we briefly overview: (a) the rationale and purpose of a formative experiment in relation to this project; (b) our methods; (c) and some preliminary findings from our on-going data analysis. What is a formative experiment? A formative experiment (a.k.a. design experiment, design-based research) is a relatively new approach to conducting education research, specifically instructional interventions in classrooms, but one that has received increasingly enthusiastic attention among education researchers. For example, the following respected journals have devoted themed issues to discussing this approach to research, including its origins, why it is needed, its advantages and limitations, and so forth: Educational Researcher (2003, Vol. 32. No. 1), Educational Psychologist (2004, Vol. 39, No. 4), and Journal of Learning Sciences (2004, Vol. 13, No. 1). Likewise, a chapter on design experiments appears in the Handbook of Complementary Methods in Education Research sponsored by the American Educational Research Association (Schoenfeld, 2006). Further, two recently published books aim to define more clearly the parameters of this approach and to offer frameworks and standards of rigor for researchers who employ it: Education Design Research (Van Den Akker, Gravemeijer, McKenney, & Nieveen, 2006) and On Formative and Design Research (Reinking & Bradley, 2008), the latter book appearing in a series devoted to research methodologies in literacy research. The latter book substantiates that this approach has strong roots and relatively extensive

Formative Experiment 3 use in literacy research. Ann Brown (see Brown, 1992), a highly regarded literacy researcher, is credited with writing the seminal chapter introducing the term design experiment. That is the term she used to describe her research that attempted to move her highly controlled laboratory studies on the role of metacognition in reading comprehension into the messy, less-controlled conditions of real classrooms. Likewise, the early work of Luis Moll (e.g., Moll & Dias, 1987) called for approaches to research that are focused on bringing about constructive change in classrooms, which is an orientation fundamental to the rationale for formative and design experiments. Another indication that this approach is entering the mainstream is that it has been used in several articles published in Reading Research Quarterly, the fields premier research journal and one of the most influential research journals in education (see Ivey & Broadus, 2007; Jimenez, 1997; Neuman, 1999; Reinking & Watkins, 2000). Formative and design experiments have distinctly different goals and are based on distinctly different epistemological assumptions when compared to conventional scientific experiments (typically referred to as quantitative methods) or investigations using naturalistic methods (typically referred to as qualitative methods). These differences are summarized succinctly in the following figure reproduced from Reinking and Bradley (2008). Figure 1.2. A Comparison of formative and design experiments to conventional methodologies Methodology/Approach Experimental Naturalistic Formative/ Design

Formative Experiment 4 Contextual Variation Dominant Metaphor(s) Guiding Question Stance Toward Intervention Operative Goal Utility
Controlled, or viewed as nuisance and neutralized by randomization Laboratory Studied, analyzed Studied, analyzed, accommodated Ecology, Engineering

Lens, rhizome, jazz

What is best most of the time? Comparison (x vs. y)

What is?

What could be?

Establish causal relations Broad generalizations across contexts, topdown policy making

Socio-cultural and ideologically positioned practice Generate thick descriptions Nuanced understanding, raised consciousness, ideological confirmation, social action Constructivism

Selection (x or y) and modification (x1->x2>x3->x4-> etc.) Put theory to work Context-specific recommendations, identifies factors and mechanisms enabling effective practice Pragmatism

Philosophy/Stance (epistemology ) Theoretical Imperative Methodological Imperative Participants Prototypical Researcher Starting point for research Relations and contributions to Practice

Positivism, postpositivism, scientific realism General laws and reductionist models Internal validity (fidelity) Pawns Chess-playing statistician Theory-driven question or hypothesis rooted in a quest for attainable truth Broad Generalizations

Deep socio-cultural understandings Interpretative trustworthiness Agents Butterfly-chasing ethnographer Theory-driven question rooted in socio-cultural awareness Deep Reflections

Workability

Ecological validity

Partners Deal-making, mixed methodologist Pedagogical goal connecting theory and practice rooted in values Specific suggestions

Formative Experiment 5 Why was a formative experiment used in Year 2 of this project? We believe the purposes and goals of this three-year project are well matched to the rationale for a formative experiment. Specifically, in this project we are interested ultimately in developing workable and effective instructional activities aimed at furthering a critical, and currently neglected, instructional goal (e.g., see Leu, 2006 for a data-based argument that digital literacy is woefully neglected in the literacy curriculum). Formative and design experiments are well suited to testing innovative, theory based interventions in authentic classroom settings where naturally occurring variables are free to operate and to perhaps affect the intervention in unpredictable ways. In addition, we believe that using this methodology in Year 2 would be an effective means for providing critical data, information, and perspectives that would guide the more carefully controlled study in Year 3. Although many researchers see formative or design experiments as providing independently useful data and conclusions, others also see this approach as being particularly useful in informing conventional experiments conducted subsequently and informed by the results of a formative or design experiment (e.g., see Gersten, 2005; McCandliss, Kalchman, Bryant, 2003). Thus, beyond producing independent data that could inform practitioners about how best to implement IRT to achieve a specific pedagogical goal, the purposes for using a formative experiment in Year 2 were to refine IRT for its more controlled implementation in Year 3 and to guide us in determining what variables were salient in that implementation, which variables to measure or control, and so forth. Formative experiments have also been identified as an approach to research well suited to testing, modifying, and developing pedagogical theories (sometimes referred to as principles of enactment in classrooms) often through innovative instructional interventions aimed at achieving specific instructional goals and transforming classroom environments (see Reinking & Bradley, 2004/2008). These characteristics of formative and design experiments were consistent with the goals of the current project and the sequence of research activities across the three years of the project. METHOD Participants and Schools Participants and schools in the formative experiment during Year 2 were drawn from a larger pool from Year 1. All middle schools participating in the project were from school districts and communities that had demographic profiles indicating a high percentage of

Formative Experiment 6 students who were likely to drop out of school. In Connecticut, schools and districts were from urban areas and in South Carolina, schools and districts were from rural areas or small towns. For the formative experiment in Year 2, we worked in two schools in Connecticut and three schools in South Carolina. Consistent with guidelines for conducting a formative experiment - in this case, a set of parallel formative experiments centered in these respective classrooms - we selected classrooms from among the larger pool using the following criteria: (a) schools and teachers who were willing to participate and committed to the projects goals; (b) schools/classrooms in which conditions were not exceptionally ideal or abominably poor to accommodate the intervention; (c) and that represented a diversity of contexts across sites. For example, in relation to the latter criteria, represented among our research sites there was variation in terms of rural versus urban, level of technological knowledge and support, and teaching focus and population (e.g., one classroom was a pull-out program for special education students, and one school/classroom had a high percentage of second-language learners). The present report focuses on data from three sites identified by the following pseudonyms for the school names: Bolivar Middle School, Streamwood Middle School, and Thompson Middle School. Bolivar School is in a small town near a medium size city. It has a high percentage of diverse Hispanic students who come from Mexico, Central and South America. We worked with Ms. Hart (all teachers and students names are pseudonyms), the 7th-grade language arts teacher, who is in her third year of teaching, but who had entered teaching as a second career. Previously to pursuing a teaching degree, she worked with technology in a business setting. So, she was technologically savvy and enthusiastic about the goals of the project. The school had been identified as a Title 1 school by the larger district of which it was a part. District policy was to provide Title 1 schools with a laptop for each student. The entire school had wireless access to the Internet; although, as is typical of most schools with Internet access, a district firewall limited what students could access. Streamwood Middle School was also located in a small town within a larger metropolitan area comprised of five school districts. Ms. Jenkins, the 7th-grade language arts teacher, was a veteran teacher of 25 years. Ms. Jenkins was often assigned to teach the lowerachieving students in the school, and the classroom that was involved in the IRT

Formative Experiment 7 intervention included a high percentage of resource students. The school also had wireless laptops for each student, although they were not often used in this classroom for instruction prior to the intervention. Thompson Middle School is a small school located in a rural school district in South Carolina. It is a combined middle and high school, with only 360 students in the sixth through twelfth grades. Ms. Watson, the 7th grade special education teacher, had ten years of teaching experience at the beginning of the project. Wireless laptops were not available to the students during the majority of the project because access to the laptop cart was not always granted to this classroom. The group of students involved in the intervention were not officially designated as self-contained", but were, none-the-less grouped specifically on the basis of being identified as most at-risk to drop out. All of the students had been identified as having special needs and had been in self-contained classrooms for the entire school day during the previous year. Since it was a small class with a maximum of ten students on any given day, students were required to use old desktop computers with slow Internet connections at the back of the room for the project. The pedagogical goal A formative experiment investigates how a theory-based, often innovative, intervention can be implemented to achieve a valued, sometimes difficult-to-achieve, pedagogical goal (Reinking & Bradley, 2004/2008). Thus, unlike other research methodologies that address research questions specified in advance, a pedagogical goal and what needs to be done to achieve it through the intervention is what drives a formative experiment. The present investigation addressed the following pedagogical goal: Among middle-school students at risk of dropping out, increase Internet reading comprehension strategies to improve reading (on-line and off-line), academic engagement, and achievement. The intervention The intervention in a formative experiment must be justified in terms of its theoretical and/or empirical validity (e.g., it might be validated as promising on the basis of previous carefully controlled experimental studies; see Reinking & Bradley, 2004/2008). There must also be a rationale for why it addresses an important pedagogical goal. The intervention investigated in the present study was Internet Reciprocal Teaching. A full discussion of this intervention and a rationale for its use in the present investigation is beyond the scope of this paper and can be found partially in another paper in the symposium of which the present paper is a part.

Formative Experiment 8 Phases of the investigation Researchers conducted this investigation using the following phases: Phase 1: During the spring and early summer prior to Year 2, we recruited schools and teachers from the set of schools fitting a demographic profile indicating a high percentage of students at risk of dropping out of school and that met the criteria outlined in a previous section discussing participants and schools. In August, before the school year began, we met with participating teachers who agreed to participate for approximately six hours each to discuss the project including the methodology of a formative experiment, the IRT intervention and its rationale, their commitments and role in data collection, as well as a host of logistical and scheduling issues. We also addressed their questions and concerns, did preliminary planning to integrate IRT into their curriculum and instructional objectives, decided on times to meet regularly during the course of the investigation. We also asked them to participate in a semi-structured interview about their experience with and views of using technology in the classroom, views of teaching language arts, and so forth. Phase 2: This phase occurred approximately between the 3rd and 6th week of the school year. Observational data were gathered to create a thick description of the environment of the classroom, school, and community. Information was gathered from public documents, interviews with teachers and administrators, and through observations of the classroom and the school. These provided a baseline of dispositions and practices regarding the use of and support for technology in the classroom. Phase 3: Baseline data were collected to characterize the initial status of students relative to the investigations pedagogical goal (identified explicitly in a subsequent section). These included the Internet Use Survey from Year 1, the Online Reading Comprehension Assessment (ORCA), the Assessment of Motivation for Online and Offline Reading (AMOOR), and the School Success Profile (SSP). The ORCA and AMOOR are assessments that were designed and piloted prior to Year 2 by TICA researchers and these measures were found to demonstrate acceptable levels of reliability and validity. These are discussed in greater detail in the paper submitted by Coiro, Castek & Henry as part of this symposium. The SSP is a standardized instrument used to measure the degree to which students may be at-risk of dropping out of school (http://www.schoolsuccessprofile.org/). Phase 4: This was the intervention phase that began with our first attempts to implement IRT. During this phase, which is at the heart of a formative experience, we

Formative Experiment 9 collected qualitative data to determine, as described in more detail in subsequent sections, what factors enhance or inhibit the effectiveness of the intervention, documented modifications of the IRT content and framework in response to that data, the effects of our modifications, what unanticipated effects the intervention produced, and if the intervention was having any effect on the instructional environment. The intervention was implemented twice weekly for approximately 12 weeks, or 24 lessons. In each classroom, the researcher collaborated with the teacher to develop IRT lessons tied to the state content standards and curricular requirements of the school. In some classrooms, the researcher delivered the intervention while the teacher assisted and observed, and in others, the teacher presented some of the lessons while the researcher assisted. Each researcher designed their lessons using the Internet Reciprocal Teaching Lesson Framework as a guide to ensure that the essential components of IRT were being integrated into instruction in each classroom (see Appendix A). However, beyond these essential components, researchers and teachers at each site collaborated to adapt IRT to accommodate local instructional practices, curricular needs, and the unique data that were being collected at each site. Phase 5: During this phase, at the end of the school year, we re-administered the baseline measures used during Phase 3. The post-intervention assessments required approximately two weeks to complete. During this time, the researchers interviewed the classroom teachers and a select group of focal students. The purpose of the semistructured interview was to gather perceptions of principal agents involved in the intervention regarding changes in learning and motivation for learning tasks that may have occurred as a result of IRT. Scheme for collecting and analyzing data During the intervention phase, the researcher and classroom teacher collaborated in planning lessons using the common lesson plan format described in the discussion of Phase 4 and included in Appendix A. The lesson plan included the objectives and targeted curriculum standards addressed in the lesson, a description of the mini-lesson and sharing activities that would be used in achieving the learning objectives, and a listing of Web sites and supplementary materials selected for use in the lesson. The lesson plan, maintained in a digital format, also provided room for researcher and teacher reflections regarding the lesson that could be added following each class period. These reflections and observations provided valuable data regarding the effectiveness of IRT in moving toward the pedagogical goal, in planning future lessons, and in evaluating the need and effectiveness of modifications of the intervention.

Formative Experiment 10 The products collected for use in the analysis of the Year 2 formative experiment included: (a) the lesson plans for each of the 20 lessons; (b) the researcher's and teacher's reflections and observations of the lessons; (c) the baseline data gathered in Phase 1; (d) student products, (e) classroom photos, and; (f) the post-intervention interviews with the teachers and focal students. Researchers uploaded these data into separate NVivo 7 projects for each individual site for analysis. NVivo 7, a product of the QSR Corporation (http://www.qsrinternation.com), allows for digital analysis of documents and other types of data and provides a means to consider the data from individual sites separately and later merge them for a cross-case analysis. The software also provides a means to establish individual student cases that can be managed in tandem with the whole-class analysis. Researchers uploaded the pre- and post-intervention data derived from the assessment measures administered during Phases 3 and 5 in an Excel spreadsheet. These quantitative data will be integrated with findings from the qualitative information gathered on individual students to provide a finer grained analysis of student response to the intervention. The researchers conducted the analysis of the documents and products collected during the five Phases according to the framework for conceptualizing, collecting, and reporting formative data as proposed by Reinking and Bradley (2004/2008). As stated in the description of Phase 4, the qualitative data were collected to determine the factors that enhance or inhibit the effectiveness of IRT in achieving the pedagogical goal, the modifications of the IRT content and framework that were required to adjust toward the pedagogical goal, the unanticipated effects the intervention produced, and whether the intervention was having any effect on the instructional environment. Therefore, the primary codes established at the start of the analysis were: (a) factors that enhance; (b) factors that inhibit; (c) modifications and effects; (d) unanticipated effects; (e) and changes in the environment. In the process of reading through and considering the various documents included in the data set, constant-comparative methods (Bogdan & Biklen, 2003; Merriam, 1988) were utilized to inductively and deductively code the data and derive and distinguish the thematic categories. The process of moving between inductive and deductive reasoning to evaluate data is supported by Morgan (2007) and described by Onweugbuzie and Leech (2006). While some of the themes generated during the analysis were deduced from the formative structure described earlier in this paragraph, others were induced from the data and created additional primary nodes, or more frequently, became

Formative Experiment 11 distinctive "branches" of the primary nodes. During the initial stages of the Year 2 analysis, the researchers responsible for coding the data from their individual sites met periodically to discuss the emerging themes and coding structures. These exchanges provided an opportunity to describe and justify interpretations and to establish structural validity of context independent themes that emerged across sites, as well as those that appear to be context dependent. Although the analysis of the complete data set continues, a preliminary listing of the emerging themes is presented in Appendix B. FINDINGS While the complete analysis of data from Year 2 is continuing, a foundational set of themes have been identified and have been useful in informing the version of IRT that is being implemented in the traditional experiment in Year 3. These emerging themes include both context independent and context dependent factors that influence the effectiveness of IRT in advancing the intervention toward the following pedagogical goal: Among middle-school students at risk of dropping out, increase Internet reading comprehension strategies to improve reading (on-line and off-line), academic engagement, and achievement. A common factor found to influence the effectiveness of IRT across all of the sites was the grouping of students during the sharing portion of the lesson. A primary goal of IRT lessons is the exchange of ideas and strategies in accomplishing a common task, and establishing and maintaining groups or pairings of students that 'worked' was as much a product of trial and error as it was of explicit teaching in how to collaborate with others. As is the nature of working with adolescents, an understanding of the evolving social relationships and climate of the classroom a mercurial beast at best is crucial in creating an instructional environment where democratic exchanges can occur. These excerpts from the Streamwood researcher's observation notes show that it takes a few weeks discover how best to arrange students for sharing: Certainly, pairing students well or choosing groups is key. It takes a few classes to see how they work, though. After the lesson, [Mrs. Jenkins] and I discussed how important selecting groups is to the success of the lesson. [Researcher notes, Lesson 4] B. and L. were a good pairing, not because they worked together particularly well, but because L. was not able to boss B. around. She (L.)

Formative Experiment 12 is hard-of-hearing and can tune others out, but B. is very confident on the computer, whereas L. is not, and he (B.) won't be bullied or ignored. [Researcher notes, Lesson 5] Another factor that was found to enhance progress toward the pedagogical goal across sites involved the students who were less confident in their academic prowess or otherwise marginalized in the mainstream classroom. As stated in the description of the Year 2 schools, the classrooms involved in the intervention were comprised of a high percentage of students who received resource services, were previously in self-contained classrooms, were wards of the State, or who did not speak English at home. Often, these students demonstrated behaviors that indicated disengagement with traditional instructional formats, particularly those that were print-based and relied on question/response teacher-led instruction. A common theme emerged from the Year 2 data that described the changes in disposition and manner of some of these students when they demonstrated skill in negotiating the Internet-based tasks involved in the lessons. One particular student from Thompson Middle, a young girl identified with behavior disorders, was known for her angry outbursts in the classroom. Although she expressed difficulty with many traditional academic tasks, she was particularly good at finding information quickly on the Internet, and was also fairly adept at showing other students how she did it. Although she was easily frustrated in some contexts, she was also quite capable at leading others in more positive and appropriate ways when her talents were highlighted and she felt connected to the process. At some of the sites, identifying an "expert" a student who was observed to demonstrate an early grasp of the targeted Internet strategy for a lesson was helpful in encouraging participation and sharing according to the principles of IRT. Often, researchers indicated that students became comfortable with and accepting of the wisdom of these experts, as described in the following excerpt from the Streamwood data: D.V. came by the trick of clicking 3x at the beginning of a paragraph to get it highlighted purely by chance. But, having been frustrated by trying to highlight within Web page, he was delighted to share it with the class many were relieved! [Researcher notes, Lesson 4] Researcher and teacher observations from several of the sites provided support for the value of noticing and highlighting the developing skills and strategies of students, particularly those who do not often achieve this level of notice in the classroom, in creating a reciprocal teaching environment utilizing an Internet-based skill set.

Formative Experiment 13 In varying degrees across the sites, access to technology was found to be a factor that could inhibit progress toward the pedagogical goal. In some instances, the filters and firewalls put in place by school districts made it difficult to make use of certain online resources, such as shared student blog sites or wikis, or to access images that could be incorporated into projects. In addition to this, inadequate bandwidth provisions made it difficult for 25 wireless laptops to simultaneously access online sites without enduring slow page loadings. These bandwidth issues also affected communication by email and frustrated students and teachers alike. In a more disturbing, context dependent example of this inhibiting factor, access to the "faster" machines was withheld from the Thompson Middle School classroom involved in the intervention. In this school, the media center specialist with whom teachers reserved the laptop cart was not comfortable with having this particular group of students use the laptops, as she felt they might break them. She communicated this to Ms. Watson, the classroom teacher, on several occasions, and the researcher documented Ms. Watsons observations each time. Not having access to the newer laptops meant that students in Ms. Watsons class had to use older desktops with very slow modems. When a lesson required that a student use more than one Web site, lack of access often meant that they were not able to finish the assignment at all. As an example of changes in the environment that occurred as a result of the IRT intervention, the following researcher in situ memo, written while analyzing the Thompson Middle School data, is offered as evidence of the influence that a novel instructional practice can have on a classroom when provided over an extended period of time: In early observations of [Ms. Watsons] classes, the researcher noted that most of the lessons involved direct instruction and little collaborative work. Students would complete worksheets, guided by [Ms. Watson] on each question. Students were not really permitted to discuss why a certain answer was correct or incorrect; they merely listened to [Ms. Watsons] explanations. After the implementation of IRT, the researcher noted many times in her observations that this was changing. Even on days on which the researcher did not go to the school for IRT lessons, students began working in their groups. [Ms. Watson] stated in her post interview that the concept of a mini-lesson at the beginning of the class was particularly useful to her. She felt more comfortable letting the students explore on their own when they were in groups that she knew worked well together based on what happened in our IRT lessons.

Formative Experiment 14 Additionally, after we had one lesson in which a previously unengaged student named D. suddenly became very interested in our lesson because he got to formulate his own research question, [Ms. Watson] began designing bigger projects that involved not only group work, but student formulated questions. D. especially seemed changed by the end of the IRT intervention. [Ms. Watson] noted in her interview that D. seemed happier in her class and did not act out as frequently in not only her class, but also in his other classes throughout the day. [Researcher memo, Lesson 17]. While these descriptions of factors that enhanced and inhibited progress toward the pedagogical goal do not comprise a complete detailing of the data, they do, when combined with the listing of primary and secondary nodes in Appendix A, provide a foundational structure of the Year 2 findings that were particularly instrumental in shaping the Year 3 version of IRT. A comprehensive report of the findings from each site is currently underway, with plans for collaboration on a cross-case analysis to follow.

Formative Experiment 15 References Bogdan, R. C., & Biklen, S. K. (2003). Qualitative research for education: An introduction to theories and methods, (5th ed.). Boston: Allyn & Bacon. Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of Learning Sciences, 2(2), 141-178. Gersten, R. (2005). Behind the scenes of an intervention research study. Learning Disabilities Research and Practice, 20(4), 200-212. Ivey, G. & Broaddus, K. (2007). A formative experiment investigating literacy engagement among adolescent Latina/o students beginning to read, write, and speak English. Reading Research Quarterly. Jimnez, R. T. (1997). The strategic reading abilities and potential of five low-literacy Latina/o readers in middle school. Reading Research Quarterly, 32, 224-243. Leu, D.J. (2006). New literacies, reading research, and the challenges of change: A Deictic perspective. (NRC Presidential Address). In J. Hoffman, D. Schallert, C. M. Fairbanks, J. Worthy, & B. Maloch (Eds.) The 55th Yearbook of the National Reading Conference. (1-20). Milwaukee, WI: National Reading Conference. McCandliss, B. D., Kalchman, M., & Bryant, P. (2003). Design experiments and laboratory approaches to learning: Steps toward collaborative exchange. Educational Researcher, 32(1), 14-16. Merriam, S.B. (1988). Case study research in education: A qualitative approach. San Francisco: Jossey-Bass. Moll, L., & Diaz, S. (1987). Change as the goal of educational research. Anthropology & Education Quarterly, 18, 300-311. Morgan, D. L., (2007). Paradigms Lost and Pragmatism Regained: Methodological Implications of Combining Qualitative and Quantitative Methods. Journal of Mixed Methods Research, 1(1), 48-76. Neuman, S. B. (1999). Books make a difference: A study of access to literacy. Reading Research Quarterly, 34, 286-311. Onwuegbuzie, A. & Leech, N. L. (October, 2006). Qualitative data analysis: A step-by step approach. Presentation provided by Education Technology Services of Clemson University, Clemson, SC. Reinking, D., & Bradley, B. A. (2004). Connecting research and practice using formative and design experiments. In N. K. Duke & M. H. Mallette (Eds.), Literacy research methodologies (pp. 149-169). New York: Guilford. Reinking, D., & Bradley, B.A. (2008). On Formative and Design Experiments. New York:Teachers College Press.

Formative Experiment 16 Reinking, D., & Watkins, J. (2000). A formative experiment investigating the use of multimedia book reviews to increase elementary students independent reading. Reading Research Quarterly 35 (3), 384-419. Schoenfeld, A. H. (2006). Design experiments. In J.L. Green, G. Camilli, P.B. Elmore, A. Skukauskaite, & E. Grace (Eds.), Handbook of Complementary Methods in Education Research (pp. 193 205). Mahwah, NJ: Lawrence Erlbaum. van den Akker, J., Gravemeijer, McKenney, S, and Nieveen, N. (Eds.) (2006). Educational Research Design. London, UK: Routledge.

Formative Experiment 17 Appendix A Internet Reciprocal Teaching Lesson Framework Teacher:__________________ School: ______________ Researcher: ____________

Date:_____________ Unit/Lesson ID____________________________________

I. TICA Objective (circle all that apply) Question Locate Evaluate Communicate Synthesize

Notes:

II. ELA Content Objective Learning Target(s):

Standard(s):

III. IRT Checklist ___ Democratic dialogue and discussion content ___ Model by teacher then student ___Opportunities to privilege struggling readers ___ Strategies emerge in relation to ___ Students as informants ___ Meaningful/authentic activities

Formative Experiment 18

IV. Types of Knowledge (circle all that apply) Declarative Notes: Procedural Conditional Reflective

V. Mini- lesson (modeled by teacher) What teacher will do:

Materials/links:

VI. Time to Explore Activity What students will do:

VII. Time to Share How students will share strategies:

VIII. What did we learn?

Formative Experiment 19 Class-generated summary:

IX. Who completed the objectives? (circle names of new experts)

X. New strategies to add to taxonomy?

XI. Assessment/product Name/description of artifact:

Formative Experiment 20

XII. Reflection Notes

Formative Experiment 21 Appendix B TICA Research: Year 2 Foundational Themes and Coding Structure I. Tree Nodes: A. Changes in Environment: Changes in the classroom environment that occurred as a result of the IRT treatment program. B. Enhance: Factors that enhance progress toward the pedagogical goal. 1. Administrative Support: Ways in which the district or school administration support the use of technology with teachers pupils. 2. Content ties: Content that works well with IRT to enhance progress toward the pedagogical goal. 3. Dispositions: Attitudes and dispositions that enhance progress toward the pedagogical goal. 4. Instructional practice: Specific instructional practices or activities that enhance progress toward the pedagogical goal. 5. Groups: Notes or observations that describe groupings or pairs that worked well to enhance progress toward the pedagogical a. Indications for individual work instances where individual work was preferable to group work. 6. Skills: Specific skills that permit participation in the IRT lessons in progressing toward the pedagogical goal. 7. Technology: Factors regarding hardware, software, access that enhance progress toward the pedagogical goal. C. Inhibit: Factors that inhibit progress toward the pedagogical goal. 1. Content ties: Content that did not work well in the IRT format. 2. Dispositions: Dispositions and attitudes that impede progress toward the pedagogical goal. 3. Instructional Practice: Instructional practices and activities that impede progress toward the pedagogical goal. a. Groups: Notes or observations that describe groupings or pairs that impede progress toward the pedagogical goal. 4. Lack of Administrative Support: Ways in which the district or school

or

goal.

Formative Experiment 22 administration does not support the use of technology in the classroom. 5. Skills: Lack of skills required to participate in IRT lessons that impede progress toward pedagogical goal. 6. Technology: Hardware, software or access issues that impede progress toward pedagogical goal. 7. Time: Insufficient time to complete an IRT lesson as planned. 8. Health and Other Issues: Instances where illness or outside influences, such as visitors in the classroom or weather concerns, influenced student dispositions on a particular day.

distracted or

D. Modifications: Modifications made in response to an effect of the IRT treatment. 1. Effective modifications: Modifications to the IRT model that had the effect of enhancing progress toward the pedagogical goal. 2. Ineffective modifications: Modifications to the IRT program that did not enhance progress toward the pedagogical goal. 3. Modifications needed: Notes that acknowledge that a modification is required. E. Unanticipated Effects: Unanticipated effects resulting from the IRT program. 1. High need students: Unexpected responses of students with high needs to IRT specific practices in IRT. II. Free Nodes: A. Standards: List of standards used during IRT for use in demonstrating the standards that were addressed during IRT lessons. May review later for how effectively they were addressed. B. Key Events: Vignettes and stories that would demonstrate well what happened in the environment as a result of the IRT lessons. C. Evidence of learning: Observation or statements that indicate learning in progress as a result of IRT.