Executive Summary

Study Island is a web-based standards mastery program based on each states standards. It provides assessment and skill practice for each grade level (Appendix C). Throughout the program, the student answers questions relating to the states standards and objectives (Appendix D) . If a student answers incorrectly, the program will give instantaneous feedback as well as remediation opportunities. The program provides differentiation though by cycling through skills that will provide optimal learning for the student. The student can practice a skill until reached mastery according to state leveled content. Motivation and engagement are provided through various online learning environments. Study Island can also be used as a progress monitoring tool for educators. The program provides educators with detailed and immediate feedback to be a tool for classroom remediation. The purpose of this evaluation is to determine if the Study Island program, invested in by a Georgia public school, improves math test scores for third grade students when it is being used as a remediation tool. Data was obtained through various sources. Monthly reports were obtained using the Study Island program. The program provides information such as login attempts, scores, time taken on assignments, and attempts/scores regarding each session/lesson. In addition, data was obtained through questionnaires, surveys, and ideally quantitative data provided from pre-CRCT assessments and the CRCT assessment. Based on 2009-2010 CRCT results from ABC Elementary, third grade students math scores increased after using the Study Island program when the program was first introduced but there was some regression following the next two years of implementation. Study Island was executed in this school in 2010. In 2009, 20.9% of second grade students did not meet standards in math. In 2010, only 16.3% of the rising second grade students, who were in third grade, did not meet in math. The state of Georgia discontinued the CRCT testing for first and second grade students after the 2010 year. Upon the first year of implementation, the program

was successful within ABC Elementary. Although in 2012, 22.6% did not meet the standards. Due to surveys and data, the justification of the decrease can be accredited to the new Common Core implementation, the lack of program training, and second grade no longer taking CRCT tests. Third grade is a high stakes year for young students; therefore, this program, if used appropriately, students should obtain the lessons, practice, and remediation they need improve math testing scores. To evaluate the program, the following evaluation questions were enlisted: How often do 3rd grade students, who need math academic remediation, use Study Island on a daily/weekly basis? What are third grade students academic gains before and after using the math content of Study Island? Recommendations Based on current evaluation results, it can be recommended that Study Island is a remedial support for mathematics to be continued within this public school. Based on this data, one can assume the Study Island would be successful, if used similarly, in other public school systems as a form of remediation. 1. Teacher surveys reveal indicate 88% are satisfied with Study Island as a user-friendly program (Appendix E). 2. On a scale of 1-5 (5 being the greatest), 88% of teachers rated Study Island as 2-3 for effectiveness. 71% of teachers surveyed that they only used the program 3-4 times a week as 57% reported using results to drive instruction (Appendix E). 3. It is recommended that staff be properly trained on how to use Study Island within the classroom in order to improve student success as well as student CRCT scores during this high stakes year. With the new Common Core standards, it is recommended that the staff training program includes how to implement Study Island to support the Common Core standards. 4. It is also recommended that the subscription to Study Island continue; however, students are

encouraged to use the program on a daily basis to maintain optimal success. 5. It is recommended that students are offered incentives for successfully mastering the standards and receiving blue ribbons (for mastering a skill). Introduction

The purpose of this evaluation is to determine if the online Study Island program, specifically the within the content area mathematics, is a successful remediation program to improve student academic achievement within the third grade setting. Study Island is a web based program that targets the Common Core and Georgia Performance standards. Study Island provides lessons as well as practice activities and incentives to earn Blue Ribbons in all content areas. Each grade level requires a specific number of lessons (Appendix D) to be completed in order to cover the standards. Typically, a student must earn a 70% or better on a particular lesson in order to earn a Blue Ribbon. This evaluation will determine if this unique program, with built-in incentives, raises third grade math test scores on the CRCT after remedial use in the classroom and if it is worth the investment. The evaluation is intended to answer the question Is Study Island a program that should continue to be implemented in ABC Elementary school for third grade students who need remediation in mathematics? This evaluation was conducted in this manner to gain a full understanding as to if this program is what it claims to be and if the school should continue the program. The strategies in which data was obtained enables the evaluators to gain feedback from students, faculty, and preferably administrators and parents that include qualitative data and personal opinions. The data automatically generated from Study Island will provide quantitative and instant data for each child to evaluate their usage both in day, time, sessions attempted/completed, number of Blue Ribbons earned, and scores. This will be analyzed meticulously when comparing CRCT scores with pre-test scores and Study Island usage and data.

The evaluation is prepared to evaluate ABC Elementary schools third grade students who use Study Island as a form of remediation in mathematics. Based on information from ABC Elementarys Information Services and the elementary schools Report Card (Appendix A), the school, on average, has approximately 25 students in each third grade classroom. The school consists of 27.3% of Asian students, 17% of Hispanic students, .2% of American Indian/Alaska students, 45.2% of Black students, 5.6% of White students, and 4.5% of multi-racial students. Based on 2012 Georgia Department of Education data, this elementary schools third grade class had a medium score of 841.37 on the math portion of the CRCT (meeting standards). However, 22.6% did not meet standards; although, 34.5% met standards and 42.9% exceeded standards. This evaluations stakeholders are students, administrators, faculty, and parents. The Study Island program financially impacts the public school system; therefore, it is necessary to determine if this program is truly beneficial to the school system. The stakeholders must know if this is a valuable resource to conclude if the program should continue to be invested in financially as well as if it is good use of students and teachers academic days. Although, this evaluation is conducted in such a way that provides valuable information the evaluation still contains some limitations. Personal opinion from questionnaires can misinform the evaluators. Technical error within the Study Island program can also mislead the evaluators. If teachers login to Study Island to remove failed sessions, this too may affect analysis of the data. Therefore, teachers should be either told not to do this or to document the student whom they edit sessions for, why, and the frequency. In addition, this is the first year of high stakes testing for third grade students. Testing anxiety and the unknown of high stakes testing may also influence final CRCT scores; thus, this would impact the evaluation results as well. Since discontinuing CRCT testing in second grade but requiring third grade as a high stakes year, students first experience with the CRCT is a crucial year. Despite the use of Study Island, this new experience and test pressure may skew scores. In addition, teachers are still

familiarizing themselves with the new Common Core standards and Study Island continues to adapt its content to better present these standards to students. This evaluation is not intended to negatively advertise Study Island or discourage the use of Study Island all together. ABC Elementary school can ultimately decided if they want to discontinue the use of Study Island or better train teachers how to effectively use it within their classrooms. Overview of the Report Contents Focus of the Evaluation Description of the Evaluation Object Study Island is a program created by Archipelago Learning in 2000. The program is internet based program that is centered on standard-based learning, practice, and mastering. Study Island provides state specific standardized instructional practice, assessment, and productivity tools to different schools in different regions to improve the performance of educators and students and promote mastery of state standards and objectives. With this program, teachers are given the ability to assess students and track their progress based on their own individualized learning plan. Students are given the opportunity to learn, practice, and review content with games and a variety of engaging activities in an effort to master the standards. The general objective of Study Island is to build the confidence in the students and their understanding of the standards, promote learning by using engaging activities, and raise student test scores on state standardized tests. The test is broken down into specific strands of the state standards, and it allows students to practice the content and move along at their own pace. After each lesson, students will receive feedback and explanations for the answers. Because the program is web-based, students are able to work and practice using Study Island outside of school as well. Student practice content should be added daily, so that students will always have a variety of practice questions to help them master the standards.

Evaluation Questions used to Study the Focus This particular school district purchased Study Island in 2008. However; ABC Elementary implemented the program in 2010 to help students through remediation efforts to master Georgias specific standards and improve CRCT scores. The program has been implemented as a remediation tool that teachers are required to use throughout the school year to help students reach grade-level goals. This evaluation will help answer the following evaluation question. 1) What are students academic gains before and after using the math content provided within Study Island? (Quantitatively) Knowing the answer to this question will help the school system determine whether or not they should continue their subscription to Study Island. This will help them to realize if the program is actually increasing student scores and promoting academic gains or is little or no change has been made at all.

Information needed to complete the evaluation In order to answer this question, we would need to use student test scores. A pre and post test (before and after use of the program) would be needed needed to add validity to the research. The scores would need to be compared to see if student test scores increases after the implementation of Study Island. This is also under the assumption that no other programs were in place to remediate students in the subject of math.

Brief Overview of Evaluation Plan and Procedures

To analyze the benefits of the Study Island program information including login information (Appendix B), student achievement, and personal opinion was collected through

questionnaires, surveys, Study Island login reports, pre-test scores, and CRCT results. The nature of the evaluation process enables quantitative and qualitative data to be gathered which takes into account personal opinion of those that this program impacts the most. The challenge of this evaluation is the length of time required to complete the formal evaluation and maintaining the same focus group as well as the same staff. This evaluation is not extremely financially stressful due to the ability to reach the target audience within the school setting. In order to collect qualitative data, questionnaires were conducted. These questionnaires included requiring staff to rate their opinion of the program, opinion of successes/failures that they have observed, usage, and the programs impact (professionally or personally). In order to collect quantitative data, the Study Island login reports (Appendix B) provided information such as frequency of use (broken down by student name), session attempts and grades, presentation of lesson, time spent in each lesson/session, and Blue Ribbons (grades 70% or higher) earned for a specific period of time. The students would complete a preassessment, in a similar CRCT type formatting, at the beginning of the year. The students identified as at risk would begin the remedial Study Island program. Then, the students would complete teacher-generated assessments within the Study Island program throughout the year. Finally, the students would complete the 3rd grade CRCT in mathematics. Finally, all of these scores would then be compared and analyzed and contrasted with peers scores that were not considered at risk in order to reflect upon the impact of the remedial program provided by Study Island.

Presentation of Evaluation Results Study Island generated information would be compared with with test scores (pre-test scores, pre-Study Island CRCT scores, and post-Study Island CRCT scores) to determine the success of the program. The data pre-Study Island, mid-Study Island, and post-Study Island

(CRCT) would be analyzed to see correlation between the programs usage and its success. Results would be provided to stakeholders, administrators, faculty and parents. Conclusions and Recommendations Program Strengths User friendly for teachers, students, and parents Simple to progress monitor student progression Simple to differentiate remediation Generates login information easily Engaging for students Involves very little preparation work for teachers When used appropriately, students math CRCT scores increase in correlation to Study Island usage. (assumptively) Automatic feedback and instruction to students Boosts self-confidence among students Standards based

Program Weaknesses Students interaction with computer more so than teacher Cost Students temptation to quickly complete a lesson without mastering the content Easy for parent to complete online assignments for his/her child Common Core & GPS merging is a weakness for both teachers the the Study Island program

Recommendations To improve students CRCT test scores in math as well as facilitate higher student achievement in the subject, the following recommendations are given.

1. The school system should set a standard for the number of hours per week students are required to use Study Island for remediation purposes. In order for the program to be more effective in impacting test scores, students need to have access to the program consistently. Students who need remediation in the subject should practice on a daily basis to strengthen their skills in math and master the standards. 2. The school system should implement an incentive program that rewards students for

earning Blue Ribbons and successfully mastering standards on Study Island. By creating an incentive program students will be motivated to try their best in school as well as at home. Using Study Island on a daily basis may become redundant to the students so by giving them something to work towards they will be more engaged while learning and mastering math standards. 3. Teachers should be required to complete proper training modules on how to use Study Island. To increase the programs effectiveness, teachers should know how to use the program properly. Not all teachers felt that the program was user-friendly. This may have led to misuse of the program thus creating little impact on the test scores. 4. Younger students (2nd grade) should have access to Study Island and be required to use the program for practice. Due to 2nd graders no longer taking the CRCT, it may benefit the school system if they offered the program to younger students so that they could have more experience with practicing test questions. With younger students having more exposure this program, it may help to improve test scores when they reach the upper grade levels (3rd, 4th, 5th).

Appendix A ABC Elementary 3rd Grade Math CRCT Results 2010, 2011, 2012

2010: 16.3% did not meet in math 2011: 10.3% did not meet in math 2012: 22.6% did not meet in math

Statewide Score Reports: http://www.doe.k12.ga.us/Curriculum-Instruction-andAssessment/Assessment/Pages/CRCT-Statewide-Scores.aspx

Appendix B Study Island Login Report Example

Appendix C Study Island 3rd grade Math Question Example

Appendix D Study Island 3rd grade Math Lessons

Appendix E Faculty Questionnaire Results (a select grade level team) 2013

References

Unknown. (2012). 2011-2012 accountability report card. Retrieved from http://www.clayton.k12.ga.us/facilities/schools/AcctRpts/115.pdf Unknown. (n.d.). Crct statewide scores. Retrieved from http://www.doe.k12.ga.us/CurriculumInstruction-and-Assessment/Assessment/Pages/CRCT-Statewide-

Scores.aspxacilities/schools/AcctRpts/115.pdf