ThinkCERCA Program Evaluation Worksheet

I. Program Information

Date: April 1, 2018

Program name: ThinkCERCA

Applicant name: Nickey Grandea

Contact person: Laura Holmes

Phone/e-mail address: 224-412-3722

II. Program Summary

Program purpose:
The purpose of the ThinkCERCA is to improve student reading, writing and critical thinking by addressing five
specific areas (CERCA): C – Claim, E - Evidence, R – Reasoning, C – Counterargument, A – Audience. Each
lesson includes reading concepts and skills, comprehension, and/or argumentative writing.

Program goal:
At Whitewater Middle School, the ThinkCERCA program will be implemented with fidelity to achieve the goal
of increasing student achievement in reading and writing by 1.5 to 2.5 grade levels during one academic year.
According to research, eighteen to twenty-three lessons must be completed each year in order to see the greatest
benefit (Pratihast & McGhee, 2016).

Program objectives:
1. By August 17, 2018, academic team teachers at Whitewater Middle School (WMS) will participate in
professional development on implementation of ThinkCERCA in the classroom with at least 85%
reporting a moderate to high degree of satisfaction with the knowledge gained.
2. By September 28, 2018, academic teams will utilize ThinkCERCA lessons at the rate of at least two per
month.
3. By October 31, 2018, students at WMS will engage in active reading and writing on ThinkCERCA, as
measured by interaction with the reading and writing portions of the program, for at least twenty-five
minutes each week.
4. By May 24, 2019, students at WMS will complete a minimum of eighteen ThinkCERCA lessons.
5. By the end of the academic school year (May 24, 2019), student achievement in reading and writing will
increase by 1.5 to 2.5 grade levels, as measured by comparing the previous and current academic school
year Milestones Test Scores.

Program description and primary activities:

ThinkCERCA is an online reading and writing program. Students complete a variety of reading and writing
activities to improve their ability to read closely and compose a written argument with evidence to support their
claims. These activities include direct instruction, reading comprehension, identification of evidence from the
text, and argumentative writing.

Page 1 of 5
III. Evaluation Plan Overview

Evaluation classification:
This evaluation will be conducted as an internal, summative evaluation with both process and outcome
evaluation questions. This evaluation will evaluate the implementation of the program, the activities of the
program, and measure the effectiveness of the program in producing the intended short-term outcomes.

Role of the evaluator:

The internal evaluator developed the evaluation plan and will guide the evaluation process. Data collection will
be carried out by county level administrators who have access to the data reports from the ThinkCERCA
program and Milestones EOG testing data. Building administrators will distribute surveys to teachers, students,
and parents in regards to attitudes toward reading. These surveys will be analyzed by the evaluator.

Evaluator qualifications:
The evaluator has a M.Ed. in science education and is currently enrolled in a specialist degree program,
majoring in instructional technology. She has completed coursework in program evaluation and is guided by a
UWG faculty member.

Evaluation timeline and completion of evaluation report:

The evaluation will take place over the course of approximately 18 months. During the initial 6 month time
period, the evaluation plan will be finalized, data collection tools will be identified (and created if necessary),
and administrators will be trained regarding data collection. During the following 9 months, the activities of the
program will be carried out, and data will be collected regarding this implementation. The final summative data
will be collected and analyzed to determine the effectiveness of the program. In the final 3 months of the
evaluation process, the evaluation report will be prepared and shared with stakeholders.

Participants who developed the evaluation plan:

Nickey Grandea -- Evaluator

This evaluation plan aims to serve the following purposes in addition to meeting funding requirements:
 To determine whether the ThinkCERCA program is implemented consistently
 To assess ThinkCERCA program short-term outcomes
 To use as basis for continued use of the program

Projected use of findings:

The process evaluation findings will provide information regarding the implementation of ThinkCERCA
activities and allow school personnel to make improvements to the implementation as needed.
The outcome evaluation findings will allow stakeholders to determine whether ThinkCERCA provides the
results reported in the research. This information validates continued use of the program or discontinuation of
the program.

IV. Audiences

Primary stakeholders for this evaluation:

 County level administrators
 School level administrators
 Teachers
 Students
Page 2 of 5
  Parents

Use of evaluation results for these primary stakeholders:

County level and building level administrators must determine whether to continue funding of the
ThinkCERCA program for another academic school year. In order for teachers to truly buy-in to a program,
they must see the benefits to their students. The outcome finding will provide the information that teachers
need to develop trust in the program. Students and parents must also trust that the program provides
improvement in student learning.

V. Evaluation Questions

Key evaluation questions to be answered by this evaluation:

Process Evaluation Questions:

 To what extent has the ThinkCERCA program been implemented with fidelity (2 lessons per
month/academic team and 25 active minutes per week/student) across the county?
 To what extent have student attitudes toward reading and writing improved?

Outcome Evaluation Questions:

 To what extent did using the ThinkCERCA program produce improvement in student reading and
writing test scores?

VI. Evaluation Design

Summary:
This evaluation will utilize a quasi-experimental (pretest/posttest) design. All teachers and students in middle
schools across the district will participate in the usage of the ThinkCERCA program. No control group is
necessary in this design because student data will be compared pretest to posttest, allowing the evaluator to
determine the extent to which each student and/or classroom group has improved from one year to the next.
The Georgia Milestones EOG assessments will be utilized as a source for this information.

ThinkCERCA usage reports will be monitored monthly to determine the extent to which the program is
implemented with fidelity across the county. In order to implement with fidelity and achieve maximum results,
students must complete at least 2 lessons per month and spend 25 minutes per week actively engaged in the
reading and writing portions of the program.

The survey data collected will be from randomly chosen students across the district. Once informed consent is
received, teachers, students, and parents will participate in surveys monthly to determine the extent to which the
ThinkCERCA program is impacting the student’s enjoyment in reading.

Data types:
Both quantitative and qualitative data will be collected in this evaluation.

For the process evaluation questions, data will be collected from ThinkCERCA usage reports, specifically the
number of lessons students complete and the number of active minutes students spend in the program on a
monthly basis. To determine the impact that ThinkCERCA has on student enjoyment of reading, survey data
will be collected. This data may be both quantitative and qualitative.
Page 3 of 5
For the outcome evaluation question, data will be collected from the Georgia Milestones EOG assessments.
These assessments will serve as the pretest and posttest assessments. This data is quantitative in nature.

Ethical considerations for this evaluation:

Although this evaluation involves collection of data from human subjects (including teachers, students, and
parents), UWG IRB approval of the evaluation plan is not required because this type of evaluation is exempt.
According to the UWG_IRB-Guidelines.pdf, “Research conducted in established or commonly accepted
educational settings, involving normal educational practices, such as (i) research on regular and special
education instructional strategies, or (ii) research on the effectiveness of or the comparison among instructional
techniques, curricula, or classroom management methods. Though the research is exempt, the investigator has
an ethical obligation to respect and safeguard students’ rights and welfare” (p. 12). Informed consent will be
obtained from all adults involved and from parents of students who will participate in the survey. To ensure
privacy and security, Qualtrics will be utilized for survey data because it meets the University Security
Standards. The other types of data collected are existing data and do not require approval. Identities of all
participants will remain secure through district security measures, and survey data will be anonymous.

VII. Data Collection – Methods and Instruments

Proposed data collection methods:

Process Data: Usage reports and survey questionnaires
Outcome Data: Quasi-experimental Pretest/Posttest Data

Instruments to be used:
ThinkCERCA usage reports (number of lessons per month per student; number of active minutes per week per
student)
Surveys created by evaluator regarding program satisfaction administered to teachers, students, and parents
Pretest/Posttest Data from Milestones EOG testing

Person(s) responsible for data collection:

County and building level administrators
Evaluator

Data collection timeline:

Process data will be collected monthly.
Outcome data will be collected at the beginning and end of the school year.

VIII. Data Management and Analysis

Data management:
All data will be managed digitally through Google Sheets, Qualtrics survey data website, existing county data
systems, and through the ThinkCERCA program.

Data analysis strategies:

Initial descriptive statistics (including mean, median, mode, and standard deviation)
Inferential statistics and regression analysis (including t-test)

Person(s) responsible for data analysis:

Page 4 of 5
Data will be analyzed by the evaluator.

IX. Strategies for Using Evaluation Findings

Reporting:
The evaluator will prepare a final evaluation report to share the process and outcome findings and
recommendations with the stakeholders and other community members. These stakeholders include county
level and building level administrators, teachers, students, and parents. The report will be made available on the
county website with the evaluator’s contact information for any questions that arise.

Evaluation debriefing:
An evaluation debriefing will be presented to the County School Board and other stakeholder representatives.
The debriefing will include a review of the evaluation in its entirety, including findings and recommendations.

Post-evaluation action plan:

A post-evaluation action plan will be prepared by the school system’s representatives and will include
recommendations provided by the evaluator and members of the evaluation debriefing group (including school
board members, stakeholders, and community members). This plan will provide information about
improvements that are needed in the implementation of the program and the continued use of the program for
the following academic school year.

X. Attachments
Logic Model

References

Guidelines for the review of research involving human subjects. Retrieved from
https://www.westga.edu/academics/research/orsp/assets/docs/UWG_IRB-Guidelines.pdf

Pratihast, S., & McGhee, D. (2016). Literacy instruction with ThinkCERCA (Rep.). Retrieved from
https://cdn2.hubspot.net/hubfs/2442542/Content_Downloads/Case_Studies/ThinkCERCA_Network_1_
ARP_Case_Study_2016.pdf

Page 5 of 5