Program Evaluation Final Report

Lara Komanecky & Lisa Witteman

Program Evaluation- Final Report
Georgia Southern University
FRIT 8435 Program Evaluation
Dr. Carlson
Spring 2013

Program Evaluation Final Report

Table of Contents
I.

Executive Summary.4
Purpose and Scope .4
Audience.4
Data Collection and Analysis4
Findings..5-6
Recommendations6

II.

Introduction..6
Purpose of the Evaluation..7
Design of the Evaluation..7

III.

Methodology
Methods of Data Collection..7
On-Task Behavior Chart..7-8
Picture Survey.8-9
Pre-and Post Modified KWL for Sixth Graders..9
Photo Interview Questions for Elementary Students10
Interview Questions for Sixth Grade10
Limitations of Evaluation..11

IV.

Results..12
Evaluation Findings12
Pre and Post -Sixth Grade Modified KWL Intrinsic and Extrinsic Findings.12

Program Evaluation Final Report

SEN K-2 Visual Intrinsic and Extrinsic Survey Findings13
On-Task Behavior Findings..14
Photo Conference Questions Findings..15
V.

Conclusions and Recommendations15

VI.

References..16

Program Evaluation Final Report

I.

Executive Summary
Purpose and Scope
Interactive white boards (IWBs) are used in almost every classroom in this school district. They
are used across content areas and have a variety of applications and uses. IWBs are currently
used to promote learning in K-6 reading programs. The scope and purpose of this evaluation
was to determine whether IWBs impact students attending behaviors and organizational
strategies in K-6 reading programs. Student behavior logs, student interviews, SEN surveys and
Photo wh questions were used to determine effectiveness of the IWBs.
Based on the percentage data results of this evaluation, we found that the use of interactive
white boards in K-6 reading programs does improve students attending behaviors and
organizational strategies; therefore, this program should be maintained.
Audience
This evaluation took place from February 22, 2013 to April 5, 2013 in the KW School System and
seeks to inform the stakeholders at KW School System on the impact of IWBs on K-6 reading
programs.
Data Collection and Analysis
Data were collected through behavior logs, interview sessions, self-reporting and surveys of 58
students participating in K-6 reading programs. Behavior logs were tracked during white board
segments, while daily observational data was collected by the evaluator. All of the evaluation
participants were observed for attentiveness during their academic period. In the study,
attentiveness was identified and operationally defined as active learning (i.e., looking at the
speaker, focusing on the whiteboard, utilizing books, clipboards/dry erase boards, manipulating
props and materials). Students off task behavior was identified and operationally defined as
disengaged (i.e., playing with clothes, shoes, distracted, looking up at the ceiling, etc.).
Attention to task was discussed at the beginning of each lesson and defined by the students as
well. Data was recorded on a 5 minute-interval tally chart during a thirty-minute lesson. Active
learners received a plus sign (+) and disengaged students received a minus sign (-). Weekly
conferences were held for general education students, during which students completed photo
interview questions assessing information retrieval. Participating sixth grade students
completed pre and post evaluations to determine if there were correlations between IWBs and
students utilization of organizational strategies. Participating AU students took part in a three
session visual survey to determine if there were correlations between IWBs and students
utilization of organizational strategies. Data herein is compiled into several charts, displaying

Program Evaluation Final Report

results. These research methods were used with both an AU self-contained group of students
and an inclusion group.

Findings
The findings of the evaluation are listed below, categorized into four areas.
Student Behavior Logs
*(IWB = Interactive white board, WWB=with white board & WOWB=without white board)

Daily 30 minute behavior logs were utilized by observers, at 5 minute intervals for six
weeks to determine students on-task behavior. Both Group A and B were assigned
numbers for student identification. Data on all eight students in Group A were recorded
daily. Data on the 48 students in Group B was randomly taken due to class size. The
qualitative data was tabulated weekly to measure on task behavior with an IWB and
without IWB with a baseline of zero and a goal of 85%. Time on task was reported
higher with the IWB in both the AU self-contained group (WWB=72%, WOWB=65%) and
the inclusion group (WWB=78%, WOWB 52%). Although the researchers did not make
their intended goal of 85%, there was active growth in each group. These data
fluctuations were attributed to outliers of age and development of students, class-size,
pre CRCT and post CRCT variables, and prior knowledge fluency.
Interviews
A modified version of the KWL format was employed before and after the study with the
sixth grade inclusion group to measure the impact of IWB lessons on students
confidence and expression of detail and interest. The qualitative interview questions
were intrinsic and extrinsic in nature, and open-ended. The responses were rated on a
3-point rubric scale (1 little detail, 2- some details, 3 very detailed). Pre-Evaluation
results were; averaged intrinsic responses, (1.75) and averaged extrinsic responses,
(2.25). Post-Evaluation results were; averaged intrinsic responses, (2.5) and averaged
extrinsic responses, (2.66). The results were compared and a percentage growth rate
was tabulated. Intrinsic responses increased by .75 and extrinsic responses increased by
.16. These data validated that IWB lessons contributed to students increased
confidence and expression of detail and interest.
SEN Survey
A qualitative SEN survey was used with the AU self-contained group. The survey was
visually designed (the employment of visual answers in place of verbal or written
responses), to measure how the IWB influenced their intrinsic and extrinsic motivation
in correlation with their implementation of organizational and retrieval strategies. An
emotion survey rating scale (1-4) was employed for the intrinsic questions (1-2) and
numbers were assigned for the students responses for the extrinsic based questions (3-

Program Evaluation Final Report

6). These data detected the students underdeveloped expressive and receptive
language skills, which is common in their pervasive disorder, in their extrinsic based
responses. Not unexpectedly, the percentages for extrinsic based responses yielded < .5 regression. In their intrinsic based responses, a significant negative correlation was
discovered in the 1.75 percent growth rate. These data percentages displayed the
participants dislike for conventional means of work tasks and an enjoyment for work
tasks on or including the IWB. The data validates that the employment of the IWB
impacts students intrinsic motivation, which theoretically, enables them to implement
organizational and retrieval strategies.
Photo wh Questions
The study included utilizing photographs to assess students engagement and retrieval
process during IWB lessons. Once a week students were shown a picture of their class
during an IWB reading large group lesson. All students were asked a combination of
questions, some open-ended and story element questions. They were instruction to
give detailed responses. The responses were rated on a 3-point rubric scale (1 little
detail without explanation, 2- some details without explanation, 3 very detailed with
descriptive explanation). The results were averaged each week and those averages were
divided into six. The final results were Group A (2.25) and Group B (2.5). Interestingly,
both groups remained relatively consistent with neither displaying a significant increase
or significant regression. With Group A, the photo questions showed that, although the
students could recognize themselves in the pictures and familiar stories, they were
unable to express detailed accounts. With Group B, the students exhibited engagement,
maturity and enjoyed detailing their photos.
Recommendations
1. IWB lessons should be implemented in K-6 reading programs, if they are not already.
2. K-6 reading programs already incorporating IWB lessons should do so at an increased level.
3. KW School System should encourage increased use of IWBs in reading programs.
4. Research should be continued to determine the effectiveness of IWBs on other academic
programs.
II.

Introduction
The majority of classrooms in KW School District contain Interactive White Boards (IWBs). These
white boards are often used in reading programs. This program evaluation sought to determine
the effectiveness of IWBs in K-6 reading programs. Two evaluation questions drove this study:
a. Can the incorporation of an interactive white board impact students attending
behaviors during guided reading?
b. Can the incorporation of an IWB increase students (general education and SWD)
utilization of organizational strategies?

Program Evaluation Final Report

Purpose of the Evaluation

Stakeholders agreed that the impact of IWBs should be evaluated in K-6 reading programs.
These findings will be shared with faculty and staff to determine future use of IWBs in K-6
reading programs, as well as the use of IWBs in other content areas.
Design of the Evaluation
Two student groups were included in this study. Group A consists of eight self-contained autistic
students. Group B consists of 48 students in inclusion classrooms. Of the 48 students in group B,
19 receive gifted services, 10 receive special education services and one receives both gifted and
special education services. Data collection methods were the same for both groups. Behavior
logs were tracked during white board segments, a modified version of traditional KWL (what you
know, what you want to know and what you have learned) was utilized for interview questions
(Group A was given super symbols to answer interview questions), and photo questions were
used with both groups.
III.

Methodology
Methods of Data Collection
Data were collected from February 22, 2013 to April 5, 2013. Behavior logs were tracked for the
duration of the study, six interview sessions took place from March 1st to April 5th, self-reporting
for sixth grade students took place on February 22nd and April 5th, and SEN surveys were
distributed three times, March 1st, March 15th and March 29th.
Figure A. -On-Task Behavior Chart
The chart below was used to record on-task behavior with and without an IWB on all
participants.

With IWB
Days of the Week

Number __ +/-

Number __ +/-

Number __ +/-

Number __ +/-

Number __ +/-

Number __ +/-

Number __ +/-

Number __ +/-

Number __ +/-

Number __ +/-

Monday
Tuesday
Wednesday
Thursday
Friday
Totals
Average

Without IWB

Program Evaluation Final Report

Days of the Week
Monday
Tuesday
Wednesday
Thursday
Friday
Totals
Weekly Average

Figure B- Picture Survey

The picture survey below was used for elementary students.
Picture Survey Questions:

Student Number____

Time______

Date________

When I use scissors to cut out story pictures then glue stick to glue them in a square, I feel.

When I use the white board and pen to move story pictures to a square, I feel..

I can find the title of a book and the author using a

I can see the bigger words, story characters, story settings, when I look at

When I watch stories on the white board and see the big words and pictures, I

Program Evaluation Final Report

It is easier to use my story checklist

at my seat looking at the story on the whiteboard or book

Figure C - Pre-and Post- Modified KWL for Sixth Graders

Prediction and What I Know- Bubble Head

(Intrinsic)

Prediction and What I Know-verbally respond

(extrinsic)

* Will you learn anything using the interactive whiteboard

(IWB)?
1 little detail
2- some details 3 very detailed

Who would you tell in another school: a teacher/child to use

the IWB?
1 little detail

2- some details

3 very detailed

What new reading skill will learn a using the IWB?

1 little detail

2- some details

3 very detailed

Do you think other children/teachers/parents learn with the

IWB?

Will you be patient taking turns at the board?

1 little detail

1 little detail

Do you think using the IWB for reading will be a good

idea?
Why?
1 little detail
2- some details 3 very detailed

2- some details

3 very detailed

Do you think the IWB will change the way you think
about reading? How?
1 little detail

2- some details

3 very detailed

2- some details

3 very detailed

Do you think it wont be fun to use the IWB? 1 little

detail
2- some details 3 very detailed

Have you ever put your own work on the IWB?

1 little detail

2- some details

3 very detailed

Would you feel embarrassed to show anyone your work on

the IWB? Why? 1 little detail
2- some details 3
very detailed
How do you feel about working on the IWB? Why? 1
little detail
2- some details 3 very detailed
What grade would learn the best using the IWB?1 little
detail
2- some details 3 very detailed
*How did answering these questions make you feel?
1 little detail
2- some details 3 very detailed

Program Evaluation Final Report

Figure D- Photo Interview Questions for Elementary Students

Interview questions Elementary

Student Number:____ Time:_____ Date:_________
Candid pictures of the students during their reading segment were employed to ask the following
questions.
What are you thinking in this picture?
1 little detail (Student did not remember specific details)
2- some details (Student remembers 1-4
specific details)
3 very detailed (Student remembers and describes specific details)
Do you remember the name of the story?
1 no title 2- half title 3 full title
What was your favorite part of that story? Why?
1 little detail without explanation
descriptive explanation

2- some details without explanation 3 very detailed with

Total_____ Average______

Figure E - Interview Questions- 6th grade

Interview questions-6th

Student Number:____ Time:_____ Date:_________

What are you thinking about in this picture?

1 little detail

2- some details

3 very detailed

When you look at the picture: Do you remember the day off the week?

1 little detail

2- some details

3 very detailed

2- some details

3 very detailed

What lesson is it?

1 little detail

What is going on in the story?

1 little detail

2- some details

3 very detailed

Who was the main character?

1 little detail

2- some details

3 very detailed
Student Number:____ Time:_____ Date:_________

Program Evaluation Final Report

Limitations of the Evaluation

Although IWBs are used in various programs throughout the school, this study focuses on their
effectiveness in reading programs only. The number of student participants in this study is also
limited. Due to time constraints, a larger number of students could not be assessed. Data
analysis tables were not preformed due to the limited sample population and relevant past
research was given honorary mention in the appendix for inspiring and guiding the study, but
not directly cited in the evaluation. Due to time constraints, a larger number of students could
not be assessed.
IV.

Results
Findings
Research showed that IWBs positively impact students attending behaviors and organizational
strategies, as displayed in Pre-and Post-Sixth Grade Modified KWL Chart, SEN K-2 Intrinsic and
Extrinsic Survey chart, on-task behavior chart and photo questions survey chart, found in the
appendices.

Program Evaluation Final Report

Figure F - Pre and Post -Sixth Grade Modified KWL Intrinsic and Extrinsic Findings

3,2,1 Averaged Rubric Scores

Pre and Post Sixth Grade KWL Results

3
2.5
2
1.5
1
0.5
0
1
2

First and Last Week

Pre

1
1.75

2
2.5

Post

2.25

2.66

The Pre and Post KWL Interview data were utilized to determine IWBs impact on students
personal and academic (intrinsic and extrinsic) organizational strategies. These data illustrated a
positive correlation between students intrinsic and extrinsic organizational strategies with the
use of IWBs in reading programs.

Program Evaluation Final Report

Figure G - SEN K-2 Visual Intrinsic and Extrinsic Survey Findings

SEN K-2 Visual intrinsic and Extrinsic

Survey

3.5

3,2,1 Averaged Rubric Scores

3
2.5
2
1.5
1
0.5
0
1
2

Number of Survey Weeks

Group A SEN Intrinsic

1
3.25

2
3

3
3.5

Group A SEN Extrinsic

1.5

1.5

Elementary students intrinsic scores significantly improved with the utilization of IWBs in their
reading programs. These data can signify that the students are personally invested in their
academic tasks.

Program Evaluation Final Report

Figure H - On-Task Behavior Findings

Averaged Percentages - Task On Task

Time On Task - with and without IWB

1

0.5

0
1

Week Number

Group A NWB

1
0.56

2
0.63

3
0.59

4
0.65

5
0.71

6
0.74

Group A WB

0.62

0.67

0.73

0.77

0.75

0.81

Time on Task:
Group B
without IWB

0.4

0.33

0.55

0.65

0.7

0.5

Time on Task:
Group B with
IWB

0.75

0.7

0.7

0.85

0.9

0.7

All groups showed an increase in on-task behavior with IWB lessons.

Program Evaluation Final Report

Figure I - Photo Conference Questions Findings

Photo Conference Questions

3
2.5
3,2,1 Averaged Rubric
Scores

2
1.5
1
0.5
Group A

0
1

Group B

Week Number

V.

Group A

1
2

2
2

3
2

4
2

5
2.5

6
3

Group B

2.25

2.5

2.25

2.75

2.75

2.5

Conclusions and Recommendations

Because research showed improved on-task behavior and improved organizational strategies
with the incorporation of IWBs into K-6 reading programs, the following recommendations are
made:
1. IWB lessons should be implemented in K-6 reading programs, if they are not already.
2. K-6 reading programs already incorporating IWB lessons should do so at an increased
level.
3. KW School System should encourage increased use of IWBs in reading programs.
4. Research should be continued to determine the effectiveness of IWBs on other
academic programs.
5. Research should further explore the correlation between IWBs utilization in the
classroom and its impact on students beliefs about learning and metacognition.
Successful implementation of this study, in conjunction with past research and other school
improvement efforts, promises positive correlation with IWBs and the students initiation and
continued self-regulation of attending (active learning) during lesson time, which research
states, improves students academic achievement and self-efficacy.

Program Evaluation Final Report

Reference Page
Honorary Mention
The following are articles that supported and gave perspective to the Evaluation Study.

Carnahan, C., Williamson, P., Hollingshead, A., & Israel, M. (2012). Using technology to support
balanced literacy for students with significant disabilities. Teaching Exceptional Children,
45(1), 20-29
Frauenberger, C., Good, J., & Keay-Bright, W. (2011). Designing technology for children with special
needs: bridging perspectives through participatory design. CoDesign, 7(1), 1-28. doi:
10.1080/15710882.2011.587013
McKenna, M., Labbo, L. & Reinking, D. (2003) Effective use of technology in literacy Instruction, in: L.
Morrow, L. Gambrell & M. Pressley (eds), Best Practices in Literacy Education, 2nd edn. (New
York, The Guilford Press).
Solvie, P. (2007). Leaping out of our skins: postmodern considerations in use of an electronic
whiteboard to foster critical engagement in early literacy lessons. Educational Philosophy and
Theory, Vol. 39, No. 7, 2007. doi: 10.1111/j.1469-5812.2007.00312.x