Group 11 Eval Report 04-30-17

Running Head: EVALUATION REPORT: MOSES VOCATIONAL PATHWAYS PROGRAM 1
Evaluation Report:
Moses Vocational Pathways Program
J. B. Campbell, Dane F. Forlenza, Tammy A. Revelle, Megan O. Standley
University of West Georgia

EVALUATION REPORT: MOSES VOCATIONAL PATHWAYS PROGRAM 2
Background of the Program and Evaluation
Program Description
The Moses Vocational Pathway program (MVP) is the creation of Tammy Ponder, the
Evaluations and Assessment Coordinator at Moses Middle School in Dallas, Georgia. While
completing research for her Specialist in Educational Leadership, Mrs. Ponder discovered
evidence that student graduation rates increase when students have clear and defined career
goals. According to Hooley, Marriott, and Sampson (2011), career development programs
have an important role to play in engaging students in their education and diminishing dropout
rates (p. 13). Their claim is supported through qualitative research of 33 high school
sophomores who were at risk of dropping out of school, and found that career guided
assistance for these students had a statistically significant impact on attainment and retention
(p.12) of the students within that school (Hooley, Marriott, & Sampson, 2011). Having a career
goal helps learning become relevant to students as it becomes the means by which they can
attain their goal. Students who are "undecided" have more of an apathetic relationship with
school and as a result are less likely to become engaged in the learning process.
Based on her studies, Mrs. Ponder wanted to create a program that allowed students to
begin to explore career pathways, have a better understanding of what the various careers
entail, and begin to create a plan for how their success in school will start them on their path
towards a career. She also wanted to have the community vested in the school and the school
vested in the community. In August of 2015, Mrs. Ponder introduced her idea for the MVP
program to the Moses faculty. At this time teachers were asked to think about what career they
would have chosen if they were not a teacher. Based on teacher career responses, each
teacher in the school was assigned a pathway and instructional level. Implementation of the
Moses Vocational Pathways program began shortly after in September of 2015.

In the original implementation, students were divided up by ability levels into different
pathways and classes. Every Wednesday morning, the school class schedule was restructured
to allow students to take part in 40-50 minutes of vocational skills training provided by faculty at
Moses Middle School. Academically at-risk students were given remedial work related to their
pathway, on-grade level students were introduced to a single pathway every nine weeks, and
advanced students were asked to research their current career interest and complete a mini-
capstone project. At the end of the school year there was a career fair served as the
culminating activity and provided the community a chance to support the schools vocational
efforts. Approximately 30 speakers shared hands on activities and/or lectures with students
about the careers they are in.
During post-planning of the implementation year, school administrative staff decided that
some changes needed to be made to the program. Beginning in September of 2016, the ability
level classes were dissolved and additional pathways were added. A new plan was initiated in
an effort to have students to experience all vocational pathways by the end of the eighth grade.
In addition, beginning in 2017-2018 school year eighth graders will be asked to complete a mini-
capstone project to showcase their learning during the community career day.
The MVP program (see the Program Logic Model below) is designed to provide students
with opportunities to connect their current academic classes to possible career choices to in an
effort to raise overall student test scores on the Georgia Milestones tests in ELA, Mathematics,
Science, and Social Studies.
In order to develop self-directed learners, teachers will engage students in hands-on,
relevant activities through the MVP curriculum that draw connections to the problem solving and
critical thinking skills necessary for the modern workforce. The revised program consists of nine
career pathways divided between the three grade levels. Student will rotate through the
pathways every nine-weeks. Students experience lessons and activities in Arts, Audio/Video,
Technology and Communication, Information Technology, Hospitality and Tourism, Agriculture,

Architecture, Healthcare, Finance, Engineering, Government and Public Safety, Marketing, and
Transportation, Distribution and Logistics. Some of the pathway courses tend to be more
hands-on than others. For example, in Architecture, students create structures using skewers
and marshmallows. In Agriculture, students are growing flowers and vegetables while in
Transportation, Distribution and Logistics, students are building model engines. When students
are in a less hands-on activity pathway, they know that they will have a chance to rotate into a
different one after nine weeks.
Program Logic Model - Moses Vocational Program
Evaluation Purpose
The MVP program is a new initiative designed to raise student test scores, however, its
effectiveness has yet to be proven. A formative evaluation is needed in order to determine

whether the program is effective in assisting this goal. The final evaluation will be used to
determine the effectiveness of the MVP program. In addition, the evaluation findings will be
used to guide program improvements in the future. To this end, our evaluation seeks to answer
the following questions which were developed based on Fourniers (1995, p.16) list of Scrivens
general logic of evaluation:
1. To what extent are the skills and concepts taught in each pathway enhancing
student mastery of academic state standards?
2. What level of student engagement do teachers observe during the activities of
vocational classes within the MVP program?
3. How do the vocational skills learned in the MVP program affect student attitudes
concerning future career pathways?
The MVP program is designed to help raise student test scores on the GA Milestones in
the four academic subjects by linking academic knowledge to relevant vocational skills. Our first
question will help determine if there is any increase in test scores that can be attributed directly
to the MVP program. If program stakeholders can identify growth in student learning across the
academic spectrum, it will help justify and validate the program.
Teacher perceptions of the program and its effects on student engagement will be
important information to consider when evaluating the program processes. The second question
is related to both student engagement and academic achievement. An student who is actively
engaged in the class activities is more likely to comprehend and retain academic content and
thus perform better on a standardized test.
The curriculum of the MVP program includes pathway-specific, hands-on activities that
students complete during 9-week sessions. Through these learning activities, students will
develop problem solving and critical thinking skills that can ultimately result in long term goal of
increased scores and passing rates on GA Milestones. The third question seeks to determine
how the program might turn students into lifelong learners which can have indirect implications
on test scores, as lifelong learners invest more time and effort into gaining knowledge
considering they voluntarily do outside of the classroom.
This qualified evaluation team will be responsible for the collection and analysis of the
evaluation data. According to Davidson (2005, p.3) this evaluation would be considered mostly
independent as the majority of evaluators are not stakeholders, however, the evaluation effort
will require the assistance of the administrators of Moses Middle School - Tammy Ponder
(Evaluation and Assessment Coordinator), Diana Johnson (Assistant Principal), and Scott
Viness (Principal). The support of these stakeholders is critical to the evaluation process in our
efforts to collect pertinent data. Also, as Mrs. Ponder is the creator of the MVP program, her
insight into the program is vital to the evaluation. In addition, the evaluation effort will be aided
by pathway teachers at Moses Middle School. They will be surveyed regarding their
understanding of the program and their opinions on student interest, and they will also
implement the student surveys to the participating eighth graders. Finally, the thoughtful insight
of the eighth grade students participating in the MVP program will be essential to an effective
evaluation. They will be surveyed regarding the influence of the program on their choice of
career pathways as they prepare for high school and beyond.
The evaluation effort is being funded by a joint venture between Moses Middle School
and the Paulding County School District. The cost of the evaluation plan will be negotiated
before work begins and details will be included in the evaluation contract (Appendix C).
Evaluation Team
The evaluation team was comprised of the following people: J.B. Campbell, Tammy
Revelle, Dane Forlenza and Megan Standley.
J.B. Campbell is a mathematics teacher at Starrs Mill High School and a technology
innovator within his school, especially in the mathematics classroom. He earned a Bachelors of
Science in Education in Secondary Mathematics from the University of Georgia in 1999, and
has been working at Starrs Mill ever since. He has 17+ years of teaching experience in all
levels of high school mathematics and holds a Masters degree in secondary mathematics
education from the University of West Georgia and is pursuing a Specialist degree in
Instructional Technology also at the University of West Georgia. He has been a member of the
school improvement plan project team twice, both times serving as a data analyst for
information collected from teachers. In addition, Mr. Campbell has earned his gifted certification
in mathematics as well as receiving College Board AP certification to teach Calculus AB and
BC.
Megan Standley is a sixth-year mathematics teacher at East Coweta High School. She
holds a Bachelors degree from Mercer University, a Masters degree in Secondary Education
with a concentration in mathematics from LaGrange College, and is currently working toward a
Specialist Degree in Instructional Technology from the University of West Georgia. Standley has
served as team leader for each of the courses she teaches for the past four years.
Dane Forlenza is an eighth grade social studies teacher at Alton C. Crews Middle
School in Gwinnett county. He has taught for the last nine years in both Gwinnett and Barrow
county schools in language arts, math, and social studies. He earned his Bachelors degree in
Middle Grades Education from the University of Georgia in 2008, and followed that up with a
Master of Science in Post-Secondary Education from Troy University in 2010. Currently, he is
working on getting a Specialist degree in Instructional Technology from the University of West
Georgia and expects to graduate in July of 2017. He is the eClass lead innovator for his school,
and actively provides professional development in instructional technology to the teachers at his
school.
Tammy Revelle has been teaching for seventeen years. She has been employed at by
the Paulding County School District for sixteen years. She is a 6th grade social studies teacher
at Moses Middle School in Dallas, Georgia where the MVP program is implemented. Prior to
her career in education Mrs. Revelle worked for the Georgia Department of Revenue, Alcohol
and Tobacco Division. In this job she was required to collect data on all alcohol distributors and
wholesalers. Mrs. Revelle earned a Bachelor's degree in history in 1991 and a Masters degree
in History in 1994 from the University of West Georgia. She is currently finishing the last
semester of her Specialist degree in Instructional Technology also from the University of West
Georgia.
Methodology
Data and Instrumentation
For evaluation question #1, teacher lesson plans were analyzed for connections
between state academic standards and the pathways curriculum. The student results on the
same state standards were measured using the GA Milestone test quantitative data. The results
from years prior to implementation of the MVP program were compared to results from MVP
years for specific standards tied to the MVP curriculum. At the same time, teacher feedback
concerning the extent to which the MVP curriculum covers the academic state standards was
examined using a mixed-methods survey - Teacher Survey Instrument #1 (Appendix A).
For question #2, current MVP teachers completed Teacher Survey Instrument #1
(Appendix A). The evaluation team began surveying participating teachers at the beginning of
second semester in the 2016-2017 school year. These survey questions asked pathways
teachers to rate the level of student engagement in their classes during pathways lessons on a
numeric Likert scale ranging from disinterested to actively involved. The results from these
teacher surveys were tabulated to create an engagement score for each pathway. The survey
also included three qualitative opinion questions to help form themes about the overall
effectiveness of the program to aid in later improvement processes.
For question #3, students answered survey questions on Student Survey #1 (Appendix
B) concerning their attitudes toward career pathways as a result of participating in the MVP
program while attending Moses Middle School. The evaluation team began surveying students
at the end of second semester of the 2016-2017 school year. Their responses, based on a
Likert scale, provided quantitative data concerning student attitudes about the program and any
effects the program has on future career choices.
For both sets of survey data, an Excel spreadsheet was used along with a survey
instrument, Google Forms, to collect, sort, and organize the collected data into manageable
data sets. Similarly, test score data from Georgia Milestone assessments was delivered in
spreadsheet form, so statistical analysis of the data could be conducted using the functions
found in Excel.
Sampling
For all evaluation questions, we took a purposeful sample that also happens to be
convenient. Only those teachers involved in teaching a vocational pathway were asked to
participate in the teacher survey. Likewise, only the 8th grade students were asked to respond
to the student survey since they are the students who will be entering high school in the fall.
There are close to 60 teachers at MMS and 50 of them teach a vocational pathway course.
There are 550 students at MMS and 180 students in the 8th grade.
Analysis
To determine if the skills taught in the pathways curriculum may be impacting student
achievement scores, the evaluation team with the permission of school administration collected
Georgia Milestone data for 7th grade students from the spring of 2015 and compared equivalent
test data for 8th grade students from the spring of 2016. Using a direct comparison of the
percentages of students who fall into the beginning learning category in 2015 and 2016, a
decrease in the percentage of students who are classified as beginner learners would indicate
positive instructional improvements within the 2015-2016 school year, especially for lower ability
students, that could be attributed to the MVP program. Teachers and administrators alike
provided insight into other instructional adjustments that might also account for any changes in
academic performance for this level student.
For the teacher survey, the six questions were composed of Likert-type responses and
the mode was used as a measure of central tendency. Furthermore, as suggested by Boone
and Boone (2012), since the data can be divided into accept and reject categories, and a chi-
square goodness of fit test was used for inference . The questions posed by this survey with the
Likert-scale responses provided quantitative data. However, the open-ended questions at the
end of the teacher surveys provided qualitative information from teachers about their
perceptions of the MVP program as a whole. The surveys were anonymous in order to protect
participants from any unintended consequences and to ensure their responses are free from
conflicts of interests.
For the student survey, the six questions were composed of Likert-type responses and
the median and frequency of response for each question was scored, as suggested by Boone
and Boone (2012) as well as Sullivan and Artino (2013). Since all six questions reflect the
students perception of the MVP program, the sum of each students total responses comprises
a Likert-scale score and the mean and standard deviation of these scores were calculated, and
a 1-sample t-test was used for inference. Even though Boone and Boone (2012) advise not
combining these two methods of analysis, the administration of Moses Middle School wanted as
much feedback from students about the program as possible. However, the Likert-scale score,
called an Engagement Score, was used exclusively to determine whether or not the program
meets the required standard for successful implementation.
This evaluation used a mixed-methods approach which we define as the simultaneous
use of both quantitative and qualitative research methods. This approach is beneficial to use
since evaluators were able to acquire solid numerical data to compare as well as open-ended
opinion questions that produced themes about the effectiveness of the program as well as
generated ideas on ways to improve the program.

Standards for Evaluation Questions
Using the comparison of data between years of Georgia Milestone tests in each subject
area, a percent difference from one year to the next can be calculate in each subject area
tested. The administration at Moses Middle School would like to see a minimum 20% decrease
in the percentage of students scoring in the beginning learner category on each subject area
test of the Georgia Milestones. Improving the academic performance of students who score at
or near the bottom of the evaluation scale will have a greater impact on their long term
educational goals. Since one of the main goals of the MVP program is to improve these test
scores, improvement at the lower level of ability on these standardized tests could be attributed
to the program.
Using the data from the teacher survey instrument as well as the analysis of this data,
the evaluation team can determine how teachers perceive the engagement of students in the
MVP classes. In addition, this analysis of data will determine to what extent students are
engaged in the overall program. The administration at Moses Middle School would like to see a
Likert-type score that meets or exceeds a median score of 4 for each of the first six question
items of the teacher survey instrument. The administration believes that if the students are not
engaged during the MVP classes, this can adversely impact student achievement. In addition,
the analysis of the open-ended responses from teachers can be used to direct curriculum
changes should consistent themes emerge from their feedback.
Student survey data will be analyzed to determine the effect, if any, of the MVP program
on the attitudes and behaviors of students as they relate to career choices. The administration
at Moses Middle School desires a student Engagement Score from student surveys that meets
or exceeds a mean score of 3.5. Any score less than a 3.5 in the eyes of the administration
would signify that the program has had little to zero impact on the students future career
choices at this time. A score at or above 3.5 will show that the program is having the desired
effect on the students thinking concerning future career opportunities.

According to Marks (2000), an increase in student engagement can be achieved through
more authentic instructional work. Thus, if student survey results do not indicate that students
are engaged in the program, then students are not viewing the program activities as relevant
and authentic to their world. This claim could be disputed or verified by the data from the
teacher surveys. If student engagement is lacking from the perspective of both teachers and
students, then adjustments to the curriculum and program activities could be made. At the same
time, if students report that they are engaged in the program, and teachers are reporting similar
results, then perhaps a connection is being formed between the skills learned in the pathways
program and possible career choices. This conclusion would be reinforced by improving test
scores as students would see the benefits of their academic coursework as preparation for the
skills and education needed for specific career choices.
This final evaluation report will be delivered to the administration of Moses Middle
School at the beginning of the 2017 school year. At any time during the evaluation process,
administrators have the option of contacting the evaluation team through email or via phone
concerning the completion of planned evaluation instruments and participation of the
stakeholders. The evaluation team will not be able to provide updates concerning the results
and implication of data collected until all components of the evaluation plan have been
completed, analyzed, and reviewed. A copy of the final evaluation report will also be delivered to
the Paulding County School District.
Findings
For the purpose of this evaluation, the team used 2015 6th grade Georgia Milestone test
data in all subject areas and compared performance levels to 2016 7th grade Georgia Milestone
test data (see Table 1). In addition, the same 2016 7th grade data was also compared to the
previous 7th grade class and a comparison of performance levels was also made (see Table 2).
The 2016 7th grade class represents the current 8th graders who were surveyed about the
Moses Vocational Program and who have been part of the program for the longest period of
time. Once collected and compared, the Georgia Milestone test data for Moses Middle School
showed some interesting trends among the same class year to year as well as between
previous class groups.
From 2015 to 2016, students performing at the Level 1 ability (beginning learner)
showed a substantial decrease in percentages across all subject areas except in Science;
understand this is a positive change in this case as students who were in Level 1 have now
moved into a higher level. Students performing at Level 2 (developing learner) also showed a
decrease or maintained performance in all but English Language Arts. An increase in Level 3
(proficient) and Level 4 (distinguished) ability levels was also found in most cases, with
exceptions for Level 3 in Science and Level 4 in English Language Arts.
Table 1: Current 8th graders: comparison of 2015 milestone scores (6th grade) to 2016
milestone scores (7th grade)
The results of a comparison between the current group of 8th grade students and the
previous class show an even greater difference between test scores, especially at Level 1 ability
level. In every subject area, the percentage of students scoring in the Level 1 category
decreased from 2015 to 2016. One of the major differences between the two student groups is
length of time involved in the Moses Vocational Pathways program.

Table 2: Current 8th graders: comparison of their 2016 milestone scores (7th grade) to 2015
milestone scores for 7th grade
For the teacher survey, 35 teachers responded to our request to complete a pencil and
paper survey. The first 6 questions from the survey were scored using a Likert scale and the
results are found in Table 3 below. The mode of the data was used to determine the measure of
central tendency even though we had originally planned to use the median score. This was in
response to further guidance from the administration concerning acceptable outcomes of the
survey. A chi-square goodness of fit test was then conducted on each of the first 6 survey
questions to determine if the expectations of the administration were possible given the
feedback from the teachers. The administration of Moses Middle School wanted to see scores
of 4 and 5 from the majority of the faculty (60%), and allowed for a percentage of the faculty
(30%) to score a 3, with only 10% of the faculty scoring a 2 or 1 on any question. In only one
situation, question #3, did the responding faculty achieve the desired mode, but in every survey
question, the probability that the faculty truly feels as stated by the administration is statistically
impossible.
Table 3: Frequency distribution of teacher responses to the teacher survey with chi-square
goodness of fit test. See Appendix A for full questions.
The probability of 60% of the teachers from Moses Middle School ranking their student engagement as a
4 or 5 is statistically impossible per the results of this chi-square goodness of fit test.
The second part of the teacher survey asked participants to answer 3 short, open-ended
questions concerning their perceptions of student engagement in the Moses Vocational
Pathways program. The survey was given using pencil and paper and the evaluation team then
transcribed their answers into a spreadsheet and beginning sorting responses into positive
takeaways of the program, weaknesses of the program, and possible changes for the program.
The responses within each category were then grouped into smaller categories according to
which part of the program was being referenced. The results of this categorization can be found
in Table 4. Overall, the weaknesses far outweigh the positive takeaways and much change (as
well as dissolving the program altogether) was requested.

Table 4: Quantitative Data From Teacher Surveys: questions #7-9
Positive Takeaways Weaknesses Possible Changes
Students make a Students do not see Students need to be

connection between this as a real class grouped by ability
academics and their and do not take it level.
future seriously. Needs based
Students are exposed Because there is no instruction
to different career grade attached, More resources
paths students are not required to improve
motivated to instructional activities
participate. Discontinue the
Majority of the program
students do not see Teachers need to be
the value in learning able to teach a career
about a career at this pathway they have an
age (11-14 years old) interest in.
Teachers have little to
no interest in
preparing lessons for
a fluff class
For the student survey, the evaluation team conducted online surveys through the 8th
grade vocational pathways classrooms using teachers as the facilitators for the survey.
Students were given access to a laptop and a hyperlink to the survey. Ten minutes of
instructional time were used to complete the six question survey and names were not recorded.
The evaluation team had 161 responses and the mean score for each question from all
responses was recorded and classified according to the standards set by the administration.
Table 5: Student Survey Mean Scores. See Appendix B for complete questions.
In addition, using the responses of the 161 students, each student received an
engagement score, which was the average of all their responses to the six survey question. The
mean engagement score for all 8th graders who responded to the survey was 3.5 out of 5 with a
standard deviation of 0.6. Our null hypothesis was that the engagement score of students at
Moses Middle School would be greater than or equal to 3.5. A 1-sample t-test for means with
0.05 significance level was conducted on the sample mean and yielded a p-value of .5, so we
failed to reject the null hypothesis that the student engagement score is greater than or equal to
3.5. Thus, it is statistically possible for the student engagement score to be 3.5 or greater.
To answer the evaluation question, To what extent are the skills and concepts taught in
each pathway enhancing student mastery of academic state standards?, emphasis was placed
on the Georgia Milestone test data. Analysis of the test data shows a trend of improved
performance during the years after implementation of the MVP program, especially among the
lowest level learner in the group. However, this improved performance could be attributed to
other interventions besides the MVP program that are outside the scope of this evaluation.
Teacher survey question #2 does not indicate that program teachers see much of a connection
between the state standards and the skills learned in the MVP program. Teacher survey
question #6 is inconclusive on whether teachers feel the program has a positive impact on a
students academic success. Even though students report learning new skills in the MVP
program, these skills do not necessarily translate to the ones tested by the Georgia Milestone
tests. Based on these findings, even though there has been improvement in test scores after
implementation of the MVP program, there is no true evidence that pathway skills and concepts
taught through the program enhance student mastery of academic state standards. To better
understand if there is true causation within this correlation, further studies would need to be
conducted across multiple years to examine consistencies.
To answer the evaluation question, What level of student engagement do teachers
observe during the activities of vocational classes within the MVP program?, an analysis of the
teach survey was conducted. Based on the results of survey questions #1, #3, #4, and #5 (see
Appendix A), teachers do not feel that students are engaged in the activities of their MVP class.
Even though most teachers feel that students enjoy the culminating activities (question #3), all
other questions related to student engagement show less than positive responses. Teacher
comments from questions #7-9 indicate a lack of student motivation in the class due to the fact
the vocational class is not considered an academic class and no grades are awarded in the
course. The student engagement score generated from the student surveys indicates that
students feel differently than teachers about the program. Students indicate that they enjoy
engaging in the culminating activities and that they see how these skills could be used later in
life. The difference between observed student behavior by teachers and actual engagement of
students in the program classes might have more to do with teacher attitudes towards the
program and less to do with student engagement.
To answer the evaluation question, How do the vocational skills learned in the MVP
program affect student attitudes concerning future career pathways?, an analysis of the student
surveys was conducted. These surveys indicate that even though students enjoy learning about
new careers (question #1), the pathway courses do not increase their desire to learn about
specific careers (question #5) or influence their choice of elective courses in high school.
Obviously, these results are premature as lasting effects of the pathway courses could manifest
later in school or even after high school. However, students did report seeing the importance of
these vocational skills for careers later in life, which indicates a connection was made between
the skills taught and possible careers. In the short term, students surveys seem to indicate that
student opinions about careers change very little due to the MVP program and that the program
has little impact on the course decisions of the future. Further long-term studies would be
necessary to determine whether pathway courses did impact student career choices during high
school and beyond.

The small scope of the study is a limitation to the validity of the study. Analyzing the
effectiveness of a program, such as the MVP program, must be done with awareness to the fact
that it is not generalizable when the study is conducted only in one location and only over the
course of one year. Suggestions to alleviate this can be found in the Recommendations section.
Conclusions
One of the primary goals of the MVP program is to assist the staff of Moses Middle
School in raising the performance level of students on the Georgia Milestone subject area tests.
The administration of Moses Middle School was looking for a 20% decrease in Level 1 students
from 2015 to 2016 in every subject area. This standard was achieved in every subject area
when comparing test data from 7th graders in 2015 and 2016, and was achieved in 3 out of 4
subject areas when comparing 6th grade data in 2015 to 7th grade data in 2016. Whether the
MVP program can be credited with the increase in student performance cannot be determined
from this evaluation, but additional research could be conducted comparing similar data from
other middle schools without the MVP program.
Another goal of the administration for the program is for students to be actively engaged
in determining and developing connections between the vocational skills learned and the
academic classroom. To this end, teachers were given the task of teaching students skills
related to vocational careers that were also connected to their academic standards, and
creating lessons that involved students in developing these skills. The administration of Moses
Middle School was expecting 60% of their teachers to respond with a 4 or 5 to all student
engagement questions and 90% of their teachers to respond to with a 3, 4, or 5. This standard
was not met based on the responses of the 35 teachers surveyed and is statistically improbable
even if all participating teachers had responded to the survey. Teachers responded in the
desired manner for only one question concerning student engagement in culminating activities.
In addition, according to the teacher surveys, many teachers do not view the program as being
influential to the students or having a positive impact on their academic performance. This is an
area of concern for the administration as their teachers are the ones who need to sell the
students on the connection of skills between the classroom and their career choices. Additional
research could be done to determine ways to increase student engagement in the classroom
and as suggested by the teacher surveys, improve teachers perceptions of the program.
Another component of the student engagement measure was the feedback from the
students themselves on the MVP courses. The student surveys sought out student perceptions
about the classes, the activities, and the connections students might make after completing the
courses. These results would be combined into the student engagement score. The
administration of Moses Middle School was expecting a mean student engagement score of 3.5
points or greater out of a total possible score of 5 points. Based on the 161 8th grade students
surveyed, the mean student engagement score was indeed 3.5 and based on the results of the
1-sample t-test, the school population student engagement score could have an engagement
score of 3.5 or higher. This standard was met based on the analysis of the results of the student
survey. This result is contrary to the teacher perceptions of the program in that students view
themselves as engaged in the courses and see a connection between the program skills and
their academic success. As previously mentioned, student attitudes may change over time as to
the true impact of the MVP program on their high school elective choices as well as possible
career choices of the future. A follow-up survey with students exiting high school who
participated in the MVP program could provide the insight needed to answer this question
definitively.
Recommendations
Bases on the results of the MVP evaluation, the evaluation team has several
recommendations for the administrators who commissioned the evaluation.

Comparing 7th grade Georgia Milestone test data from 2015-2016, every subject area
had an increase in scores and 3 out of 4 subject areas had score increases when comparing 6th
grade data in 2015 to 7th grade data in 2016. Since the findings of this evaluation cannot be
proven that the MVP program caused the increase in student performance, the evaluation team
recommends additional research be conducted by examining test data from other middle
schools with similar demographics, but without the MVP program.
Analysis of the data collected from the teacher survey determined many teachers do not
view the program as being influential to the students or as having a positive impact on the
academic performance of the students. It is recommended that administration find positive
ways to motivate the MVP teachers since they are the face of the program with whom students
interact. This could be accomplished by allowing teachers to present approved pathways they
are interested in as well as making sure the pathways correspond with the subject area of each
teacher.
Student survey data shows that students view themselves as engaged in their courses
and see a connection between the program skills and their academic success; however, the
pathway courses do not increase their desire to learn about specific careers or influence their
choice of elective courses in high school at this moment in time. In later years, it is possible that
students may recall MVP experiences and they may contribute to elective and career choices.
It is recommended that a follow-up survey be administered to graduating high school students
who participated in the MVP program to provide more insight.
In addition to the above recommendations, if administrators decide to continue the MVP
program, the evaluation team recommends more data be collected over at least two years. The
sum of the three years worth of data can be analyzed to provide a more stable overview of the
MVP program curriculum effectiveness. Long-term collection of data could alleviate some of the
limitations formed when the study, like this one, is only done once.
References
Boone Jr, H. N., & Boone, D. A. (2012). Analyzing likert data. Journal of extension, 50(2), n2.
Davidson, E.J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation.
Thousand Oaks, CA: Sage.
Fournier, D.M. (1995). Establishing evaluative conclusions: A distinction between general and
working logic. New Directions for Evaluation, 68, 15-32.
Georgia Milestones 2014-2015 Statewide Scores. (n.d.). Retrieved April 28, 2017, from
http://www.gadoe.org/Curriculum-Instruction-and-
Assessment/Assessment/Pages/Georgia-Milestones-Statewide-Scores.aspx
Georgia Milestones 2015-2016 Statewide Scores. (n.d.). Retrieved April 28, 2017, from
http://www.gadoe.org/Curriculum-Instruction-and-
Assessment/Assessment/Pages/Georgia-Milestones-2015-2016-Statewide-Scores.aspx
Hooley, T., Marriott, J., & Sampson, J. P. (2011). Fostering college and career readiness: How
career development activities in schools impact on graduation rates and students life
success. Derby: International Centre for Guidance Studies, University of Derby, 1-35.
Marks, H. M. (2000). Student engagement in instructional activity: Patterns in the elementary,
middle, and high school years. American educational research journal, 37(1), 153-184.
Sullivan, G. M., & Artino Jr, A. R. (2013). Analyzing and interpreting data from Likert-type
scales. Journal of graduate medical education, 5(4), 541-542.

Appendices
Appendix A
Teacher Survey Instrument #1
The Teacher Survey Instrument is comprised of six items that are specifically designed to
measure each teachers perception of the MVP program components. On the basis of
responses to these items, the median score will be computed from all teacher responses for
each question. Scores will be classified as Area of Concern (less than 3.5), Acceptable (3.5 to
4.5) and Exemplary (4.5 to 5.0). The three additional qualitative questions will be used to
determine any themes amongst the opinions of the teachers in order to improve the MVP
program in the future.
1. My students are excited to learn about a new career in my vocational

pathways course.
Strongly Agree Agree Neither Agree nor Disagree Disagree Strongly Disagree
2. My vocational pathways course teaches skills that relate to state academic standards.
3. My students enjoy the culminating activities in my vocational pathways class.
4. My students care about doing well on class assignments in my vocational pathways

course.
5. The vocational pathways curriculum I am responsible for teaching challenges all of my

students.
6. I believe the vocational pathways course I teach has a positive impact on students
academic success.
7. Based on my experiences, the strengths of the MVP program are
8. Based on my experiences, the weaknesses of the MVP program are
9. What changes or improvements would you like to see to the MVP program at Moses
Middle School?
Responses will be scored according to the following five point scale: Strongly Agree=5;
Agree=4; Neither Agree nor Disagree=3; Disagree=2 and Strongly Disagree=1
Each engagement item will be presented in order from high to low using the median score of a
Likert scale.
Appendix B
Student Survey Instrument #1
The Engagement Scale is comprised of six items that are specifically designed to measure each
students level of engagement. Based on the responses to these items, an Engagement Score
will be computed averaging the scores of the six engagement items. Scores will be classified as
Not Engaged (less than 3.5), Engaged (3.5 to 4.5) and Highly Engaged (4.5 to 5.0).
1. I am excited to learn about new careers.
2. My vocational pathways classes teach me new skills.
3. I enjoy the culminating activities in my vocational pathways class.
4. I understand how these skills will be important to me later in life.
5. I am more interested in a specific career because of my vocational pathways classes.
6. I plan to take classes in high-school related to a career pathway I learned about in

middle school.
Responses will be scored according to the following five point scale: Strongly Agree=5;
Agree=4; Neither Agree nor Disagree=3; Disagree=2 and Strongly Disagree=1
Each engagement item will be presented in order from high to low using the median score of a
Likert scale.
Appendix C
Evaluation Contract
EVALUATION CONTRACT
This is an agreement between JB Campbell, Dane Forlenza, Tammy Revelle, and Megan
Standley (hereinafter referred to as the Evaluator) and Moses Middle School/Paulding County
School District (hereinafter referred to as the Evaluation Client).
GENERAL INFORMATION
Title of Project: Moses Vocational Pathways Program Effectiveness Evaluation
Scope of Work: The Evaluator will collect data (or use previously collected data) and
analyze this data to answer three evaluation questions regarding implementation or the
effectiveness of the Moses Vocational Pathways program as it relates to the mutually agreed
upon components of the programs operation or intended outcomes in a formative capacity.
WORK STEPS
Work steps include the following: a) develop a program logic model for the evaluation; b)
literature review as it relates to the function of the program and/or importance of the program in
the local setting; c) develop evaluation questions that are mutually agreeable between the
Evaluator and the Evaluation Client that will drive data collection and analysis; d) identify and
document data collection methods; e) define data collection resources; f) define the sample of
participants for which data collection and analysis will be applicable; g) have evaluation plan
peer- and self-reviewed; h) collect or gather relevant data; i) analyze the data using valid and
reliable analysis techniques (e.g., statistics) and tools; j) review initial findings with Evaluation
Client and incorporate input as necessary; k) prepare draft evaluation report and incorporate
feedback from the Evaluation Client as necessary, and l) submit a final evaluation report.
FIELD VISITS
All on-site fieldwork, data collection, interviews, and/or observations shall be coordinated with
the Evaluation Client. Off-site fieldwork is not expected, but if required will be coordinated with
the Evaluation Client.
PROJECT MANAGEMENT
Background: The purpose of this evaluation is to formatively investigate the effectiveness of the
Moses Vocational Pathways program. Evaluation findings will be used to improve the program.
Performance Period: Evaluation planning will take place during January-April 2017, and data
collection and analysis will occur between May 2017 and June 2017.
Type of Contract: Time and materials. Any costs of the evaluation will be covered by the
Evaluator.
CONTRACT AWARD MEETING
The Evaluator shall not begin work on the evaluation until the Evaluator and the Evaluation
Client have met and approved the evaluation plan that is outlined within this Evaluation
Contract/Statement of Work.
GENERAL REQUIREMENTS
1. If desired, the Evaluation Client may designate another organizational representative to

serve as the Evaluation Client to act on his or her behalf. A signature is needed by both the
Evaluation Client (who represents organizational/building level authority) and the designated
Evaluation Client.
2. The Evaluation Client will approve and provide access to Georgia Milestones test scores
of participating students, previous years of Moses Middle Schools Georgia Milestones results.
3. The Evaluation Client will authorize surveys, interviews, or observations for use with
participating teachers and students in the MVP program.
4. Requests for additional data or research beyond the items listed above will require written
approval by the Evaluation Client and will be attached to the Evaluation Contract as an
Addendum.
5. All written deliverables shall be phrased in acceptable terminology of the field; words
shall be defined in layperson language. If necessary, statistical and other technical terms shall be
defined in a glossary of terms and referenced for validity and usefulness.
6. Electronic copies of the draft deliverables will be submitted to the Evaluation Client via
email for review and feedback. When the Evaluator and Evaluation Client meet to review a
deliverable, a hard copy will also be provided by the Evaluator. If there is not a response from
the Evaluation Client within three business days from the data the item was delivered, it shall be
deemed approved. The Evaluator will have three business days to deliver the final deliverables
from the data of receipt of the Evaluation Clients comments. All deliverables shall be delivered
in software used by the Evaluation Client.
7. The Evaluation Report will be written to adhere to the APA Style (6th Edition).
8. A formal presentation of the evaluation findings will be conducted with the Evaluation
Client.
SPECIFIC MANDATORY TASKS AND ASSOCIATED DELIVERABLES
Description of Tasks and Associated Deliverables:
The Evaluator will provide the specific deliverables described below.
Deliverable 1: An Evaluation Plan which describes a background of the Moses Vocational

Pathways program, a program logic model, the agreed upon evaluation questions, a description
of the evaluator, a detailed description of the intended evaluation activities including the sample,
data collection methods, data analyses, relevant instruments including program documentation,
contract, and formative/summative metaevaluation forms.
Task 1: The Evaluator will develop the program logic model to be used to guide the program
evaluation. The program logic model will be shared with the Evaluation Client for feedback.
Task 2: The Evaluator will use the program logic model to identify evaluation questions. The
questions will be mutually agreed upon by the Evaluation and Evaluation Client.
Task 3: A literature review will be conducted to defend the purpose of the program, its function,
and the need for evaluation work, useful instruments, as well as to identify relevant standards
for drawing an evaluative conclusion.
Task 4: The Evaluator will develop or find instruments for collecting data, or identify relevant
existing datasets.
Task 5: The Evaluator will write a data collection and analysis plan.
Task 6: The Evaluator will have a draft of the evaluation plan formatively metaevaluated by
peers in the MEDT 8480 course.
Task 7: The evaluator will conduct a summative metaevaluation of the evaluation plan.
Deliverable 2: A data collection and analysis report to inform the Evaluation Client on the
Evaluators ability to successfully collect data and complete the relevant data analyses.
Task 1: The Evaluator will collect new or gather existing data.

Task 2: The Evaluator will use the methodology described in the evaluation plan to analyze the
data.
Task 3: The Evaluator will share the results of the data analysis with the Evaluation Client for
input, and update the Evaluation Client on emerging analyses that may be relevant or if certain
planned analyses are not feasible.
Deliverable 3: A draft evaluation report which contains all relevant components of the
evaluation report will be provided to the Evaluation Client for input and reaction.
Task 1: The Evaluator will write an evaluation report starting with the text of the evaluation plan.
First the evaluator will update this text to the past tense to reflect that the data collection and
analyses have been conducted. The specific methodologies, if different than what was reported
in the plan should be updated as well to reflect the realities of the evaluation effort. The
evaluation report will include a write up of the evaluation findings in response to the evaluation
questions, and include a description of any limitations associated with the effort, and
recommendations for improving the program as well as next steps for further evaluation work.
Deliverable 4: A final evaluation report in APA format.
Task 1: The Evaluator should decide which feedback from the Evaluation Client to include. In
some cases, the Evaluator will choose not to incorporate specific feedback, and will instead
justify the decision not to change the findings.
Task 2: A formal presentation with the Evaluation Client and any parties designated by the
Evaluation Client will be conducted to discuss findings and important recommendations.
SCHEDULE FOR DELIVERABLES
Deliverable Due Date
Evaluation Plan, which includes: April 16th, 2017

Background Information on Program to be Evaluated
Program Logic Model
Evaluation Questions (previously agreed upon)
Sampling Plan (if necessary)
Data Collection and Analysis Plan
Instruments
Evaluation Contract
External Metaevaluation (peer review) forms
Internal Metaevaluation (self-reflection) form
Data Collection and Analysis Report May 2017-June 2017
Draft Report for Evaluation Client to Review July 15th, 2017
Evaluation Report July 31st, 2017
Presentation to Evaluation Client and Organization August 7th, 2017
The Evaluator shall provide all deliverables to the Evaluation Client as agreed upon in the
schedule established at the initial meeting, and outlined in the table above. Unless otherwise
specified, the number of draft copies and the number of final copies shall be the same. If for
any reason any deliverable cannot be delivered within the scheduled time frame, or the contents
of the deliverable changes, the Evaluator is required to explain why in writing to the Evaluation
Client, including a firm commitment of when the work shall be completed. This notice to the
Evaluation Client shall cite the reasons for the delay and the impact on the overall project. The
Evaluation Client shall then review the facts and issue a response in accordance within three
business days.
CHANGES TO STATEMENT OF WORK

Any changes to this statement of work shall be agreed upon and approved by both the
Evaluator and the Evaluation Client. A copy of each change will be documented and kept in a
project folder along with all other products of the program evaluation project. Any costs
associated with the changes will be at the Evaluators expense. Any unexpected travel must be
pre-approved by the Evaluation Client.
SCHOOL RESPONSIBILITIES
The school shall provide access to technical and procedural information regarding the Moses
Vocational Pathways program. The schools shall provide a copy of a confidentiality statement
(if required) upon request by the Evaluator. If required, the Evaluation Client agrees to work
with the Evaluator to adhere to any district-level data access or data use agreements.
CONTRACTOR EXPERIENCE REQUIREMENTS
The Evaluator will perform this evaluation as an authentic learning experience to fulfill
requirements in the Ed.S. program at the University of West Georgia, College of Education,
Department of Media and Instructional Technology. The professor for this course is Dr. Carl
Westine (Email: cwestine@westga.edu, Office Telephone: 678-839-6095).
CONFIDENTIALITY AND NONDISCLOSURE
This evaluation effort is considered to be an internal evaluation for the purposes of program
improvement, the results of which will not be released or disseminated beyond Moses Middle
School. Only staff members as approved by the Evaluation Client will be able to view the
findings of the program evaluation. All data will be kept confidential, and no student names or
identifying characteristics/information will be used in the evaluation report.
EVALUATION PLAN AND CONTRACT APPROVAL
The evaluation plan including the evaluation contract was reviewed and accepted by:
Evaluation Client: Evaluation Client Representative (if different):
Scott Viness
Print Name Print Name
Moses Middle School

Name of Organization Name of Organization
Sviness@paulding.k12.ga.us
Email Address Email Address
<NOT NEEDED> <NOT NEEDED>

Signature Date Signature Date
Evaluator:
J.B. Campbell, Dane Forlenza, Tammy Revelle,

Megan Standley
Print Name
4/16/17
Signature Date
Appendix D
Peer Metaevaluation
Qualitative Metaevaluation Form Using The Program Evaluation Standards, 3rd Edition
MEDT 8480
Evaluator: Group # 11
J.B. Campbell, D.F. Forlenza,
T.A. Revelle, M.O. Standley
Metaevaluator: Group #9
L. Bryant, C. Walker, 4/4/17
M. Dulaney, A. Norred
NAME DATE
Overall Comments:
Highly Relevant Standards:
Utility - U1, U2, U3, U4, U5, U6
Feasibility - F1
Propriety - P1, P4, P5, P6
Accuracy - A1, A2, A3, A6
Evaluation Accountability -
In addressing the utility standards, the evaluation team has ensured transparency to the stakeholders
and addressed their needs. The evaluation plan has identified goals and other aspects of the MVP by
creating a program logic model, a program description, and needs assessments for the stakeholders.
In addressing the feasibility standards, the team has ensured the evaluation procedures were practical
and the disruption was at a minimum while needed information was obtained.
In addressing the propriety standards, the evaluation team considered necessary professional standards
and ethics related to privacy concerns and safeguarding of personal information. The evaluation team
has also included stakeholders, including students themselves, in the data collection process.
In addressing the accuracy standards, this group has taken measures to ensure that data from the
evaluation is accurate and reliable. An example can be seen in the use of a teacher survey, the purpose
of which is largely to correlate and substantiate findings from the student surveys. The group has also
ensured that evaluation results are measured in a way that is consistent with the stated goals of the
program.
In addressing the evaluation accountability standards, the team has encouraged the group to use
adequate documentation of the evaluation focused on improvement and accountability throughout the
evaluation process.
Additional Feedback:
Minor grammatical errors.
STANDARDS STANDARD STATEMENTS RELEVANCE TO QUALITATIVE FEEDBACK/COMMENTS

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
UTILITY
U1 Evaluator Evaluations should be conducted by This standard is partially met. The evaluation team
Credibility qualified people who establish and has done a great job describing the credibility of
maintain credibility in the evaluation X most team members, including their credentials,
context. such as academic degrees, teaching experience,
knowledge, and leadership capabilities. However,
make sure all members of the team are included in
the description to ensure the trust of the
stakeholders of all evaluation team members.
There is one team member who is missing from the
description.
U2 Attention to Evaluations should devote attention to This standard is met. The evaluation team describes
Stakeholders the full range of individuals and groups in detail the individuals who are invested in the
invested in the program and affected Moses Vocational Pathways Program. Those
by its evaluation. X stakeholders affected by its evaluation are
administrators, educators who teach a vocational
pathway class, and the current 8th grade students
who are upcoming high school freshmen. The
stakeholders involved were identified through an
analysis of individuals impacted by the student
achievement scores in Milestones state testing.
U3 Negotiated Evaluation purposes should be This standard is met. The evaluation purposes have
Purposes identified and continually negotiated clearly been identified by needs assessments
based on the needs of stakeholders. (teacher career responses, qualitative research of
X high school dropouts, Milestones testing), a
program description (increase of student
graduation rates when students have clear and
defined career goals), a program logic model, and
evaluability assessments (teacher and student
surveys, Milestones test data). The program has
already experienced a change in which specific
needs were addressed by the school administrative
staff during post-planning in the implementation

year. The changes made were significant to the
addition of pathways to the program and the
elimination of ability leveled classes.
U4 Explicit Values Evaluations should clarify and specify This standard is met. The values of the program are
the individual and cultural values strongly held by all the stakeholders involved,
underpinning purposes, processes, and including administrators, educators, and students.
judgments. X The program creator, Ms. Ponder, also has invested
contributions from the community for the success
of the program. Engaging the stakeholders
throughout the process is evident. The data
collection and analysis procedures will be used to
interpret the findings of the surveys (quantitative
and qualitative) along with the Milestones test data
to place judgement upon the program.
U5 Relevant Evaluation information should serve This standard is met. The identified stakeholders
Information the identified and emergent needs of contribute information that is equally useful
stakeholders. through surveys, which includes open-ended
X questions to address emergent needs. The surveys
are conducted in an anonymous online
environment to ensure stakeholders of their privacy
in forthcoming information. The evaluation team
recognizes that information from different sources
can provide a trusting environment for the
stakeholders.
U6 Meaningful Evaluations should construct activities, This standard is met. The evaluation team has
Processes and descriptions, and judgments in ways implemented meaningful processes and products
Products that encourage participants to to encourage the participation of all stakeholders.
rediscover, reinterpret, or revise their X With the creation of student interest surveys and
understandings and behaviors. the teacher surveys, the evaluation team can
gather information about how these two groups of
stakeholders feel and think about the program.
Also, by revisiting and revising the initial design
with regards to the addition of pathways and the
ability leveled classes, this change engaged
stakeholders by building trust and enhancement of
different views.
U7 Timely and Evaluations should attend to the This standard is partially met. A thorough
Appropriate continuing information needs of their description is given regarding the data instruments
Communicating multiple audiences. and the analysis of the data collected, however, will
and Reporting X the data be reported to stakeholders in a timely
manner and be distributed appropriately? The
Milestones test is conducted at the end of the
school year, and data is soon received afterward,
but when will the surveys take place and how soon
can this information be distributed to the
stakeholders? You might want to be more explicit in
the timing of the survey implementation and
reporting of data.
U8 Concern for Evaluations should promote This standard is partially met. An informal
Consequences responsible and adaptive use while assessment (surveys) and a formal assessment
and Influence guarding against unintended negative X (Milestones) are implemented within this
consequences and misuse. evaluation from the team, however, be more
specific as to when the informal assessments are
implemented. Also, identify both formal and
informal means of communication, such as emails,
written mini-reports, or PLCs, between

stakeholders to create connectivity and
collaboration, such as between the administrators
and teachers involved in the program goals.

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
FEASIBILITY
F1 Project Evaluations should use effective X This standard is met. The evaluation consistently
Management project management strategies. relate to all stakeholders in a professional manner.
The evaluation plan honor participants privacy
rights. Also the plan alert to and address
participants; concerns about the evaluation.
F2 Practical Evaluation procedures should be X This standard is partially met. The evaluation plan
Procedures practical and responsive to the way the tailor methods and instruments to information
program operates. requirements. Also mention appoint competent
staff and train staff. The plan engage locals to help
conduct the evaluation. However be more specific
with the procedures that the staff are qualified to
carry out.
F3 Contextual Evaluations should recognize, monitor, X This standard is partially met. The evaluation plan
Viability and balance the cultural and political anticipate different positions of different interest
interests and needs of individuals and groups. Also involve stakeholders throughout the
groups. evaluations. However, be more specific with report
divergent views.
F4 Resource Use Evaluations should use resources This standard partially is met. The evaluation plan is
effectively and efficiently. efficient, inform decisions and foster program
improvement. However be more specific with
x generating new insight and and minimize time
demands on program personnel.

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
PROPRIETY
P1 Responsive and Evaluations should be responsive to The standard is met. The evaluation plan includes
Inclusive stakeholders and their communities. all administrators, teachers, and students who are a
Orientation X part of the program. The evaluation allows for
stakeholder perception and engagement of
students involved.
P2 Formal Evaluation agreements should be The standard is partially met. The needs of student
Agreements negotiated to make obligations explicit engagement is addressed in the evaluation.
and take into account the needs, X However, no differentiation of cultures is not
expectations, and cultural contexts of mentioned in the evaluation plan.
clients and other stakeholders.
P3 Human Rights Evaluations should be designed and The standard is partially met. The identities of
and Respect conducted to protect human and legal individuals being surveyed during the evaluation
rights and maintain the dignity of X are protected. The surveys are anonymous, yet it
participants and other stakeholders. does not indicate how and where the individuals
being surveyed would be selected.
P4 Clarity and Evaluations should be understandable The standard is met. The evaluation and surveys
Fairness and fair in addressing stakeholder X are presented in an clear and understandable
needs and purposes. fashion for teachers and students being evaluated.
The evaluations will provide information for the
stakeholders that will allow them to make changes,
if needed, to the MVP program.
P5 Transparency Evaluations should provide complete The standard is met. The evaluators have provided
and Disclosure descriptions of findings, limitations, X clear indication of how the scores would be used
and conclusions to all stakeholders, for the stakeholders. The administration is seeking
unless doing so would violate legal and scores from student surveys with a mean score of
propriety obligations. 3.5, scores of less indicate that the program is not
impacting the students.
P6 Conflicts of Evaluations should openly and honestly The standard is met. Both questionnaires provide
Interest identify and address real or perceived X questions that address the interest of the program
conflicts of interest that may and whether the program has an impact of the
compromise the evaluation. student interest in careers for future education in
high school.
P7 Fiscal Evaluations should account for all The standard is partially met. The evaluation does
Responsibility expended resources and comply with X mention activities in the program, it does not
sound fiscal procedures and processes. individualize the activities in the program - which
activities had a positive or negative outcome.

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
ACCURACY
A1 Justified Evaluation conclusions and decisions X This standard is met. The evaluation will look at
Conclusions should be explicitly justified in the teacher and student perceptions about the MVP
and Decisions cultures and contexts where they have program as well as examine standardized test
consequences. scores. The context of the evaluation is consistent
with the conclusions that will be drawn.
A2 Valid Evaluation information should serve This standard is met. Evaluation data will examine
Information the intended purposes and support x student achievement, the primary goal of the MVP
valid interpretations. program. Evaluation will also examine student and
teacher perceptions about the MVP program, which
forms much of the basis for why program leaders
believe their program can be successful.
A3 Reliable Evaluation procedures should yield X This standard is met. Evaluation data will use
Information sufficiently dependable and consistent standardized test scores, consistent with the goals
information for the intended uses. of the program. Teacher and student perceptions
will be measured via survey.
A4 Explicit Evaluations should document X This standard is met. In evaluating the MVP
Program and programs and their contexts with program, there is not a great deal of data to
Context appropriate detail and scope for the examine. The evaluation will examine improvement
Descriptions evaluation purposes. in student achievement through one measure:
Standardized testing. As this is the intended goal of
the program, little other detail is required.
A5 Information Evaluations should employ systematic X This standard is partially met. Data will be collected
Management information collection, review, from standardized tests and from student and
verification, and storage methods. teacher surveys. The plan does not detail
specifically how data from the surveys will be
reviewed, verified, or stored, other than stating
how the surveys will be scored.
A6 Sound Designs Evaluations should employ technically X This standard is met. The likert scale employed by
and Analyses adequate designs and analyses that the student and teacher surveys are appropriate for
are appropriate for the evaluation the evaluation purpose in that they measure
purposes. student perceptions about the MVP program and
seek to examine correlation with the teacher
survey. Standardized test scores will need only to
be compared with previous years scores to
determine if the programs goal of 20%
improvement is being reached.
A7 Explicit Evaluation reasoning leading from X This standard is met. The evaluation seeks to verify
Evaluation information and analyses to findings, conclusions that might be drawn from the student
Reasoning interpretations, conclusions and surveys by correlating them with teacher surveys.
judgments should be clearly and
completely documented.
A8 Communication Evaluation communications should This standard is partially met. The evaluation plan
and Reporting have adequate scope and guard X does not detail how the evaluation results will be
against misconceptions, biases, communicated to the program directors. However,
distortions, and errors. by including both teacher and student surveys as a
tools to measure student perceptions and
engagement in the MVP program, the plan seeks to
guard against distorted data by seeking to correlate
survey information between the two groups.

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
EVALUATION ACCOUNTABILITY
E1 Evaluation Evaluations should fully document X The standard is partially met. The evaluation collect
Documentation their negotiated purposes and descriptions of implemented evaluation and collect
implemented designs, procedures, descriptions of the evaluation procedures However
data, and outcomes. unclear on analyze the data and record outcomes.
E2 Internal Evaluators should use these and other X This standard is fairly met. The evaluation is not
Metaevaluation applicable standards to examine the clear on assigning someone responsible for
accountability of the evaluation documenting and assessing the evaluation process
design, procedures employed, and products. Also designate or define the
information collected, and outcomes. standards to be used in judging the evaluation.
E3 External Program evaluation sponsors, clients, X This standard is fairly met. The evaluation does not
Metaevaluation evaluators, and other stakeholders clearly mention budget appropriately and
should encourage the conduct of sufficiently for conducting the meta evaluation.
external metaevaluations using these
and other applicable standards.
Appendix E
Self Evaluation
Summative Metaevaluation Form Using The Program Evaluation Standards, 3rd Edition
MEDT 8480
Metaevaluator: Group # 11
J.B. Campbell, D.F. Forlenza, 4/30/17
T.A. Revelle, M.O. Standley
NAME DATE
Overall Comments:
Highly Relevant Standards:
Utility - U1, U2, U3, U4, U5, U6
Feasibility - F1, F3
Propriety - P1, P4, P5
Accuracy - A1, A2, A5, A6, A8
Evaluation Accountability E1
In addressing the utility standards, we ensured that we met all of the highly relevant standards by
ensuring transparency to the stakeholders of the MVP program. We have presented ourselves as highly
qualified external evaluators who will use relevant information and meaningful processes to report
information in a timely manner to the stakeholders of the program.
In addressing the feasibility standards, we ensured that we met all of the highly relevant standards of
the program in using professional practices that are sensitive to the political and social make up of
stakeholders and participants.
In addressing the propriety standards, we met all of the highly relevant standards necessary for
professional standards and ethics related to privacy concerns and safeguarding of personal information.
We included stakeholders in the form of both students and teachers in the collection of data to ensure
responsiveness to the stakeholders and community
In addressing the accuracy standards, we met all of the highly relevant standards with the exception of
A8, which was partially met. The quantitative data that was collected through standardized test scores
and survey results, is to the best of our knowledge, reported correctly, fairly, and accurately to evaluate
the effectiveness of MVP on standardized test scores and student and teacher engagement. Our
qualitative data is reported fairly, and the conclusions generated maintain consistency with the results
of the data.
In addressing the evaluation accountability standards, we met all of the highly relevant standards in
order to use adequate documentation focused on improvement and accountability throughout the
evaluation process.
Additional Feedback:
For our first effort at a program evaluation, we are proud of what we have accomplished and feel that
we have gained valuable experience towards future evaluations of programs in our schools.

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
UTILITY
U1 Evaluator Evaluations should be conducted by This standard is met, as we have provided a

Credibility qualified people who establish and comprehensive overview of our credibility as
maintain credibility in the evaluation evaluators through listing all of our academic
context. degrees, teaching experience, knowledge and
X leadership capabilities.
U2 Attention to Evaluations should devote attention to This standard is met as we described all individuals
Stakeholders the full range of individuals and groups invested in the Moses Vocational Pathway Program.
invested in the program and affected We explained that the stakeholders affected by the
by its evaluation. evaluation are administrators, educators who teach
a vocational pathway class, and the current 8th
grade students. The stakeholders involved were
X identified through an analysis of individuals
impacted by the student achievement scores on
Georgia Milestones state testing.
U3 Negotiated Evaluation purposes should be This standard is met. We have clearly identified the
Purposes identified and continually negotiated evaluation purposes in the categories of needs
based on the needs of stakeholders. assessments, description of the program, a program
logic model, and the evaluability assessments. The
program has already experienced a change in which
specific needs were addressed by the school
X administrative staff during post-planning in the
implementation year. The changes made created
the addition of pathways to the program and the
elimination of ability leveled classes.
U4 Explicit Values Evaluations should clarify and specify This standard is met. All of the stakeholders,
the individual and cultural values including administrators, educators, students, and
underpinning purposes, processes, and the program creator, Ms. Ponder, have invested
judgments. contributions from the community into the eventual
success of the program. We have made it apparent
that we will engage stakeholders through the
X process, and shown how the findings of quantitative
and qualitative data throughout the evaluation
process will be used to judge the effectiveness of
the program.
U5 Relevant Evaluation information should serve This standard is met. Information from stakeholders
Information the identified and emergent needs of is contributed through surveys with open-ended
stakeholders. questions to address emergent needs. In addition,
the surveys are conducted anonymously to ensure
the privacy of stakeholders. We also ensured that
we collected both relevant qualitative and
X quantitative data to ensure a comprehensive
evaluation was being conducted.
U6 Meaningful Evaluations should construct activities, This standard is met. We have implemented
Processes and descriptions, and judgments in ways meaningful processes and products to encourage
Products that encourage participants to participation of all stakeholders. This has been done
rediscover, reinterpret, or revise their through the inclusion of a plethora of surveys such
understandings and behaviors. as student interest and teacher, through which we
can begin to understanding how these two vital
X groups of stakeholders feel about the program.
U7 Timely and Evaluations should attend to the This standard is met. We provided a detailed outline
Appropriate continuing information needs of their for the data instruments we would be using to
Communicating multiple audiences. collect information, and reported the information
and Reporting accurately and in a timely manner to the
stakeholders. We were able to compare Milestones
results from the 2015 and 2016 school years to
X show an increase in achievement after MVP
implementation. In addition, we were able to report
timely on student and teacher engagement with the
program.
U8 Concern for Evaluations should promote This standard is partially met as the Milestones data
Consequences responsible and adaptive use while instrument yielded a positive impact on Moses
and Influence guarding against unintended negative Middle School and validated the programs
consequences and misuse. intended purpose for implementation; however, the
X results of both the teacher and student instruments
perhaps acted as bastions for discontent with MVP,
and only succeeded in riling up certain stakeholders
to not want to continue with the use of the
program.

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
FEASIBILITY
F1 Project Evaluations should use effective X This standard is met as we consistently adhere to
Management project management strategies. professional practices by both protecting the
participants privacy rights and addressing
participants and stakeholders concerns with the
evaluation.
F2 Practical Evaluation procedures should be X This standard is partially met. Our evaluation
Procedures practical and responsive to the way the procedures are practical but as outsiders to the
program operates. community our program logic model and certain
aspects of student interest surveys offer some
generalizations that could be used to apply to a
variety of different evaluations and not just MVP.
F3 Contextual Evaluations should recognize, monitor, X This standard is met. We balanced the cultural and
Viability and balance the cultural and political political interests of all stakeholders in the
interests and needs of individuals and evaluation process. In reporting results, we were
groups. very transparent with reporting all viewpoints and
attitudes of the program, even ones that depicted
the program in a negative light.
F4 Resource Use Evaluations should use resources This standard is met. The evaluation plan is
effectively and efficiently. efficient, informs decisions and foster program
improvement. In addition, we were to generate
specific conclusions that either validated the
x program, or provided suggestions for improvement.

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
PROPRIETY
P1 Responsive and Evaluations should be responsive to The standard is met. We included all
Inclusive stakeholders and their communities. administrators, teachers, and students who are a
Orientation part of the program. We also allowed for
X stakeholder perception and engagement of
students involved.
P2 Formal Evaluation agreements should be The standard is met. Our Evaluation Contract
Agreements negotiated to make obligations explicit includes detailed descriptions for all of the
and take into account the needs, requirements, logistics, and expectations of clients
expectations, and cultural contexts of and all stakeholders.
clients and other stakeholders. X
P3 Human Rights Evaluations should be designed and The standard is partially met. The identities of
and Respect conducted to protect human and legal individuals being surveyed during the evaluation are
rights and maintain the dignity of protected. However, due to the nature of the
participants and other stakeholders.
X program only being at one school and not across an
entire district or state, complete anonymity cannot
be guaranteed for participants of surveys.
P4 Clarity and Evaluations should be understandable The standard is met. The evaluation and surveys
Fairness and fair in addressing stakeholder are presented in a clear and understandable fashion
needs and purposes. X for teachers and students being evaluated. The
evaluations will provide information for the
stakeholders that will allow them to make changes,
if needed, to the MVP program.
P5 Transparency Evaluations should provide complete The standard is met. We provided clear indication
and Disclosure descriptions of findings, limitations, of how the scores would be used for the
and conclusions to all stakeholders, X stakeholders. The administration is seeking scores
unless doing so would violate legal and from student surveys with a mean score of 3.5,
propriety obligations. scores of less indicate that the program is not
impacting the students.
P6 Conflicts of Evaluations should openly and X The standard is partially met. We openly addressed
Interest honestly identify and address real or in the evaluation purpose section that an impartial
perceived conflicts of interest that may evaluation of the MVP program will take place;
compromise the evaluation. however, we never alluded to the fact that one of
the members of our evaluation team is a teacher at
Moses Middle School thus there being a certain
concern of conflict of interest both as a stakeholder
and external evaluator.
P7 Fiscal Evaluations should account for all X The standard is met. We acknowledge that while
Responsibility expended resources and comply with this standard is met it really isnt applicable to our
sound fiscal procedures and processes. evaluation as there was no need for fiscal
responsibility due to the fact that money wasnt
involved in the evaluation process.

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
ACCURACY
A1 Justified Evaluation conclusions and decisions X This standard is met. We analyzed teacher and
Conclusions should be explicitly justified in the student perceptions about the MVP program
and Decisions cultures and contexts where they have through qualitative survey results as well as
consequences. examined standardized test scores of the Georgia
Milestones Assessment over a two year time period.
The context of the evaluation is consistent with the
conclusions that will be drawn.
A2 Valid Evaluation information should serve This standard is met. Evaluation data will examine
Information the intended purposes and support
x student achievement, the primary goal of the MVP
valid interpretations. program. Evaluation will also examine student and
teacher perceptions about the MVP program, which
forms much of the basis for why program leaders
believe their program can be successful.
A3 Reliable Evaluation procedures should yield X This standard is partially met. Teacher and student
Information sufficiently dependable and consistent perceptions accurately measured the success of the
information for the intended uses. program; however, a two-year span of standardized
test results from Ga Milestones doesnt necessarily
directly correlate increased test scores to the MVP
program. A multi-year study would be needed to
provide more validity to MVPs impact on test
scores.
A4 Explicit Evaluations should document X This standard is met. In evaluating the MVP
Program and programs and their contexts with program, there is not a great deal of data to
Context appropriate detail and scope for the examine. The evaluation will examine improvement
Descriptions evaluation purposes. in student achievement through one measure:
Standardized testing. However, due to the fact that
Milestones testing is relatively new within the state
of Georgia, there isnt a long history of test scores
to look at in order to determine factors attributed
to growth and regression.
A5 Information Evaluations should employ systematic X This standard is met. All data both quantitative and
Management information collection, review, qualitative is meticulously explained, categorized,
verification, and storage methods. and clearly presented within the evaluation report
through charts, tables, and text.
A6 Sound Designs Evaluations should employ technically X This standard is met. The Likert scale employed by
and Analyses adequate designs and analyses that the student and teacher surveys are appropriate for
are appropriate for the evaluation the evaluation purpose in that they measure
purposes. student perceptions about the MVP program and
seek to examine correlation with the teacher
survey. Standardized test scores are broken down
by grade level to show whether the 20% increase in
testing achievement occurred.
A7 Explicit Evaluation reasoning leading from X This standard is met. The evaluation seeks to verify
Evaluation information and analyses to findings, conclusions that might be drawn from the student
Reasoning interpretations, conclusions and surveys by correlating them with teacher surveys.
judgments should be clearly and
completely documented.
A8 Communication Evaluation communications should X This standard is partially met. We present the
and Reporting have adequate scope and guard quantitative data in its complete unbiased,
against misconceptions, biases, undistorted form in tables 1-3 in the findings
distortions, and errors. section of the evaluation report. Table 4 provides
positive takeaways and weaknesses as quantitative
data which is a slight distortion as these are kind of
the opinions, and should not be considered data.

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
EVALUATION ACCOUNTABILITY
E1 Evaluation Evaluations should fully document X The standard is met. The evaluation fully documents
Documentation their negotiated purposes and all purposes, designs, procedures, data, and
implemented designs, procedures, outcomes for the evaluation process.
data, and outcomes.
E2 Internal Evaluators should use these and X This standard is met. An internal metaevaluation
Metaevaluation other applicable standards to examined all aspects of the evaluation plan to
examine the accountability of the ensure its compliance with design, procedures,
evaluation design, procedures collection of information, and outcomes.
employed, information collected, and
outcomes.
E3 External Program evaluation sponsors, clients, X This standard is met. We had other evaluators
Metaevaluation evaluators, and other stakeholders provide a detailed formative metaevaluation to
should encourage the conduct of ensure compliance with the 30 PES standards.
external metaevaluations using these
and other applicable standards.

Group 11 Eval Report 04-30-17

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Group 11 Eval Report 04-30-17

Uploaded by

Copyright:

Available Formats

Running Head: EVALUATION REPORT: MOSES VOCATIONAL PATHWAYS PROGRAM 1

Moses Vocational Pathways Program

J. B. Campbell, Dane F. Forlenza, Tammy A. Revelle, Megan O. Standley

University of West Georgia

Background of the Program and Evaluation

Moses Vocational Pathways program began shortly after in September of 2015.

about the careers they are in.

Science, and Social Studies.

In order to develop self-directed learners, teachers will engage students in hands-on,

Technology and Communication, Information Technology, Hospitality and Tourism, Agriculture,

different one after nine weeks.

Program Logic Model - Moses Vocational Program

effectiveness has yet to be proven. A formative evaluation is needed in order to determine

general logic of evaluation:

student mastery of academic state standards?

2. What level of student engagement do teachers observe during the activities of

vocational classes within the MVP program?

concerning future career pathways?

academic spectrum, it will help justify and validate the program.

thus perform better on a standardized test.

considering they voluntarily do outside of the classroom.

career pathways as they prepare for high school and beyond.

Revelle, Dane Forlenza and Megan Standley.

in mathematics as well as receiving College Board AP certification to teach Calculus AB and

Master of Science in Post-Secondary Education from Troy University in 2010. Currently, he is

Data and Instrumentation

examined using a mixed-methods survey - Teacher Survey Instrument #1 (Appendix A).

effectiveness of the program to aid in later improvement processes.

effects the program has on future career choices.

academic performance for this level student.

meets the required standard for successful implementation.

This evaluation used a mixed-methods approach which we define as the simultaneous

generated ideas on ways to improve the program.

Standards for Evaluation Questions

changes should consistent themes emerge from their feedback.

effect on the students thinking concerning future career opportunities.

According to Marks (2000), an increase in student engagement can be achieved through

skills and education needed for specific career choices.

concerning the completion of planned evaluation instruments and participation of the

the Paulding County School District.

previous class groups.

exceptions for Level 3 in Science and Level 4 in English Language Arts.

length of time involved in the Moses Vocational Pathways program.

questions concerning their perceptions of student engagement in the Moses Vocational

well as dissolving the program altogether) was requested.

Table 4: Quantitative Data From Teacher Surveys: questions #7-9

Positive Takeaways Weaknesses Possible Changes

Students make a Students do not see Students need to be

conducted across multiple years to examine consistencies.

To answer the evaluation question, What level of student engagement do teachers

program and less to do with student engagement.

school and beyond.

other middle schools without the MVP program.

recommendations for the administrators who commissioned the evaluation.

schools with similar demographics, but without the MVP program.

academic performance of the students. It is recommended that administration find positive

It is recommended that a follow-up survey be administered to graduating high school students

who participated in the MVP program to provide more insight.

In addition to the above recommendations, if administrators decide to continue the MVP

Thousand Oaks, CA: Sage.

working logic. New Directions for Evaluation, 68, 15-32.

Marks, H. M. (2000). Student engagement in instructional activity: Patterns in the elementary,

scales. Journal of graduate medical education, 5(4), 541-542.