Running head: SMART GIRLS EVALUATION

The Implementation of Smart Girls in Guilford County Schools

EVALUATION PLAN FOR SMART GIRLS
Gillian Adler, Jennifer Aiken and Laura Bolton

SMART GIRLS EVALUATION

Section 1: Conceptualization of Evaluation

Purpose:
Through conducting a process evaluation, Smart Girls, an adolescent pregnancy prevention
program, will be assessed to determine whether it is being implemented the way it was intended.
Additionally, this evaluation will examine the program activities and how they work to accomplish the
programs short-term outcomes.

Broad evaluation questions:

Is Smart Girls being implemented in the way it is initially intended by the key stakeholders?
How is the program operating to engage and retain middle and high schools in Guilford county?
Is Smart Girls reaching areas that have a higher risk of teenage pregnancies?
Is the program effective in increasing self-esteem and decision-making skills among teenage girls?
Key stakeholders:
The key stakeholders are the Smart Girls program directors and volunteers, parents of girls
participating in Smart Girls, school officials, participants, funders, and Guilford County Health
Department. These groups are considered key stakeholders due to their personal insight, interaction, and
involvement with the program. Their input will be needed throughout the evaluation to ensure
expectations are being met and to enhance program sustainability.

Assumptions:
Abstinence-plus sex education is beneficial for every adolescent girl.
Abstinence-plus sex education helps prevent adolescent pregnancy.
The curriculum is implemented in a sufficient amount of time to influence attitudes, skills and behaviors.
Parents believe that someone in the school system is qualified to discuss sex education with their teen.
Student behavior will not vary significantly during observations by evaluators.

Contextual factors:
Religious beliefs can potentially influence willingness to participate in an abstinence-plus sex education

program.
The demographics of the participants may vary.
The culture of our education system typically prioritizes subjects like math and science over health and

consequently abstinence-plus sex education.

Cultural perceptions within this geographical location, which make it difficult to fully implement

abstinence-plus sex education due to the historical preference for teaching abstinence-only.
Age of the educator volunteer can influence the responsiveness of the participants.

SMART GIRLS EVALUATION

3
Section 2: Program Description

Program Purpose: To provide young girls with the knowledge and skills needed for them to become
empowered young women who are ultimately happy, healthy and successful in their lives (Smart Girls
Adolescent, 2013). The program is designed to provide young girls with the skills and knowledge to
translate into good decision making skills, which work towards preventing adolescent pregnancy.
Program Goals and Objectives:
The program goals and objectives listed below are aimed to provide teenage girls with the skills and
knowledge to prevent pregnancy:
1) To enhance self-esteem among teen girls
2) To sharpen decision making skills

Increase use of contraception for those participants who reported sexual involvement
Improve knowledge of sexuality and consequences of sexual activity
3) To assist girls to set goals for the future

Increase healthier attitudes towards postponing sex and the use of contraception
4) To strengthen family connections

Increase communication between teen girls and their parents about sexuality

Target Audience: Young females ranging from age 11-14 along with their parents/guardians.
Program Location: Smart Girls is implemented in Guilford County.
Results of Previous Evaluation:
The Smart Girls programming has been evaluated twice, once in 2003 and another time in 2011.
In 2003, the programs impact was evaluated at the beginning of the program and at a six-month followup. According to the evaluation, participating girls showed improved knowledge of sexuality and
consequences of sexual activity, healthier attitudes towards postponing sex and use of contraception,
increased use of contraception for those participants who reported sexual involvement, and increased
communication with parents about sexuality (Smart Girls Adolescent, 2013). However, it was not made
clear if this was done by an internal or external evaluator. An evaluation conducted in 2011 showed that
there needed to be booster lessons included as part of the program to help make sure that attitudes and

SMART GIRLS EVALUATION

decision making skills acquired do not weaken. It also showed that the personal/self sexuality
expectations among the participating girls changed significantly over time. It was recommended that
another evaluation be conducted to determine whether Smart Girls reduces actual incidences of teen
pregnancy, and that the program would benefit from additional observations (Graves, Sentner, Workman
& Mackey, 2011). Missing from these evaluations was determining the impact of parental involvement
and also school recruitment and retention, which we will examine in our evaluation.
Logic Model:
Resources

Health Educators
to Teach
Curriculum
Classroom space

Activities

Outputs

Short-term
outcomes

Long-term outcomes

Incentive
Creation and
Distribution

Incentives
distributed each
class

More engagement
and attendance
among teens

Increase in overall
participation rate for
Smart Girls

Deliver
Curriculum

6-8 modules
over a semester
(Sept December)Modules:
Introduction,
Self-esteem,
Decision
Making, Peer
Pressure,
Anatomy/Bodie
s,
Contraception,
STDs, Teen
Dating Abuse,
Goal Making &
Post-test

Increase in selfesteem

Increase in
confidence in their
own sexuality

Office supplies
Access to
curriculum
Incentives
Parental Consent

Increase in decisionmaking skills

Increase in
knowledge about
birth control

Postponing sexual
behaviors
Less teenage girl
pregnancy

Students
receiving
curriculum
Parent
Involvement (i.e.
parental consent)

Parents
involved by
doing
worksheets,
roleplaying

Increase/
enhance
communication
skills between
parents and teens

Healthier
relationships for teens

Recruitment and
Retention

Schools
implementing

Increase in reach to
middle and high

Changes in attitude
toward abstinence-

More parent/
guardian involvement

SMART GIRLS EVALUATION

5
Smart Girls
curriculum

schools

plus sex education in

schools

Maintained presence
in current middle
and high schools
The resources listed in the above logic model are critical elements of the Smart Girls program.
They help to support the process and completion of listed activities and their respective short and longterm outcomes. The creation and distribution of incentives is the first activity listed in the logic model due
to its importance to the overall program. This activity essentially supports Smart Girls effort toward
increasing active participation. The incentives are utilized to attract girls to the program and encourage
regular attendance. During this particular activity, Smart Girls program instructors identify, create or
purchase, and distribute incentives to the participating girls. It is expected that an incentive will lead to
the short-term outcome of an increase in participating girls desire to attend classes and resulting in more
program engagement and attendance among teens. It is anticipated that eventually, there will be an overall
increase in the Smart Girls participation rate.
The next activity is delivering Smart Girls curriculum, which is made up of 6-8 modules, over the
course of the semester. Each module covers pertinent topics and creates opportunities for participating
girls to be active learners and to develop the knowledge and skills that are necessary for them to reach the
Smart Girls program goals. It is anticipated that at the conclusion of curriculum delivery, participating
girls will develop an increase in self-esteem, decision making skills, and knowledge about birth control. It
is expected that when these short-term outcomes are achieved, it will lead to an increase in confidence in
their own sexuality, girls will postpone sexual behaviors, and there will be less teen pregnancies. The title
and description for each module can be found in Appendix A.
The next activity is increasing parent/guardian involvement through the Smart Girls program.
Parental involvement is encouraged through the distribution of take home materials, usually worksheets.
The worksheets provided to parents are parent-teen connection worksheets that encourage parent and
child time. Parents and their children have the opportunity to discuss and write down statements they

SMART GIRLS EVALUATION

believe to be true for education, family, friends, careers, dating/relationships, and the meaning of
sexuality. Smart Girls intends for parent/guardian participation to be ongoing throughout program
implementation. The anticipated short-term outcome of this activity is identified as an increase or
enhancement in communication skills between parents/guardians and the participating girls. It is expected
that if communication skills are enhanced between parents/guardians and girls, then there will be healthier
relationships for teens as they get older and an increase in parent/guardian involvement.
The last activity listed in the logic model is, recruitment and retention. This is an important
activity as it reflects and contributes greatly to the sustainability of Smart Girls. In order for Smart Girls
to be sustainable, there needs to be schools willing to implement the curriculum. The anticipated shortterm outcome is an increase in the reach to middle and high schools. This means more schools are
captured by Smart Girls, specifically schools in those areas that are more rural or do not have many
available resources. The additional short-term goal is a maintained presence in the middle and high
schools that already participate in the Smart Girls program. Smarts Girls aims to continue relationships
with existing participating schools. These outcomes reflect the aim to recruit new schools, as well as
retain existing schools participation. It is anticipated that if these short-term outcomes are achieved, then
there will be a change in attitude toward abstinence-plus sex education in our schools.
Section 3: Evaluation Plan
Specific Evaluation Questions:
1. To what extent is the curriculum captivating, engaging, and current to participants?
2. To what extent is the program being implemented the way it was initially intended to be implemented?
3. To what extent is the program reaching the schools that are at a higher risk versus convenience?
4. How well are the trainers prepared to implement the curriculum?
5. To what extent are the resources (i.e. worksheets, roleplays) enriching the learning experience for
students?
6. To what extent is the input of the school being implemented into the classes?
7. Do the students consistently attend the program?
8. How are the students engaging in the activities?
9. What activities are the participants most engaged with?
10. How did the program influence the participants' self-esteem?
11. To what extent do participants have an increased understanding of what an informed decision looks
like?

SMART GIRLS EVALUATION

12. How confident are the students about birth control options?
13. What changes have parents seen in their child's willingness to talk about personal matters (i.e. their
child's sexuality)?
14. To what extent, are current high and middle schools retaining the programming?
15. To what degree, are more high and middle schools participating?
Experimental Design:
To conduct a process evaluation, we will be using an interrupted time series design with multiple
groups. For Smart Girls, a simple random sample will be used to select the schools. There will be 3
schools will receive and be evaluated on the implementation of the Smart Girls curriculum for the first
three months (i.e. September to December) and 3 additional schools will undergo this process during a
subsequent three months (i.e. January to April) during health classes. This design was chosen to identify
possible threats to validity, such as maturation (i.e. aging).
Variables Measured:
Variables measured will be attitude, critical thinking skills, self esteem, expectations, parent-child
relationships, student participation and school retention rates.
Instrumentation:
Observation
Observations will occur in the classrooms when Smart Girls is being implemented, and each
observation will continue throughout the allotted time period for the class. A graduate assistant will be
present in the class to observe any side conversation, engagement in the material and disengagement
behaviors. Engagement will be how often students respond to questions, ask questions whether
anonymously or in person, stay on topic, etc. Disengagement is not answering questions, asking
inappropriate/random questions during class, sleeping, texting, etc. In-class discussions will be observed
to determine level of engagement in each topic, quality, inclusion of multiple classmates, staying on topic,
and respect of different attitudes. We will use a checklist to document these items and to compare across
classrooms.
Surveys
Surveys will provide feedback from key informants at the schools (i.e. coaches, guidance
counselors, other teachers) on whether they notice a change in attitude or knowledge about life skills.

SMART GIRLS EVALUATION

Second, surveys to parents will inquire about if they see more engagement in talking about personal
matters such as sex and relationships with their child, and parents perceptions of how the program has
affected their child. Last, students will have a pre-post test that measures changes in self-esteem, attitudes,
and knowledge of sexuality. We will use SPSS to find descriptive statistics for the students pre/posttest.
We will do a thematic analysis for the key informant and parent interviews.
Testimonials
Testimonials will track how engaged students are in the activity for each week. The testimonials
will be in the form of question and answer (i.e. what did you think of the lesson this week?). Responses
will be based on how much is written, if every question is answered, if the answers are thoughtful, and the
length of the answers. These testimonials collected each week from the students are about the modules
and classroom learning environment.
Fidelity checklist
While the fidelity checklist is part of the program, we will be determining what items were
checked off the list by the educator and reasoning behind not checking off items.

Data collection plan:

Observations
Graduate assistants will attend and observe the same learning modules at the six different schools.
The graduate assistants will then compare the observation checklist results between participating students
and sites.
Survey
School key informants will be interviewed at their schools by a graduate assistant halfway
through the program and at the end. These interviews will be voice-recorded and transcribed. Parents will
be interviewed over the phone by a graduate assistant at the beginning and end of the program. These
phone interviews will also be recorded. Students will have a test before and at the end of the program
while class is in session. These tests will be collected by the educator and then handed to the graduate
assistants at the end of the program.
Testimonials

SMART GIRLS EVALUATION

The testimonials will be completed during class or taken home to be completed with parents/on
their own. At the beginning of each week, the graduate assistants will collect these testimonials.

Fidelity Checklist
The fidelity checklist will be completed by the educator at the end of each class and collected at
the end of program by the graduate assistant.
Evaluation Crosswalk
See Appendix B.
Section 4: Reporting Plan
Our reporting method is tailored to fit the audience of Smart Girls program directors, parents of
girls participating in Smart Girls, school officials and the funders. To best engage these various groups,
we will take multiple approaches that fit to the audiences, like newsletters to the parents, personal
discussions with top officials from the participating schools, a press release to the community, posting on
Smart Girls website and social media, and a final report with an executive summary to the program
directors and funders. While we will also be creating a final report, we would also like to have a
scheduled interim report to share results and seek reactions from stakeholders. For example, at PTA
meetings, we would like the opportunity to share with parents what we are doing to assess the Smart Girls
programming. Additionally, we plan to have a publication in either discussing our findings.
Section 5: Detailed Budget
See Appendix C.

Section 6: Detailed Timeline

See Appendix D.

SMART GIRLS EVALUATION

10

References
Graves, K.N., Sentner, A., Workman, J., Mackey, W. (2011). Building Positive Life Skills the
Smart Girls Way: Evaluation of a School-Based Sexual Responsibility Program for
Adolescent Girls. Health Promot Pract. 12(3), 463-471.
Smart Girls Adolescent Pregnancy Prevention Programs. (2013). Retrieved from:
http://www.myguilford.com/humanservices/health/health-and-wellness/health-education- andtraining-2/teen-pregnancy-prevention/smart-girls-adolescent-pregnancy-prevention-programs/

SMART GIRLS EVALUATION

11

Appendix A
Title and Description of Modules
Introductions: The goal for this module is to introduce rules and expectations for the class, as well as a
distribute a pre-test to the participating girls. During this time, the material is introduced, parent/guardian
permission forms are collected, girls review and sign the participation agreement, and the pre-test is
completed.
Self-esteem: The goal for this module is for the girls to define and recognize contributing factors of selfesteem. During this time, there is an interactive discussion overviewing self-esteem. There are activities
that accompany the topic of self-esteem such as, an ice-breaker, dealing with boosters and downers, and
daily affirmation. At the completion of the module, the girls are asked to write down a question and drop
it in the questions box. These questions will be reviewed at the start of the following class meeting.
Decision Making: The goal for this module is for girls to identify decision making values that resonate
with them and are then asked to rank those values. Prior to delving into the curriculum, the instructor
reviews questions from the previous class in the questions box. During that time, the girls are given the
opportunity to discuss the meaning of values and are asked to list specific values as they relate to school.
Again as previously done, the girls are encouraged to write down questions and drop them in the
questions box.
Peer Pressure: This modules goal is for girls to acquire the skills to actively say no to peer and
relationship pressure based on their values, which were identified during the previous class. The girls are
also encouraged to identify the effects it may have on self-esteem. Prior to delving into the curriculum,
the instructor will review questions from the questions box. Once the questions are reviewed, the focus of
class is on peer pressure. There is discussion around peer pressure and the definition. To support the
overall module, there are activities that focus on being assertive to peer pressure and how to say no and
still keep friends. Like previous classes, the girls will be encouraged to leave questions in the box to be
reviewed next class.
Anatomy/Bodies: The goal for this module is girls will learn about female anatomy and how the
reproductive system works. Prior to delving into the curriculum, questions from the questions box will be
reviewed from last week. Following, will be a discussion surrounding the female reproductive system.
Discussion will include an activity that provides the opportunity for girls to label parts of the reproductive
system. There will also be a discussion surrounding the menstrual cycle and reproducing children.
Concluding the class, girls are encouraged to leave questions in the question box.
Contraception: The goal for this module is for girls to learn what contraception is, how it works, their
options, and how to discuss birth control with partners. Like always, questions from the questions box
will be discussed from last week. The class will continue with a discussion and activities focusing on why
teens wait for sex, myths and facts about birth control, and talk it out role play. Concluding the class,
girls are encouraged to put questions in the question box for next week.

SMART GIRLS EVALUATION

12

STDs/STIs: The goal for this module is for girls to learn about the different types of STDs/STIs and how
to deal with them. Before beginning the course, the questions from the question box will be reviewed.
There will then be discussion and activities focusing on common STDs/STIs, how STDs/STIs are
transmitted, and role play for how to ask a partner about their status. Girls are then encouraged to add
questions in the question box.
Teen Dating Abuse: The goal for this module is for girls to be a able to recognize the signs of an
unhealthy relationship and how to intervene or stop one. First, questions in the question box will be
reviewed. After, discussion and activities focus on what dating means, the differences between a healthy
and unhealthy relationship, and a case study on dating abuse. The girls are encouraged to leave questions
in the question box.
Goal Making and Post-test: The goal of this module is for girls to know the importance of making goals
and how to achieve those goals. First, questions from the question box will be reviewed. After, discussion
and activities will focus on why setting goals are important, how to set sexual limits, and map out life
plans (girls will be give sentence prompts for this activity). Once the discussions and activities are
completed, the post-test will be distributed.

SMART GIRLS EVALUATION

13
Appendix B
Evaluation Crosswalk

SMART GIRLS EVALUATION

14

Appendix C
BUDGET

DETAILED BUDGET FOR INITIAL BUDGET PERIOD

DIRECT COSTS ONLY

FROM

THROUGH

Sept 2016

May 2017

List PERSONNEL (Applicant organization only)

Use Cal, Acad, or Summer to Enter Months Devoted to Project
Enter Dollar Amounts Requested (omit cents) for Salary Requested and Fringe Benefits
NAME

Agency Head

ROLE ON
PROJECT

PD/PI

Cal.
Mnths

Acad.
Mnths

Summer
Mnths

40,000

12,000

N/A

12,000

30,000

9,000

N/A

9,000

10,000

4,000

N/A

4,000

Project
6
Coordinator Coordinator

Graduate
Staff
8
Assistant
Graduate

Staff
8
Assistant

INST.BASE
SALARY

SALARY
REQUESTED

FRINGE
BENEFITS

TOTAL

10,000

4,000

N/A

4,000

29,000

SUBTOTALS
CONSULTANT COSTS

N/A
EQUIPMENT (Itemize)

Voice recording technology for qualitative surveys, 100

SUPPLIES (Itemize by category)

8 worksheets at 6 sites for 45 students at each site, .05 for printing costs = $108
Pen and paper = $15
Surveys at 6 sites, administered twice to 15 key informants + 45 parents + 45
students at each site at .05 printing costs = $63
6 teachers, 8 fidelity checklists per teacher, .05 for printing = $2.40
*Above numbers reflect for one group, total reflects for both groups
TRAVEL

Graduate assistants from the office to the school site for 96 (for both groups)
observations, .53 cents per mile for an average 20 mile round trip

0
100

376.80*

1,017.60

SMART GIRLS EVALUATION

INPATIENT CARE COSTS
OUTPATIENT CARE COSTS

15

ALTERATIONS AND RENOVATIONS (Itemize by category)

OTHER EXPENSES (Itemize by category)

CONSORTIUM/CONTRACTUAL COSTS

DIRECT COSTS

SUBTOTAL DIRECT COSTS FOR INITIAL BUDGET PERIOD (Item 7a, Face Page)
FACILITIES AND
ADMINISTRATIVE COSTS

CONSORTIUM/CONTRACTUAL COSTS

Page

$ 30,494.4
0

TOTAL DIRECT COSTS FOR INITIAL BUDGET PERIOD

PHS 398 (Rev. 6/09)

1494.4

Form Page 4

SMART GIRLS EVALUATION

16

Appendix D
Timeline for Smart Girls Evaluation
Group 1: Gray
Group 2: Blue
Group 1 & Group 2: Green
Evaluation
Activity
Pretest for
students
In-depth
interviews
with
knowledgeabl
e informants
Classroom
observation
Qualitative
Interviews
with
parents/guardi
ans
Testimonials
through
worksheets
from students
Analyze
fidelity
checklist
Posttest for
students
Analyze data
(i.e.
pre/posttest,
interviews, &
observations)

Sept.
2016

Oct.
2016

Nov.
2016

Dec. 2016

Jan. 2017

Feb. 2017

March
2017

April
2017

May
2017

SMART GIRLS EVALUATION

17