Evaluation of Implementing Partnership for

Health at University of North

Carolina-Greensboro Student Health Services

HEA 625
December 3, 2017

Submitted by:
Rachel Faller
Breana Floyd
Haley Love
Section 1: Conceptualization of Evaluation

Purpose and Overall Questions:

This evaluation plan discusses a formative evaluation of the Partnership for Health (PfH)
program at University of North Carolina at Greensboro (UNCG) Student Health Services (SHS). PfH is a
High-Impact Prevention (HIP) intervention intended for HIV-positive patients at community-level HIV
clinics. Because UNCG is a different environment than where PfH is traditionally implemented,
adaptations may need to be made to improve its efficacy. The purpose of this evaluation is to determine
how to improve the PfH intervention at UNCG SHS. The overall questions for this evaluation are:
● How is PfH implemented/delivered?
● What do providers and participants think of PfH?

Key Stakeholders:
PfH was developed by the University of Southern California (USC) and will be implemented at
UNCG SHS through funding from the Centers for Disease Control and Prevention (CDC). Key stakeholders
include:
● UNCG SHS providers and staff​ will implement the intervention, collect data on health conditions
and behaviors of participants, and communicate and interact with participants.
● UNCG students and faculty/staff who are HIV-positive and aware of their status ​will be the
individuals receiving PfH information and services.
● UNCG students and faculty/staff who are HIV-negative or unaware of their status ​may be
exposed to information on HIV/STD testing and/or safe sex practices through PfH participants or
marketing material. Also, they utilize SHS services and interact with HIV-positive individuals at
UNCG.
● Family members and close friends ​who​ ​communicate and interact with HIV-positive individuals
at UNCG may be exposed to information on HIV/STD testing and/or safe sex practices through
PfH participants or marketing material.
● Greensboro community members​ who interact with HIV-positive individuals at UNCG may be
exposed to information on HIV/STD testing and/or safe sex practices through PfH participants or
marketing material.
● CDC ​will fund the intervention and collect data from UNCG SHS for reporting purposes.
● The evaluation team ​will​ ​consist of UNCG graduate students from the Department of Public
Health Education (PHE) and will work with stakeholders to collect information on how PfH can
be improved.

Assumptions:
One assumption is there are individuals at UNCG who are HIV-positive, use SHS services for
HIV-related treatment, are aware of their status, and engage in risky behaviors leading to HIV
transmission. We also assume there is a large enough population willing to disclose HIV status and
participate to make collecting surveys, conducting interviews, and conducting a focus group feasible.
A second assumption is UNCG SHS has capacity to perform necessary functions for successful
implementation of PfH. Providers will need to take time to discuss sexual risk behaviors with
HIV-positive patients. If there is not enough time for these discussions, then implementation and
evaluation will be unsuccessful because not enough patients will receive the intervention and not
enough providers will conduct the intervention. Similarly, staff will need to take time to collect surveys
and participate in the focus group for the intervention and evaluation to succeed.
We also assume responses to surveys and discussions in the focus groups are honest
representations of patient and provider perceptions of PfH.
1
 Additionally, we assume PfH addresses concerns specific to HIV-positive patients at UNCG. PfH
was originally intended for local HIV clinics and may not address concerns specific to HIV-positive
students or faculty/staff at UNCG. One purpose of this evaluation is to determine if changes PfH are
needed to make it more useful to this population.
Further, we assume the incentives offered for completing the survey and/or participating in
focus groups are enough to encourage participation (see ​Data Collection Instrumentation and Plan
below). Some patients may not think a 20% coupon or $10 Visa gift card is worth the time or effort of
participating. Also, some staff or providers may not feel lunch is incentive enough to participate in the
focus group.
Finally, we assume we will receive IRB approval for the evaluation and patient consent to
participate.

Contextual Factors:
Stigma surrounding HIV infection is one factor that could affect the intervention. This stigma
may make it difficult to persuade patients to complete surveys and recruit patients for interviews and
the focus group. Stigma could also make HIV-positive individuals at UNCG hesitant to speak with
providers about their disease and sexual behaviors; similarly, some providers may not feel comfortable
talking to an HIV-positive patient about these topics. We may also not be able to reach all HIV-positive
individuals at UNCG because they may not be aware of their status. Some HIV-positive individuals may
not want to go to SHS for care, preferring to go a clinic or personal physician anonymously. Stigma could
therefore make successful implementation of PfH at SHS difficult or impossible.
Another factor is SHS serves all students and staff and therefore could have priorities in other
areas of health. Focusing on other priorities might cause SHS staff and providers to neglect
implementation of PfH.
Finally, PfH is an intervention originally designed to be used in local HIV clinics. SHS is different
since it handles a variety of health conditions. PfH may not be implemented as successfully at SHS
because of these differences.
There are many other contextual factors; however, the evaluation team determined these three
as most likely to have major influence on implementation and evaluation of PfH.

Section 2: Program Description

Purpose:
The goals of PfH are:
● “To train health care providers and staff in HIV outpatient clinics to talk with their patients about
the importance of protecting themselves and their sex partners and disclosing their HIV status to
sex partners before having sex with them.
● To improve patient and provider communication about safer sex and disclosure.
● To decrease unsafe sexual behaviors among persons living with HIV.
● To increase disclosure of HIV status to sex partners” (Richardson et al., 2004a).

Program Objectives:
● All health center staff trained and capable of facilitating prevention counseling.
● Providers deliver PfH to all HIV-positive patients visiting SHS.
○ Provider engages with all HIV-positive patients briefly (for 3-5 minutes) on disclosure of
HIV status and self and partner protection.

2
 ○ Provider delivers techniques on effective communication for all HIV-positive patients
reporting engagement in high-risk sexual behaviors (i.e., multiple partners; unprotected
sex).
○ Providers offer free condoms to all HIV-positive patients during their appointment.
○ Providers give all HIV-positive patients a PfH brochure.
○ All HIV-positive patients visiting SHS develop at least one behavior goal, with assistance
from provider if needed, and providers record goals in patient’s medical record. During
subsequent visits, providers follow-up with all HIV-positive patients about progress in
achieving their goals.
● Providers make referrals for all HIV-positive patients needing counseling and services not
provided by SHS.

Target Audience and Location:

● The target audience of PfH are HIV-positive individuals at UNCG aware of their status.
● The location of PfH implementation is UNCG SHS.

Previous Evaluations:
PfH underwent a summative evaluation to assess “efficacy of brief, safer-sex counseling by
medical providers of HIV-positive patients during medical visits” (Richardson et al., 2004b). Results of
the evaluation indicated brief provider counseling emphasizing negative consequences of unsafe sex
decreased HIV transmission behaviors (i.e., self-reported unprotected anal or vaginal intercourse [UAV])
in HIV-positive patients. UAV was reduced 38% in patients with two or more sex partners; similar results
were seen in patients who had casual sex partners (Richardson et al., 2004b).
We were unable to find published materials on a formative evaluation of PfH.

Logic Model:
PfH’s logic model is detailed in ​Table 1​ below. Inputs to PfH include trained providers, SHS staff,
UNCG resources (e.g., SHS office, etc.), money (CDC funding), volunteers (to assist with implementation
and data collection), and medical records (to identify HIV-positive patients). These inputs lead to activity
and participation outputs. Activity outputs include tangible intervention materials (i.e., posters, flyers,
brochures, etc.) and provider-to-patient education on reducing HIV transmission risks. Participation
output is the percentage of HIV-positive individuals receiving tangible activity outputs and/or
provider-to-patient education. Short-term outcomes for PfH, increased knowledge of HIV transmission
risks and consequences of unsafe sex behaviors, are immediate effects of activity outputs. Intermediate
outcomes, increased communication between patient and provider and decreased high-risk sex
behaviors, result from short-term outcomes. Long-term outcomes result from these intermediate
outcomes, i.e. decreased stigma resulting from increased communication and decreased HIV incidence
resulting from decreased high-risk sex behaviors. Finally, long-term outcomes help fulfill public health
goals: normalization of HIV and community awareness (through decreased stigma) and decrease in HIV
prevalence (through decrease in HIV incidence).

Table 1. PfH Logic Model

Inputs Outputs Outcomes Goals

Activity Participation Short-term Intermediate Long-term

-Providers -Posters -Percentage -Increased -Increased -Decreased -Normalizatio

3
 -Medical -Flyers of knowledge of communicatio stigma n of HIV and
records -Brochures HIV-positive HIV n between -Decreased community
-Provider -Provider individuals transmission patient and HIV awareness
training pocket receiving risks provider incidence -Decrease in
-UNCG guides materials -Increased -Decreased HIV
resources -Chart and/or knowledge of high-risk sex prevalence
and stickers provider consequence behaviors
participation -Provider-to- education s of unsafe
-Money patient sex behaviors
-SHS staff education
-Volunteers

Section 3: Evaluation Plan

Evaluation Questions and Variables:

Evaluation questions and variables are listed in the ​Evaluation Crosswalk​ below (see ​Table 2​).
Under each question, in the far left column, are the variable(s) being used to answer the question. The
data collection instrument(s) being used for each variable are indicated by an “X” in the corresponding
column(s).

Table 2. Evaluation Crosswalk

Question and ​Variable(s) Pre-Intervention Interviews Satisfactio Patient Provider Medical
Surveys n Surveys Focus Focus Records
Group Group

How do patients and providers feel about the intervention?

Patient satisfaction with X X X

communicating with
providers

Patient satisfaction with the X X X

intervention

Provider self-efficacy X
satisfaction in carrying out
the intervention

Provider satisfaction with X

the intervention

Who received PfH training prior to program implementation?

Number of providers X
receiving PfH training

Who previously received information presented in the intervention from a provider?

Number of patients X
receiving previous provider
education on HIV

4
 Type of previous provider X
education HIV patients
received

How many HIV-positive patients received a brochure during their visit?

Number of patients X
receiving a brochure

How many HIV-positive patients responded to the survey?

Number of patients X X
completing the
pre-intervention and
satisfaction surveys

How many HIV-positive patients developed at least one behavioral goal during their visit?

Number of patients setting X

at least one behavioral goal

What were the goals that HIV-positive patients developed?

Types of behavioral goals X

set by patients

How many HIV-positive patients were given condoms during their visit?

Number of patients X
receiving condoms during
the appointment

What was the average length of time (in minutes) providers communicated with patients about risky sexual
behavior and consequences of the behavior?

Length of X
provider-to-patient
intervention (in minutes)

Do patients believe this intervention is needed to address risky sexual behaviors in HIV-positive individuals?

Patient attitudes on need X X

for HIV risk education in
HIV-positive persons

Evaluation Design:
The proposed formative evaluation is a case study since the purpose is to create an in-depth
understanding of PfH implementation at UNCG SHS. The case study will be investigated over the course
of of one semester. Six months prior to the semester will be used for recruitment, training, and planning
as well as review of medical records. The case study has a quasi-experimental design because there is no
random assignment; all HIV-positive patients will receive the intervention.

5
 SHS staff will use medical records to identify HIV-positive patients. A pre-intervention survey will
be given to patients prior to their appointments to assess if they previously received provider education
on HIV and, if so, what kind (e.g., transmission risk, condom use, etc.)​.
Interviews will be conducted twice a month by an evaluator during the semester. Interviewees
will be recruited through answering a question on the satisfaction survey (see ​Data Collection
Instrumentation and Plan​ below) agreeing to be contacted for an interview. Interviews will be done at a
time and place convenient to the patient.
During the semester, each HIV-positive patient attending SHS will be given a paper satisfaction
survey after their appointment and before going to the checkout desk. If the patient turns in a
completed survey to the checkout desk, then they will receive a 20% off coupon for UNCG’s student
bookstore. The surveys will help the evaluation team quantify satisfaction with the intervention.
At the end of the semester, participating patients will be contacted in random order and asked
to participate in an hour-long focus group at Elliott University Center (EUC). The recruiting process will
continue until ten participants agree to participate. Focus group participants will receive a $10 Visa gift
card.
Similarly, SHS staff and providers will be invited to participate in a separate hour-long focus
group. Staff and providers will be recruited through an email sent twice a week inviting them to
participate in the focus group. Emails will be sent until five staff/providers agree to participate in the
focus group. Participants will be provided lunch. Focus groups will provide detailed data describing
satisfaction with the intervention and improvements to be made.
Medical records of HIV-positive patients will be utilized to determine details of patient-provider
encounters (i.e., length of provider-to-patient education, behavioral goals set by patients, and if patients
received condoms and/or a brochure from providers).

Data Collection Instrumentation and Plan:

● Pre-Intervention Surveys ​will assess if patients had any previous provider education about HIV
risk behaviors and, if so, what topics were covered (e.g., condom use, risk of transmission, etc.).
SHS staff will provide patients with a short survey to complete while patients are waiting for
their appointment. Staff will collect the surveys when patients are called for appointments.
● Interviews ​will detail patients’ satisfaction with PfH. A member of the evaluation team will
conduct two interviews per month during the semester, for a total of eight interviews. Each
month, two random participants answering “Yes” to a question on the satisfaction survey about
willingness to participate in an interview will be recruited by SHS staff. A member of the
evaluation team will contact the participant and conduct an interview at a convenient time and
place for the patient. Each interview will last 15 minutes and interviewees will receive a $10 Visa
gift card.
● Satisfaction surveys​ given to all participants will quantify patient satisfaction with PfH. After
appointments with HIV-positive patients, providers will give out paper satisfaction surveys. The
patient will turn in the survey at the front desk. Those turning in the survey will receive a 20%
off coupon for the student bookstore.
● An hour-long patient focus group​ will consist of ten participants and collect information on
patient satisfaction with PfH, improvements to be made, appropriateness of
materials/education received, positive or negative effects on patient-provider communication,
and other topics of importance to implementation of PfH that may arise. After the semester,
patients receiving the intervention will be contacted by SHS staff in a random order and asked to
participate in the focus group. Recruitment will continue until ten participants agree to attend.
The focus group will be conducted in EUC. Participants will receive a $10 Visa gift card.

6
 ● An hour-long provider focus group​ with providers and staff at SHS will collect information on
provider satisfaction with PfH, provider self-efficacy in implementing PfH, improvements to be
made, positive and negative effects on patient-provider communication, and other topics of
importance to implementation of PfH that may arise. Providers and staff at SHS will be invited to
attend a focus group at EUC through an email sent twice a week from the evaluation team.
Emails will be sent until five providers agree to participate in the focus group. The focus group
will last one hour and lunch will be provided.
● Medical records​ will be reviewed to determine HIV status (to identify patients for the
intervention), length of provider-to-patient education, what behavioral goals the patient
developed, and if the patient received condoms and/or a brochure from the provider. Relevant
de-identified data from medical records will be collected by SHS and given to evaluators for
analysis.

Section 4: Reporting Plan

Once data from the evaluation is collected, evaluators will report their findings to all potential
stakeholders of this program intervention and evaluation. Evaluators will prepare four deliverables that
include the following information:

● a description of the PfH program,

● goals of the PfH program,
● the group being targeted by this intervention,
● a description of the stakeholders affected by this intervention,
● a description of where the intervention was implemented,
● details of the data collection instruments,
● the type of evaluation conducted,
● the purpose of the evaluation,
● evaluation questions, and
● results of the evaluation.

To reach all stakeholders of this intervention and evaluation, the deliverables will include an
infographic, a brochure, an article to be published in an academic journal, and a poster to be presented
at a conference.
● Infographic - ​An infographic will creatively display information in a way that all people,
regardless of their education level or knowledge of the content, can understand. The infographic
will be displayed a) physically as flyers posted in the SHS office, in county and/or state health
departments, as well as in various primary care offices and b) digitally on televisions that are
available in the aforementioned settings. Evaluators will communicate with the staff and
providers to determine if they want to display the infographics at the aforementioned locations
and, if so, to coordinate where in these facilities to display the infographics.
● Brochure - ​Brochures are a convenient method of disseminating detailed information because
they allow stakeholders to receive information at their own pace and in their own space. Thus,
brochures will be placed in the SHS office, in county and/or state health departments, as well as
in various primary care offices. Evaluators will communicate with the staff and providers at
these locations to determine if they want to disperse the brochures and if so, where in these
facilities to display them.
● Journal Article - ​Information gathered from this evaluation can be used to inform service
providers, academic institutions, and other stakeholders about the benefits and challenges of
7
 developing and implementing interventions for individuals with HIV/AIDS who engage in risky
sex behaviors. Thus, evaluators will develop a report based on the information about the PfH
intervention and evaluation. The report will be submitted to and hopefully published in an
accredited journal in a related field of study. If the article is published, anyone at any time will
be able to access the data from this evaluation and apply it to their work in better serving
individuals with HIV/AIDS.
● Poster - ​A poster will be developed to disseminate information about the PfH intervention and
evaluation findings. This poster will be publicly displayed in the PHE office at UNCG for students,
professors, and other individuals to read and discuss at their leisure. Additionally, a proposal will
be drafted and submitted for the poster to be presented at a conference on topics related to
HIV/AIDS. If the proposal is accepted, the poster will be accessible to a wider audience than the
UNCG PHE office, but the information presented in the poster will only be disseminated for a
certain amount of time at the conference.

Section 5: Budget

Table 3​ below outlines the proposed budget for the evaluation. The budget is divided into four
sections: personnel, materials/equipment, incentives, and overhead. Note that program costs, such as
visual materials and provider time to provide patient education, are not included as they are not part of
the evaluation. Also, travel and per diem are not needed for this evaluation, as all personnel are local
and no travel is anticipated.
● Personnel​ includes costs associated with evaluation staff (Graduate Assistants [GAs]), SHS staff
assisting with the evaluation, Undergraduate Research Assistants (URAs), and a statistician
consultant. Salary estimates for GAs are based on current standards for UNCG assistantships
($18/hour, 20 hours/week). SHS staff pay estimates are based on data from the United States
Bureau of Labor Statistics (BLS) (BLS, 2016). Five SHS staff will assist with collection and entering
of sensitive data. We anticipate this will take 10% of staff time (4 hours/week for 7 months).
Volunteer URAs will participate in non-sensitive data collection and entry as needed. Finally, a
statistician will consult and assist with data analysis for up to 80 hours during the course of the
evaluation.
● Materials/Equipment​ includes facilities and equipment not included in Overhead costs (see
Overhead​ below), printing of surveys and reports, reporting materials, and general office
supplies. Note that computers are not included in the table below since all personnel will either
use personal computers (for tasks without sensitive data) or UNCG computers, which are
covered under ​Overhead​. The Birch Room in EUC will be reserved for the focus groups since it
has sofas and chairs and may be more comfortable for participants. Survey and report printing
estimates are based on prices from staples.com for color printing (Staples, 2016). Printed
materials include posters and brochures which will be used for reporting (see ​Reporting Plan
above) and estimates are from posterpresentations.com, fedex.com, and staples.com (FedEx,
n.d.; PosterPresentations.com, n.d.; Staples, n.d.). General office supplies, such as pens and
paper, are also included in this section. Submitting evaluation findings to a journal will be
considered professional development and will have no other costs (journal will be a free journal
to submit to).
● Incentives​ for the participants include Visa​Ⓡ​ gift cards and a catered lunch. Coupons for the
student bookstore, which are provided to patients completing the satisfaction survey, are not
included in the budget since they are an incentive to purchase items that may not have been
purchased otherwise; therefore, the coupons will generate money for the bookstore rather than

8
 take it away. Visa​Ⓡ​ gift cards will be given to 18 participants. A catered lunch from Panera Bread
Company will be provided to SHS staff and providers participating in the provider focus group.
● Overhead​ covers general operations, facilities, and services provided by UNCG and is calculated
as 46% of the total cost of the evaluation before overhead is included. These include
communication (telephones, fax machines), computer software, Internet access, miscellaneous
printing, GA/URA offices, financial services, etc.

Table 3. Evaluation Budget

BLS, 2016; 2​​ UNCG, EUC, n.d.; 3​​ Staples, n.d.; 4​​ PosterPresentations.com, n.d.; 5​​ FedEx, n.d.; 6​​ Panera Bread Company,
1​

n.d.

Section 6: Timeline

9
10
 Works Cited

Bureau of Labor Statistics (BLS). (2016).​ ​Occupational Employment and Wages, May 2016. Retrieved
from: ​https://www.bls.gov/oes/2016/may/oes434171.htm​.

Centers for Disease Control and Prevention (CDC). (2017). Behavioral interventions. Effective
Interventions - HIV Prevention That Works. Retrieved from:
https://effectiveinterventions.cdc.gov/en/HighImpactPrevention.aspx

FedEx. (n.d.). Posters, Signs, and Banners. Retrieved from:

http://www.fedex.com/us/printing-services/poster-sign-banner-printing.html​.

Panera Bread Company. (n.d.). Catering. Retrieved from: ​https://catering.panerabread.com/#!/​.

PosterPresentations.com. (n.d.). Professional Tri-Fold Science Poster Boards. Retrieved from:

https://www.posterpresentations.com/html/science_poster_board_display.html​.

Richardson JL, Stoyanoff S, Hawkins M, Weiss JM; University of Southern California (USC), Keck School of
Medicine; Neumann MS; Centers for Disease Control and Prevention (CDC). (2004a). Partnership for
Health: brief safer sex intervention for HIV outpatient clinics. Starter kit. Available at
https://effectiveinterventions.cdc.gov/Files/Starter_Kit_Guide.pdf​.

Richardson, J. L., Milam, J., McCutchan, A., Stoyanoff, S., Bolan, R., Weiss, J., ... & Chou, C. P. (2004b).
Effect of brief safer-sex counseling by medical providers to HIV-1 seropositive patients: a multi-clinic
assessment. ​Aids​, ​18​(8), 1179-1186.

Staples. (n.d.). Print and Marketing Services​. Retrieved from:

https://www.staples.com/sbd/content/copyandprint/index.html​.

University of North Carolina at Greensboro (UNCG), Elliot University Center (EUC). (n.d.). Rates and Fees.
Retrieved from: ​https://euc.uncg.edu/reservations/rates-and-fees/​.

11