Sample Clinical Performance

Improvement Project (PIP)

Implementing Recovery-Based Assessments in
Behavioral Health Settings that Serve Transitional Age
Youth (TAY)

Overview of the PIP Process

Step 1: Select & Describe the Study Topic
Step 2: Define the Study Questions
Step 3: Use a Representative and
Generalizable Study Population
Step 4: Select the Study Indicator(s)
Step 5: Use Sound Sampling Techniques
Step 6: Reliably Collect Data
Step 7: Analyze Data and Interpret Study Results
Step 8: Implement Intervention and Improvement
Strategies
Step 9: Plan for Real Improvement
Step 10: Achieve Sustained Improvement

PIP Must be Client Centered

IDENTIFY PROBLEM

Impact on
Programs

Impact on Family
INTERVENTION

Impact on
Community

Impact on
CLIENT

Step 1: Select and Describe the Study

Topic
Define the Problem
Transitional Age Youth (TAY) often desire to play a more
active role in their treatment and need more pathways to do
so.
Scores on Satisfaction with Treatment Planning were low on a
Consumer Perception Survey.
On the Recovery Self-Assessment in one county, the lowest
of the five scale scores was on Involvement in Treatment.
46% of interviewed county administrators mentioned wanting
a TAY-specific standardized measure to compare programs.
Current assessments of the effectiveness of TAY programs
might lack enough client voice about program impact.
Although self-report recovery-oriented measures have been
shown to be useful for adults, little is known of their impact
with TAY populations.

Step 1: Select and Describe the Study

Topic
What is the overarching goal of the PIP?
The overarching goal of this PIP is to
implement recovery-based self-report
outcomes measures that are expected to
help with individual client recovery through
better shared decision making and
monitoring of progress, as well as increased
understanding and engagement of
Transitional Age Youth clients with their own
recovery process, within the two years of
the PIP project.

Step 1: Select and Describe the Study

Topic
This might be accomplished through:
Stakeholder-driven selection of appropriate measures
Ample training and support for clinicians in using the
measures to engage clients and aid in their recovery
Adequate funding and time for providers to
administer and use the self-report measures in
treatment
Countywide training and support for interpreting the
results at both the individual and aggregate levels,
for individual treatment and aggregate program
planning

Step 2: Define the Study Questions

MAIN STUDY QUESTIONS

Does implementing self-report recovery-oriented measures

help Transition Age Youth clients in their recovery and
increase their perception of being in control of their own
treatment?
Does implementing a TAY-specific outcomes and satisfaction
measure across all local TAY consumers and programs provide
clinical and administrative insight into TAY systems of care?

SECONDARY STUDY QUESTIONS

What are the other specific impacts of implementing these

measures on treatment and program planning?
What are the perceptions of staff and clients of the
usefulness of these measures, and is the burden of
completion minimal?

Step 3: Use a Representative and

Generalizable Study Population
Primary data collection is through the MHSIP
State Consumer Perception Survey.
Thus, our study population will be all TAY
clients who receive services at an eligible
program during a 1-week period.
A representative sample of clinical staff will
be used for the staff survey, with information
about job position to ensure inclusive
representation.
Focus groups may be less representative and
generalizable, but provide richer information.

Step 4: Select the Study Indicator(s)

Study indicators will be collected from
multiple sources including, but not limited to:
1. Client data
Recovery Self-Assessment (RSA-R) or
similar
MHSIP State Consumer Perception Survey
Focus groups with clients to ask about how
the measures help in their recovery and/
or affect their perceptions of involvement
in treatment

Step 4: Select the Study Indicator(s)

2. Staff (program managers, clinicians,
others):
Recovery Self-Assessment (RSA-R) or similar
Staff survey on impact of the
implementation (ratings and open-ended
items)
Focus groups with staff to ask about
measures implementation, burden, clinical
usefulness, and impact on client recovery

Step 4: Select the Study Indicator(s)

Specific examples of some study indicators from these
measures (the specific measures for this PIP are still being
determined):
Perception of positive outcomes from services (MHSIP)
Impact on functioning (MHSIP)
Client satisfaction in general (MHSIP)
Satisfaction with Treatment Planning (MHSIP)
Client involvement/engagement (RSA-R)
Client perception that they control their treatment (RSAR)
Focus groups with staff and clients can tap into any of
these indicators and reveal unexpected benefits or
consequences

Step 5: Use Sound Sampling Techniques

Demonstrate that the data collection
methods currently used for the State
Satisfaction and RSA-R survey are sound.
Include programs for all TAY groups of
interest.
Analyze past data for non-response bias.

Step 6: Reliably Collect Data

State MHSIP Consumer Perception Survey

RSA-R or similar measure administered with it

Inclusion of all relevant programs
Multiple languages as necessary
Clear instructions for respondents and those
administering the surveys at the programs

Web Survey of Staff

Sufficient sample size
Inclusionary recruiting methods

Step 7: Analyze Data and Interpret Study

Results
Answer research questions:
Did the intervention have an impact?
Was it differential by group?
Age (16-18 vs. 23-25)
Diagnosis
Race/Ethnicity/Cultural Factors
Gender Identity
Program Type

Complete Model
Data Sources
Client-reported Data
Focus groups and a questionnaire
administered with the State MHSIP
Consumer Satisfaction Survey

Staff Data
Focus groups and a web-based
questionnaire administered to staff

Does implementing self-report recovery-oriented measures improve TAY

recovery and satisfaction?
What are the specific impacts of implementing these measures?
Is the impact of the intervention differential by group (e.g., age, diagnosis,
program type)?
What are the perceptions of staff and clients of the usefulness of these
measures?

Step 8: Implement Intervention and

Improvement Strategies
What next?
Using what you have found, develop a plan:
Implementation:
If there was an impact, how does this affect future
plans? Will we continue? Will we expand?
What if there was no impact or minimal impact?
Are there adequate resources to support scaling
up?
Improvement:
Did we select the right self-report measure?
What additional training is needed by providers?
Can we make it more culturally competent/
sensitive?

Steps 9 and 10: Plan for Real Improvement

and Achieve Sustained Improvement
It is essential to determine if the reported change is real
change, or the result of an environmental or unintended
consequence, or random chance.

The following questions should be answered in the

documentation:
Reliability and validity of measures used to examine
PIP
Strength of the statistical evidence supporting that
the improvement is true improvement (not due to
chance)
Plan for monitoring and sustaining improvement

Thank you!
Andrew Sarkin, PhD
asarkin@ucsd.edu
hsrc.ucsd.edu
mhse.ucsd.edu

Simple Conceptual Model

Does implementing self-report recovery-oriented
measures help Transition Age Youth clients in their
recovery?

Clients before

Clients without selfreport measures

Client Outcomes

PIP Validation Tool