CBR 106:

Developing a Client Satisfaction Survey

Welcome & Introductions
Name & affiliation

Experience with client satisfaction surveys

What you want to learn today



Workshop Objectives
Upon completion of this workshop, participants will be able
to:

Describe the importance and benefits of Client
Satisfaction Surveys (CSS)
List the steps required to develop a CSS tool
Analyze and critique sample CSS questions
Begin the development of their own CSS tool


Agenda


Definition of terms
Purposes and benefits of CSS
Steps in the CSS development process

Lunch

Analysis & critique of CSS tools and questions
Development of your own CSS tool
Workshop Evaluation


Surveys

Knowing what the client wants is the key factor to success in
any type of organization or business

Organizations need to know what their members want

The best way to find this information is to conduct a survey
What is a Client Satisfaction Survey (CSS)?


A relatively simple but effective way to evaluate the delivery
and quality of the services your agency provides to clients

Provides you with data that tells you what clients really
think about your services and the people that deliver them
(rather than what you think they think!).


What does a CSS do?
Identify trends in client satisfaction
Measure results over time
Identify new service areas
Improve your existing services
Provide indicators of agency and staff strengths and areas
of improvement
Improve accountability
Benefits of CSS

An opportunity to target areas for (continuous) improvement

An opportunity to present the full range of services offered to
your clients on an annual basis


Clear Objectives
The Clients
The Survey Questions
The Results
Implementing Change

What Makes a Successful CCS?
The Steps in a CSS Project
Establish the goals of the project
Determine your sample
Choose interview methodology
Design questionnaire
Pre-test the questionnaire
Administer survey
Collect and enter data
Analyze data
Present data
Step 1: Establish Goals
Establish the goals of the project

What do you want to learn?
What will you measure (e.g. overall service, specific services,
program efficiency or responsiveness, etc.) ?
Why do you want to measure this factor?
What are the possible benefits?

Try not to mix too many purposes into a single CSS.
Keep it focused.
Step 2: Determine Sample
Determine your sample

Who will you interview?
Who is your clientele?
Will this group provide you with the necessary data to meet
the goals of the project?


Keep the sample large enough for a sufficient amount of
data but small enough so that it is manageable.


Step 3: Choose Interview Methodology
Choose an interview methodology

How will you conduct the interview?
- mail out; questionnaires; individual interviews; case
study; focus group; on-line surveys

Make up of your clientele
- Language; accessibility options; age; education
Step 4: Design Questionnaire
Create your questionnaire

What you will ask?
What do you NEED to know?
Will the questions asked meet the goals of the project?


TYPES OF QUESTIONS


Researchers often use a combination of QUANTITATIVE
and QUALITATIVE questions to assess client satisfaction.
Quantitative Questions
Aiming for clear answers (that can be quantified)

Closed questions
- (yes/no, agree/disagree, etc., answers)

Often focused on the descriptors/variables and their
relationship
- E.g. ---- # of days in hospital

The Checklist
Please select which in-patient services you used in the last
month?

Individual therapy
Group therapy
Medication
Twelve-step program (NA, CA, AA)
Other (please describe)
________________________



Ranking
Please order which in-patient services you found the most
helpful (where 1= most helpful & 5 = least helpful)?

_ Individual therapy
_ Group therapy
_ Medication
_ Twelve-step program (NA, CA, AA)
_ Other (please describe)
________________________




Rating / Likert Scale

Please rate the degree to which you agree/disagree with the
following statement:

I was completely satisfied with the in-patient
services I received.

Strongly agree Agree Disagree Strongly Disagree




Typically open-ended (e.g. how questions)

A way of finding out more in-depth information about what
client feels or experienced

Can be time consuming for client to respond to and for CSS
administrator to process
Qualitative Questions
Sample Qualitative Questions
How satisfied were you with the responsiveness of the
program staff in meeting your needs?

How might the program might improve its services to better
meet your needs?

Exercise 1 Evaluating Questions
Working in small groups discuss the differences
between questions 1 & 2.

In what ways have our services benefited you?

Can you tell us the ways in which our services have
affected you?
Exercise 2 Critiquing Questions
Critique the following questions:

1. How would you rate our organizations internal capacity
(circle one)?
1-poor 2-fair 3-good 4-great 5-excellent

2. Did the group sessions ever adversely affect you?
Structuring Your Questions: Things to Avoid
Avoid technical terms and acronyms, unless the
respondents know what they mean

Avoid using an odd-number rating scale, such as a 1-5
scale (e.g. poor, fair, good, great, excellent) as people
will tend to choose the middle choice

Avoid biased or loaded questions

Avoid questions that are unclear or can be interpreted in
more than one way



Structuring Your Questions: Dos
Ensure the questions are valid

Check that they really measure what they are aiming to measure

Ensure the questions are reliable

Ensure they are understood the same way by different people and
that a person answering similar questions will answer similarly

Make sure your questions accept all the possible answers

Depending on the goal of the project, include demographic
information


More Dos

Keep it short.
Long surveys are a turn-off to most people.

Keep it simple.
Limit the number of open-ended questions.

Keep it real.
Use language that is easily understandable to your client.

Make it look good.
Organize your questions logically, use a simple font and
keep it looking neat and tidy.
Exercise 3 Designing Questions

In your small group, develop a CCS (on flip chart paper) that
measures peoples satisfaction with the workshop thus far.

Determine specifically what you want to measure and why.

Include a combination of quantitative and qualitative
questions (5-8 total).

Keep in mind the question design dos and donts discussed
earlier.



Exercise 3 Continued The Gallery Walk
Post your groups CSS on the wall. Silently walk around the
room to look at the other groups CSSs.

Choose two (not your groups) to respond to. Record the
questions and your answers. No one will see your
responses.

Note similarities and differences between the design of CSSs
and whether and how the CSS design effects your
responses.
Debrief of Exercise 3

What did you notice about the various CSSs that
were created?
What was similar? What was different?
Any surprises?
What worked well?
What could be improved?
What did you learn about CSS design from this activity?
Step 5: Pre-test the Questionnaire
How to Pre-test?

Test questionnaire with a small number of interviews
Test the survey on the same kinds of people you will
include in the main study

Why Pre-test?

Pre-testing can reveal unanticipated problems with question wording,
instructions, structure, etc
It can help you see if the interviewees understand your questions
It helps to determine if the responses meet the objectives
Step 6: Administer CSS
Distribute the survey and include the following:

A cover letter/introduction explaining the purpose of the
CSS and how the information will be used
When it needs to be returned
A thank you to the client



Step 7: Collect & Enter Data
Collect and enter data

Expect to collect data 2-3 weeks after distributing

Determine ahead of time (before or while CSS is being
developed) data management program (e.g. NVIVO or
NuDist for qualitative; SPSS, SAS, Stata for quantative)

Determine ahead of time who will enter data; provide them
with adequate training


Step 8: Analyze Data
Identify themes & patterns
Frequencies
- counts of things, how many times things happen; get
mean, median, mode
Descriptives
- percentages or raw numbers of demographics and
other factors that would be nonsensical to take an
average of such as gender or ethnicity
Outliers
- things that fall outside the expected values

Step 9: Report Results
Produce the reports

Who is the target audience?
Identify specific areas for change
Identify action outcomes of the CSS
Make results available to clients
Exercise 4 Question Revision

Review your CSS questions with a partner and gather their
feedback.

Specifically focus on what extent are the questions:

Clear and simple?
Valid and reliable?
Unbiased?
Manageable in number?




Reflection & Next Steps
Reflect on and record 1-3 things you learned today.

Record 1-3 related goals you have going forward.

Record 1-3 related next steps you plan to take upon leaving
the workshop.
Workshop Objectives
Having completed this workshop you should now be able to:

Describe the importance and benefits of Client
Satisfaction Surveys (CSS)
List the steps required to develop a CSS tool
Analyze and critique sample CSS questions
Begin the development of their own CSS tool


Workshop Evaluation


Your feedback is extremely important!

Please complete the workshop evaluation.

Thank you!



CBR 106:
Developing a Client Satisfaction Survey