3rd OFID-Questionnaire Develop-NK 2012-FINAL PDF

A Tool Kit for Questionnaire
Development
Dr. Nadir Kheir PhD MPS FNZCP
Assistant Professor & Coordinator of Continuing
Professional Pharmacy Development
College of Pharmacy, Qatar University, Doha, Qatar
15-Feb-12
Questionnaire-OFID 2012-NKheir
The great popularity with questionnaires

is they provide a quick fix for research
methodology. No single method has been
so abused.
Gillham B. Developing a questionnaire (real world research). London: Continuum, 2000.
A study is unethical if it is scientifically

unsound, causes undue offence or trauma,
breaches confidentiality,
or wastes peoples time or money
Boynton P, Greenhalg T. Selecting, designing, and developing your questionnaire. BMJ 2004, 328: 1312-1315
15-Feb-12
Learning Outcomes
After the completion of this session, the participant will be
able to:
1.
Describe the main uses of survey questionnaires
2.
list the major stages of questionnaire development
process
3. discuss the known pitfalls that could affect questionnaire
development
4. Explain what is meant by reliability and validity
5. Examine questionnaire face validity
6. Develop a short questionnaire with consideration to the
main components of the development processes
15-Feb-12
Useful Resources
A guide to good survey design. Statistics New Zealand 1995.
Publishing & Media Services Division
Research methods knowledge base. Survey Research. Available in:
http://www.socialresearchmethods.net/kb/survey.php
QuestionPro (Survey software). Available in:
http://www.questionpro.com/akira/logout.do
15-Feb-12
Survey Research
Can be a questionnaire OR an interview

Important area of measurement in applied social research
Considered branch of social science
Any measurement procedures that involve asking questions
Can be a short paper-and-pencil feedback form to an intensive oneon-one in-depth interview
Study samples chosen from target population to discover
inferences
Study relations between behaviors and other sociological variables
Different from a census (where the whole population is studied)
15-Feb-12
Purposes of Surveys: Description

Making descriptive assertions about a population
Answering the what questions:
- what is the distribution of certain traits?
Examples:
-the attitude towards new service
-the quality of life of diabetics
-the skill level of vocational workers
15-Feb-12
Purposes of Surveys: Explanation

Making explanatory assertions
For example:
-why some persons abuse pain killers?
-why do students resort to cramming in exams?
-what are the reasons for the increased visits to the staff clinic
during 2010 compared to 2009?
Explanatory objective needs multivariate analysis
-Cramming could be explained by quality of teaching, number of
assessments, class size
- Abusing pan-killers can be associated with issues around access,
employment status, relaxed laws
- Increased visits to the clinic in 2010 could be explained by a big
conference at the University, an endemic, etc
15-Feb-12
Purposes of Surveys: Exploration

Also: pilot survey, Focus Group, qualitative research
Before a large study, you need to refine your
methodology through a dress-rehearsal survey
Focus group to explore deep meaning and complex
phenomena
No attempt to select representative sample
Might raise some questions and possibilities
They identify logistical, sampling, cost, design issues
In pilot study: a small sample size means its results
might not useable as stand alone data
15-Feb-12
Types of Surveys
1. Interviews: face-to-face, telephone
2. Mail surveys, Groups, drop-off surveys,
online surveys
3. The Focus Group
All Utilize a Questionnaire or questions
15-Feb-12
Questionnaire
Questionnaires are usually paper-and-pen or online
instruments that the respondent completes
Type of Questionnaire
Characteristics
Mail survey questionnaires
Adv: inexpensive; can send the exact instrument to a wide

number of people; allow the respondent to fill it out at their
own convenience
Disadv: Low response rate, not suitable for detailed
responses, errors, missing data, inaccuracies
Group-administered
questionnaire
Adv: convenience; high response rate; low errors

Disadv: difficult to assemble, costly, inconvenience to
participants
Household drop-off survey
Adv: the respondent can fill in private, when it's convenient;

the interviewer makes personal contact with the respondent
Disadv: response may have to be posted back or collected in
person
15-Feb-12
Developing a Questionnaire
A complex and iterative process
This means the designer needs to go back and
forth to resolve issues (change questions to suit
respondents leading to reduce/change objectives, etc)
Multiple evaluations during the developmental

phases to reduce burden at the end
15-Feb-12
A Questionnaire:
Offer an objective means of collecting
information about peoples knowledge, beliefs,
attitudes, and behaviour
Can be used as the sole research instrument
(such as in a cross sectional survey) or within
clinical trials
If inappropriately developed, will lead to poor
quality data, misleading conclusions, and
woolly recommendations
15-Feb-12
Before Settling on a Questionnaire:

Consider: is a questionnaire the right tool?
Map the area for logistical problems, sampling
issues, range of possible responses
Seek information available in the literature
If such information are not available, conduct a
focus group or pilot study
15-Feb-12
When can an Existing Questionnaire

be Used?
If the questionnaire address the same objective
If the questionnaire is validated in the same
cultural context
First and best option!
BUT BE CAREFUL:
Importing an existing questionnaire
Translating an existing questionnaire
Examples: Quality of Life Questionnaires
15-Feb-12
Why develop your own questionnaire

if one is available?
Boynton P, Greenhalg T. Selecting, designing, and developing your questionnaire. BMJ 2004, 328: 1312-1315
15-Feb-12
15-Feb-12
Steps in Developing a Questionnaire:

1. The Background
A thorough understanding of the problem through
literature search
Establish clear objectives, purpose, main question
Conduct a scoping exercise to ensure that you clarify
goals and if necessary reach an agreed compromise
This will also flag up potential practical problems (e.g.
how long the questionnaire will be and how it might be
administered)
Decide the process used to select the respondents
Seek statistical advice if generalizability of results is
required, differences measured, for comparing results
15-Feb-12

The Background
Consider all people who would use the questionnaire
Participants: general public? Consider literacy
Interviewers, coders, data entry staff,
program writers, researchers who analyze
Tasks might be shared, or several might be done by one
individual (list tasks as reminder)
Respondents: VIP group
Why: cannot be trained, and they give their
time and effort free of charge
15-Feb-12

2: Questionnaire Conceptualization
Decide: what will the questionnaire measure (knowledge,
attitudes, perceptions, opinions)
Generate statements/questions for the questionnaire
Start with broad statements
Follow with a list of detailed information needs
Looks like a list of tables in the final result section
Involve other parties who are within the research team
A link among the objectives of the study and their translation
into content is established
15-Feb-12
Attitudes and Practices of Pharmacy Students, Pharmacists,

and Academics on Pharmacokinetic Teaching and Application
in Clinical Practice
Aim: To assess the attitudes and practices
Hypothesis: PK is a knowledge-based
service that is poorly utilized in
practice by pharmacists
Identify Populations and
questions :
Students
of undergraduate pharmacy students,

practicing pharmacists, and pharmacy
academics on the on PK teaching and
learning as well as application in clinical
practice
Attitude:
importance, relevance, etc
Existing questionnaire on
Barriers,
challenges
(exploration)
students attitudes
towards
an
undergraduate courses in
pharmacy?
Pharmacists
Faculty/Teachers
15-Feb-12
Attitude: importance, relevance

Need of validation exercise?
Practice: application, satisfaction
Barriers, challenges (exploration)
Practices: teaching strategies, satisfaction

Attitudes: satisfaction with application,
need for change
Questionnaire-OFID
2012-NKheir

3: Format and Data Analysis
Turning questions into a questionnaire
The focus is on writing statements/questions
Selection of appropriate scales of measurement
Questionnaire layout
Question ordering
Number questions
Add cover note or introduction, routing instructions, codes,
page numbers
In online survey, most done automatically (to the scoring and
producing results)
15-Feb-12
Poorly Designed questions

I frequently engage in exercise
:strongly agree through to strongly
disagree.
frequently implies frequency, so a
frequency based rating scale (with
options such as at least once a day,
twice a week, and so on) would be
more appropriate.
I Regularly, engage in exercise
One person can regularly engage in
exercise once a month whereas
another person can regularly do so
four times a week
Other words to avoid in question
stems include: commonly, usually,
many, some, and hardly ever
15-Feb-12

4: Establishing Validity
Development of a valid and reliable questionnaire is
a must to reduce measurement error
Measurement error is the "discrepancy between
respondents' attributes and their survey responses "
This requires pre-testing the questionnaire and

correlating its results with other variables
Poorly constructed questions and wrong response
scaling lead to poor validity
15-Feb-12

Validity is the amount of systematic or built-in error in
measurement
The following questions are addressed in Step 4:
1. Is the questionnaire valid?
2. Does it represent the content?
3. Is it appropriate for the sample/population?
4. Is the questionnaire comprehensive enough to collect all the
information needed to address the purpose and goals of the
study?
5. Does the instrument look like a questionnaire?
15-Feb-12

Desk Evaluations
-Looking for potential problems
-The designer before user evaluation
-Simplify, but careful
-Conduct at each stage, after a
change
-Done by more than 1 person
Content analysis , Social and Face

Validity
15-Feb-12
User Evaluations
The designer knows too much about the
questions
-Other users also need to try the
questionnaire
-Self-Admin: Observing respondents fill
a questionnaire
-Also: check completed questionnaire
and ask about errors, missing data,
changes required
-3rd person observes and take notes
-Follow-up questions to check
understanding and difficulties
Pitfalls with Questions

Also see: http://www.socialresearchmethods.net/kb/quesword.php
1. Hard to understand, badly written

2. Overly educated:
Question: How often do you engage in exercise or use other nonpharmacological remedies?
Place your comments here:
15-Feb-12

3. Abbreviations that respondents may not be aware of:

Q. What
is your ID No.
Q. Which type of diabetes do you have:
T2DM T1DM
4. Use of undefined terms:
Q. What
was your income last year?
Last year may mean 12 months ending today, financial year,

calendar year.
Income: poorly defined, before or after tax? etc
15-Feb-12

5. Questions that do not fit the objectives:

Irrelevance, undue effort and burden
6. Ambiguity: what exactly do you mean?
Q. Was your performance better in your ability to
do daily exercise or your sleep quality this year?
7. Questions that are too long:
Includes questions with too many options that do
not suit phone interviews or can be burdensome to
respondents
15-Feb-12

8. Double or triple questions:

Q. During the past 2 weeks, did you have to see your
physician or another healthcare professional?
Yes No
Q. Do you agree that insulin injections should be taken
before food and stored in a fridge to avoid harmful effect
of temperature?
Yes No
15-Feb-12

9. Unreasonable recall period:

Q. In the last 12 months, how many times did you
feel numbness in your left arm?
10.Double negative:
Q. Do you agree or disagree that pharmacy
graduates from the College of Pharmacy in Qatar
University should not be asked to register in order
that they practice Pharmacy in Qatar?
15-Feb-12

11.Questions that are based on an assumption:

Q. What is the your spouses qualification:
No education
Primary school
Secondary school
College
University graduate
Post-graduate qualification
Assumption: Respondent has a spouse
Adding a Not Applicable is not enough (might not be read)
Problematic: if no spouse and next questions about spouse
A respondent might tick (No education) as an option when no spouse!
15-Feb-12

12. Social Desirability and other bias (leading,

loaded, unbalanced):
Q. In a week, how often do you miss taking your
pills?
Q. Physicians in this hospital are:

Very caring Fairly caring Not very caring
15-Feb-12

13. Responses that are not mutually exclusive:

Q. Which of these over-the-counter classes of
medicine have you sold in the last two weeks from
your pharmacy:
15-Feb-12
Non-steroid anti-inflammatory Drugs

Analgesics
Laxatives
Anti-hypertensives
Drugs acting on the cardiovsacular system

14. Response options that are not exhaustive

Q. To which of the following do you most refer your
patients for further investigation:
Hamad General Hospital
Primary health care clinic
Private health clinic
There might be other referral sites. How to

resolve this?
15-Feb-12

1. Crowded, untidy (in typed questionnaire)
2. Hard to read (in typed questionnaire)
3. Inconsistent use of terms (medicine and
drug)
4. Inappropriate titles: can be confusing,
respondents may skip sections (avoid)
5. Routing instructions that are hard to follow
15-Feb-12

6. Inconsistent lay out: Yes before No then
changes
7. Lay out that is likely to cause mistakes:
No Yes Dont know
Formatting issues resolved in online surveys.
Try: http://www.surveymonkey.com/Default.aspx
15-Feb-12
Survey Errors
Sampling Error
Non Sampling Error
-Resulting from collecting information from

a fraction of the population
-These are hard to measure

-May be larger than sampling errors can result
in bias
Extent of error depends on:

1. Sample size
Examples:
1. Poor definition of target population
2. Including or excluding groups
3. Selecting a frame that does not match the
target population
4. Poor design that does not give every
respondent the same chance of being
selected
5. Non-responses*
6. Poor definition of objectives
7. Time frame: seasonality factors
8. Quality of data collection
9. Inadequate interviewer training
2. Variability of the characteristic :

greater variation= greater error
3. Sample design: use of information
known about the population reduces
sampling error
15-Feb-12
A Checklist of Considerations
From: http://www.socialresearchmethods.net/kb/quesplac.php
Some important issues to be considered in

designing a questionnaire:
Start with easy, nonthreatening questions

Put more difficult, threatening questions near end
For historical demographics, follow chronological order
Ask about one topic at a time
When switching topics, use a transition
Reduce response set (the tendency of respondent to just keep checking the
same response)
For filter or contingency questions, make a flowchart
15-Feb-12
Ethics of Questionnaire Survey

From: http://www.socialresearchmethods.net/kb/quesplac.php
Thank the respondent at the beginning for allowing you to

conduct your study
Keep your survey as short as possible -- only include what is
absolutely necessary
Be sensitive to the needs of the respondent
Be alert for any sign that the respondent is uncomfortable
Thank the respondent at the end for participating
Assure the respondent that you will send a copy of the final
results (if s/he desires, and ensure to fulfill your promise )
15-Feb-12
Finally
An exercise
Find the pitfalls (mistakes, errors, weaknesses)
in online survey:
https://www.surveymonkey.com/s/5JCKYXX
15-Feb-12
Questions?
15-Feb-12

3rd OFID-Questionnaire Develop-NK 2012-FINAL PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

3rd OFID-Questionnaire Develop-NK 2012-FINAL PDF

Uploaded by

Copyright:

Available Formats

A Tool Kit for Questionnaire

The great popularity with questionnaires

A study is unethical if it is scientifically

Can be a questionnaire OR an interview

Purposes of Surveys: Description

Purposes of Surveys: Explanation

Purposes of Surveys: Exploration

All Utilize a Questionnaire or questions

Mail survey questionnaires

Adv: inexpensive; can send the exact instrument to a wide

Adv: convenience; high response rate; low errors

Household drop-off survey

Adv: the respondent can fill in private, when it's convenient;

Multiple evaluations during the developmental

Before Settling on a Questionnaire:

When can an Existing Questionnaire

Why develop your own questionnaire

Steps in Developing a Questionnaire:

Steps in Developing a Questionnaire:

Steps in Developing a Questionnaire:

Attitudes and Practices of Pharmacy Students, Pharmacists,

of undergraduate pharmacy students,

Attitude: importance, relevance

Practices: teaching strategies, satisfaction

Steps in Developing a Questionnaire:

Poorly Designed questions

Steps in Developing a Questionnaire:

This requires pre-testing the questionnaire and

Steps in Developing a Questionnaire:

Steps in Developing a Questionnaire:

Content analysis , Social and Face

Pitfalls with Questions

1. Hard to understand, badly written

Pitfalls with Questions

3. Abbreviations that respondents may not be aware of:

was your income last year?

Last year may mean 12 months ending today, financial year,

Pitfalls with Questions

5. Questions that do not fit the objectives:

Pitfalls with Questions

8. Double or triple questions:

Pitfalls with Questions

9. Unreasonable recall period:

Pitfalls with Questions

11.Questions that are based on an assumption:

Pitfalls with Questions

12. Social Desirability and other bias (leading,

Q. Physicians in this hospital are:

Pitfalls with Questions

13. Responses that are not mutually exclusive:

Non-steroid anti-inflammatory Drugs

Pitfalls with Questions

14. Response options that are not exhaustive

There might be other referral sites. How to

Pitfalls with Questions

Pitfalls with Questions

Non Sampling Error

-Resulting from collecting information from

-These are hard to measure

Extent of error depends on:

2. Variability of the characteristic :

Some important issues to be considered in