INTRODUCTION TO QUESTIONNAIRE DESIGN

OBJECTIVES

To understand:
Objectives of questionnaires Advantages and disadvantages Design of questionnaires Type of questions used Common problems and pitfalls

WHAT IS A QUESTIONNAIRE?

An instrument (form) to

collect answers to questions collect factual data

gathers information or measures

A series of written questions/items in a fixed, rational order

WHY USING A QUESTIONNAIRE?

A well designed questionnaire:

Gives accurate and relevant information to your research question Minimizes potential sources of bias Will more likely be completed

As simple and focused as possible

Statistical Terms
Population: Discrete group whose members can be defined by a set of at least one shared characteristic

Sample: A selection of that target population (can also be a subsample)

Random Sample: Sample in which every member of a given population has an equal chance of being selected Margin of Sampling Error (MOSE): Every random sample has a margin of error (statistical price you pay for not interviewing everyone)

Response Rate: Rate at which people agreed to participate in survey. If those who refuse are different from those who agree, results may be skewed (higher response rate = less chance for skew)

ADVANTAGES OF QUESTIONNAIRES

Can

reach a large number of people relatively easily and economically quantifiable answers easy to analyse

Provide

Relatively

DISADVANTAGES OF QUESTIONNAIRES

Provides

only limited insight into problem

Limited response allowed by questions Maybe not the right questions are asked

Varying

response

Misunderstanding/misinterpretation

Need

to get it right first time

Hard to chase after missing data

TYPES OF QUESTIONNAIRE
to face Telephone Interviewer -administrated
By
Face

mail E-mail/Internet Self-administrated

SELF-ADMINISTERED QUESTIONNAIRE

Advantages:
Cheap and easy to administer Preserves confidentiality Completed at respondent's convenience No influence by interviewer

SELF-ADMINISTERED QUESTIONNAIRE

Disadvantages:
Low response rate Questions can be misunderstood No control by interviewer Time and resouces loss

INTERVIEW-ADMINISTERED QUESTIONNAIRE

Advantages:
Participation by illiterate people Clarification of ambiguity Quick answers

INTERVIEW-ADMINISTERED QUESTIONNAIRE

Disadvantages:
Interviewer bias Needs more resources Only short questionnaires possible

Especially on telephone

Difficult for sensitive issues

Basic Survey Process

Steps:
1. Deciding on a mechanism 2. Drawing a sample

3. Designing the questionnaire

4. Fielding a survey and collecting the data

5. Analysis
6. Reporting

THE QUESTIONNAIRE DESIGN PROCESS

Convert the Research Objectives into the Information Needed Method of Administering the Questionnaire

Content of the Questions

Motivating the Respondent to Answer

Determining Type of Questions

Question Design Criteria

Determine the Questionnaire Structure

Physical presentation of the Questionnaire

Pilot Testing the Questionnaire

Administering the Questionnaire

CONVERTING THE RESEARCH OBJECTIVES INTO

INFORMATION AREAS

ADMINISTRATION AND DESIGN IMPLICATIONS

MODE OF ADMINISTRATION: SCHEDULE

Now I am going to give you a set of cards. Each card will have the name of one television serial (Hand over the cards to the respondent in a random order). I want you to examine them carefully (give her some time to read all the names). I would request you to hand over the card which has the name of the serial you like to watch the most. (Record the serial and keep this card with you). Now of the remaining nine serials name your most favorite serial (continue the same process till the person is left with the last card)

1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

T.V. SERIAL 1 2 3 4 5 6 7 8 9 10

RANK ORDER ___________________ ___________________ ___________________ ___________________ ___________________ ___________________ ___________________ ___________________ ___________________ ___________________

MODE OF ADMINISTRATION: TELEPHONE

Please listen very carefully; I am going to slowly read the name of ten popular T.V. serials. I want to know how much you prefer watching them. You need to use a 1 to 10 scale, where 1 means I do not like watching it and 10 means I really like watching it. For those in between you may choose any number between 1 and 10. However, please remember that the higher the number the more you like watching it. Now, I am going to name the serials one by one. In case the name is not clear I will repeat the list again. So, the serials name is-------------------. Please use a number between 1 and 10 as I had told you. O.k. thank you, the next name is---------------------. And so on till all the 10 names have been read out and evaluated. SERIAL 1. Balika Badhu 2. Sathiya 3. Sasural Genda Phool 4. Bidai 5. Pathshala 6. Bandini 7. Laptaganj 8. Sajan Ghar jaaana Hai 9. Tere liye 10. Uttaran

1 1 1 1 1 1 1 1 1 1

2 2 2 2 2 2 2 2 2 2

3 3 3 3 3 3 3 3 3 3

4 4 4 4 4 4 4 4 4 4

5 5 5 5 5 5 5 5 5 5

6 6 6 6 6 6 6 6 6 6

7 7 7 7 7 7 7 7 7 7

8 8 8 8 8 8 8 8 8 8

9 9 9 9 9 9 9 9 9 9

10 10 10 10 10 10 10 10 10 10

MODE OF ADMINISTRATION: MAIL

In the next question you will find the names of ten popular Hindi serials that are being aired on television these days. You are requested to rank them in order of your preference to watch these programmes. Start by identifying the serial which is your most favorite, to this you may give a rank of 1. Then from the rest of the nine, pick the second most preferred serials and give it a rank number of 2.Please carry out this process till you have ranked all 10. The one you prefer the least should have a score of 10. You are also requested not to give two serials the same rank. The basis on which you decide to rank the serials is entirely dependent upon you. Once again you are asked to rank all the 10 serials. SERIAL Balika Badhu Sathiya Sasural Genda Phool Bidai Pathshala Bandini Laptaganj Sajan Ghar Jaaana Hai Tere Liye Uttaran RANK ORDER ___________________ ___________________ ___________________ ___________________ ___________________ ___________________ ___________________ ___________________ ___________________ ___________________

1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

STAGES IN DESIGNING A QUESTIONNAIRE (1)

Planning the study:
Decide on goals Identify risk factors Know

for getting bitten by dogs

the subject a hypothesis

Literature, experts on dog bites Postmen more likely to get bitten by dogs than the normal population

Formulate

Define

information needed to test hypothesis

Occupation, owning dog, outdoor activities, attitude towards dogs

STAGES IN DESIGNING A QUESTIONNAIRE (2)

Determine study population:
Know

the respondents

Occupation Special sensitivities Education Ethnic Language

Questionnaire needs to be adapted to your population, not the opposite!

STAGES IN DESIGNING A QUESTIONNAIRE (3)

Design questions:

Content of the questions Format of the questions Presentation and layout Coding schedule (if appropriate)

Pilot and refine questionnaire

WHAT MAKES A WELL DESIGNED QUESTIONNAIRE?

Good appearance

easy on the eye

Short and simple

Relevant and logical High response rate Easy data summarisation and analysis

QUESTION ORDER

Decide on order of items/questions

Easy difficult General particular Factual abstract

Where to place sensitive questions? Be aware of ordering effects!

QUESTION ORDER
Questions should be ordered so as to seem logical to the respondent First questions should be relevant and easy Questions are effectively ordered from most salient to least salient Demographic questions should not be covered at the beginning Potentially objectionable questions are placed near the end

QUESTION ORDER
Group

questions by topic/ response options

questions

Starting

Simple With closed format Relevant to main subject Non-offending Neither demographic nor personal questions

Dont

put most important item last

QUESTIONNAIRE INTRODUCTION

Survey/interview introduction

Who you are/ what is your topic Why you are investigating Where you obtained the respondents name How and where you can be contacted Guarantee of confidentiality Length of interview (be honest)

Usefulness of study should be clear to all respondents

CONTENT OF QUESTIONS

Clear focus on research question

Avoid sidetracking
Avoid unnecessary information

Demographic information

Contact information (if non-anonymised)

DO:

Use simple wording Be brief Be specific

DO NOT:
Be vague Be condescending or talk down to respondent Use biased wording Use abbreviations or scientific jargon Use objectionable questions Be redundant

FORMAT OF QUESTIONS
Adjust to responding audience Professionals vs. public Keep sentences simple and short Define key words Remember option dont know

FORMAT OF QUESTIONS

Ask for one information at a time

Do you own a dog or have frequent contacts with dogs?

Yes No

Use mutually exclusive and exhaustive answer options Vertical order of answer options

BE ACCURATE
Do you often touch dogs?

Yes No

vs.
How often did you touch a dog during the past 3 months?

Once Twice Three times or more Not at all Dont know

BE APPROPRIATE
Are you a drunk? Yes No

vs. How often have you consumed alcoholic beverages during the past 6 months?
Daily 2-6 times/week Once a week Less than once a week Dont know

BE OBJECTIVE
Did you drink the strange brownish drink in Prague?
Yes No

Vs.
Which beverage did you consume?
Water Beer Wine Karkad None of them Dont know

BE SIMPLE
Did you smoke not less than a mean amount of 7 cigarettes/2 days from 1999 onwards?
Yes No

vs.
Did you smoke an average of 2 pack of cigarettes/week for the last 5 years?
Yes No Dont know

BIAS
Bias = systematic differences in the measurement of a response

INFORMATION BIAS

Recall bias

Cases more likely to remember than controls

Observer bias
Different interviewer different interpretations Different interpretation of similar questions Reduce by structured questionnaire

NON-RESPONSE BIAS

Those who respond are different from those who do not

Telephone interviews: more females, elderly

Reduce
Ensure high response rate Random choice of interview partners Correct during analysis (eg age, sex)

FORMAT OF QUESTIONS
Two main question formats
Closed

format forced choice

Always Sometimes Never

Yes No Dont know

Open

format free text

What is your most distressing symptom? Please describe: ________________________________________________ ________________________________________________

OPEN OR CLOSED?
Closed Advantages:

Simple and quick Reduces discrimination against less literate Easy to code, record, analyze Easy to compare Easy to report results

CLOSED QUESTIONS

Disadvantages:

Restricted number of possible answers Loss of information Insert field others

Possible compromise:

OPEN QUESTIONS

Advantages:
Not directive Allows exploration of issues to generate hypothesis

qualitative research, focus groups, trawling questionnaires

Used even if no comprehensive range of alternative choices Good for exploring knowledge and attitudes Detailed and unexpected answers possible

OPEN QUESTIONS

Disadvantages:

Interviewer bias Time-consuming Coding problems Difficult to analyze! Difficult to compare groups

CLOSED QUESTIONS
1.

Straightforward response

What is your age in years?

How long have you owned a dog? What is your sex (gender)?
Male Female

___ years
___ years

Did you stay in Hotel X on 23/7/05?

Yes No Dont know

CLOSED QUESTIONS
2. Checklist

Which of the following outdoor activities did you do last week?

Running Walking Hiking Cycling Swimming

CLOSED QUESTIONS
3. Rating scale

Did you do use sunscreen during the following outdoor activities during the past six months? Always Sometimes Seldomly Never Running Walking Cycling

CLOSED QUESTIONS
4. Rating scale Numerical How useful would you think that information on the risk of biting from stray dogs would be?

(please circle)

1 2 Not at all useful

7 Very useful

Analogue How much is your pain severe (put the tick on the line)
0 10

CLOSED QUESTIONS
5. Scales for measuring attitude (Lickert)
Stray dogs carry a higher risk of rabies No, I strongly disagree No, I disagree quite a lot No, I disagree just a little Im not sure about this Yes, I agree just a little Yes, I agree quite a lot Yes, I strongly agree

PROBLEMS AND PITFALLS

Avoid questions that ask two things at once - you wont know which bit people are answering: Have you ever had stomach ache and diarrhoea? Ambiguity..... Do you go to the woods a lot?

PROBLEMS AND PITFALLS

Avoid

jargon/abbreviations/slang

How often do you get up at night to PU? (pass urine) Should IVDUs be treated in the community?
Avoid

not mutually exclusive options

What age are you? 16-20 20-25 25-30 35-40

PROBLEMS AND PITFALLS

Avoid leading questions

Do you think that the food in the hotel made you sick? Did the hotel staff seem unhygenic to you? Do you agree that the hospital staff were close to exhaustion?

Avoid making questionnaire too long Typographical / spelling errors

PILOTING AND EVALUATION

Pilot with a similar group of people to your intended subjects Highlights problems before starting

Effects of alternative wording Overall impression on respondents and interviewers Final polishing after several amendments

PRESENTATION AND LAYOUT

Clear consistent layout

Adequate space to answer

Large

font size

Appropriate page breaks Avoid

experimental layouts fancy logos printed on recycled paper/is an equal opportunity employer etc

PRESENTATION AND LAYOUT

Using colour or printing questionnaire on coloured paper may help Use filter questions, if necessary Give clear instructions about how to answer the questions

WAYS TO INCREASE PERCEIVED REWARD

Show positive regard Say thank you Ask them for advice Give social validation Give a tangible reward Make the questionnaire interesting State an upcoming deadline

WAYS TO DECREASE PERCEIVED COST

Avoid subordinating language Do not embarrass the respondent Do not inconvenience the respondent Make questions appear short and easy Avoid asking personal information (if it must be asked, use soft wording and inform about confidentiality) Keep subsequent requests similar

CREATE A NAVIGATIONAL PATH

Instructions should be placed right where they are needed Matrices are confusing: it is best to order questions from top to bottom, and left to right

Use larger font to attract attention

Use color shading to attract attention or show groupings Use spacing and similarities to show groupings

WAYS TO ESTABLISH TRUST

Provide a token of appreciation Provide a sense of legitimate authority Make completing the questionnaire seem important Remind respondent of previous relationship with sponsor (if applicable)

SUMMARY
A well designed questionnaire: Will give appropriate data which allow to answer your research question Will minimise potential sources of bias, thus increasing the validity of the questionnaire Will much more likely be completed

FINALLY, KEEP YOUR QUESTIONNAIRE SHORT AND THE QUESTIONS SIMPLE, FOCUSED AND APPROPRIATE