Questionnaire & Form Design

Questionnaire Design looks easy.but..

We all think we are experts at asking questions Mistakes often do not lead to obvious errors
Interviewers and responders do their best

Consequences of mistakes are not immediate Disparate interests involved

Client, interviewer, respondent, data entry persons

About 26 steps involved!

Mistakes could occur in any one of these
Questionnaires are part of a lengthy communication process

Dont shoot me, Im just the statistical analyst

Extreme phrasing of question or combination of questions is the most frequent source of distortion in survey results So lets see how we can design good questionnaires

Questionnaire Definition
A questionnaire is a formalized set of questions for obtaining information from respondents.

Types of Questions

Choosing Question Structure Unstructured Questions

Unstructured questions are open-ended questions that respondents answer in their own words.
How do you feel? __________________________________

Choosing Question Structure Structured Questions

Structured questions specify the set of response alternatives and the response format. A structured question may be multiple-choice, dichotomous, or a scale.

Choosing Question Structure Dichotomous Questions

A dichotomous question has only two response alternatives: yes or no, agree or disagree, and so on. Often, the two alternatives of interest are supplemented by a neutral alternative, such as no opinion, don't know, both, or none.

Do you intend to buy a new car within the next six months? _____ Yes _____ No _____ Don't know

Layout Considerations
Make your questionnaire Short Simple Sweet

For interviewers
Make it short
15 minutes by phone 30 to 40 minutes in-home
use show cards and shuffle packs

Make instructions explicit

Put them in capital letters

For Respondents
Make it simple
Be respectful
Address concerns up front Thank you at the end

Make it flow
general > interesting > difficult or tedious > interesting > sensitive > classification

For data entry persons

Make it sweet (looking)
Use pre-codes Lay-out matters
Questions on left, responses to right Include a space for survey/project number

Questionnaire Design Process

Questionnaire design issues

Factual data Problem of understanding Problem of willingness to answer Problem of ability to answer

Are Several Questions Needed Instead of One?

Sometimes, several questions are needed to obtain the required information in a clear manner. Consider the question,
Do you think Coca-Cola is a tasty and refreshing soft drink? (Incorrect)

Such a question is called a double-barreled question, because two or more questions are combined into one. To obtain the required information, two distinct questions should be asked:
Do you think Coca-Cola is a tasty soft drink? and Do you think Coca-Cola is a refreshing soft drink? (Correct)

Overcoming Inability To Answer Is the Respondent Informed?

In situations where not all respondents are likely to be informed about the topic of interest, filter questions that measure familiarity and past experience should be asked before questions about the topics themselves.

A don't know option appears to reduce responses without reducing the response rate.

Overcoming Unwillingness To Answer Effort Required of the Respondents

Most respondents are unwilling to devote a lot of effort to provide information.

Overcoming Unwillingness To Answer

Please list all the departments from which you purchased merchandise on your most recent shopping trip to a department store. (Incorrect) In the list that follows, please check all the departments from which you purchased merchandise on your most recent shopping trip to a department store. 1. 2. 3. 4. . . . 16. 17. Women's dresses Men's apparel Children's apparel Cosmetics ____ ____ ____ ____

Jewelry Other (please specify)

____ ____

(Correct)

Choosing Question Wording Define the Issue

Define the issue in terms of who, what, when, where, why, how? Who, what, when, and where are particularly important. Which brand of shampoo do you use? (Incorrect)

Which brand or brands of shampoo have you personally used at home during the last month? In case of more than one brand, please list all the brands that apply. (Correct)

Choosing Question Wording Use Unambiguous Words

In a typical month, how often do you shop in department stores? _____ Never _____ Occasionally _____ Sometimes _____ Often _____ Regularly (Incorrect)

In a typical month, how often do you shop in department stores? _____ Less than once _____ 1 or 2 times _____ 3 or 4 times _____ More than 4 times (Correct)

Choosing Question Wording Avoid Leading or Biasing Questions

A leading question is one that clues the respondent to what the answer should be, as in the following: Do you think that patriotic Americans should buy imported automobiles when that would put American labor out of work? _____ Yes _____ No _____ Don't know (Incorrect)

Do you think that Americans should buy imported automobiles? _____ Yes _____ No _____ Don't know (Correct)

Choosing Question Wording Avoid Implicit Assumptions

Questions should not be worded so that the answer is dependent upon implicit assumptions about what will happen as a consequence. 1. Are you in favor of a balanced budget? (Incorrect) Are you in favor of a balanced budget if it would result in an increase in the personal income tax? (Correct)

2.

Determining the Order of Questions

Opening Questions The opening questions should be interesting, simple, and non-threatening. Type of Information As a general guideline, basic information should be obtained first, followed by classification, and, finally, identification information.

Difficult Questions Difficult questions or questions which are sensitive, embarrassing, complex, or dull, should be placed late in the sequence.

Determining the Order of Questions

Effect on Subsequent Questions General questions should precede the specific questions (funnel approach). Q1: What considerations are important to you in selecting a department store? Q2: In selecting a department store, how important is convenience of location? (Correct)

Determining the Order of Logical Order Questions

The following guidelines should be followed for branching questions: The question being branched (the one to which the respondent is being directed) should be placed as close as possible to the question causing the branching. The branching questions should be ordered so that the respondents cannot anticipate what additional information will be required.

Some thumb-rules for Questionnaire Design

1. Dont crowd the questions
makes the questionnaire look difficult

2. Dont split questions across pages. 3. Put special instructions on the questionnaire
use a different typeface (bold, CAPITALS, or italics). put Dont read above the question. put Skip immediately after trigger answers.

4. Precode all closed questions.

Some thumb-rules for Questionnaire Design

5. Use vertical formats for closed questions
NO! What is your favorite brand of cereal?
A ___ B___ C____ Other_____

Yes! What is your favorite brand of cereal?

1.BRAND A 2...BRAND B 3...BRAND C 4...OTHER

Some thumb-rules for Questionnaire Design

6. Record the interview start and stop times.
7. Provide a beginning (a greeting) and an end to the interview (A thank you).
Hello, my name is (your name), and Im calling from Harmon Research, a marketing research company. I need to speak with the person who is most responsible for health care decisions in your household. Is that you?

Example of a Precoded Questionnaire

The American Lawyer
A Confidential Survey of Our Subscribers

(Please ignore the numbers alongside the answers. They are only to help us in data processing.) 1. Considering all the times you pick it up, about how much time, in total, do you spend reading or looking through a typical issue of THE AMERICAN LAWYER?

Less than 30 minutes....................-1

30 to 59 minutes............................-2 1 hour or more..-3

Pretesting
Pretesting refers to the testing of the questionnaire on a small sample of respondents to identify and eliminate potential problems. A questionnaire should not be used in the field survey without adequate pretesting. The pretest sample size varies from 15 to 30 respondents for each wave.

Pre-test, pre-test, pre-test!

Discuss with client, data entry persons, and respondents The career you save may be your own!

Types of errors
Random sampling errors: sample too small sample not representative inappropriate sampling method used

Research design errors: bias introduced measurement error data analysis error population definition error scaling error question construction error

Interviewer errors: recording errors cheating errors questioning errors respondent selection error Respondent errors: non-response error inability error falsification error Hypothesis errors: type I error (also called alpha error)
the study results lead to the rejection of the null hypothesis even though it is actually true

type II error (also called beta error)

the study results lead to the acceptance (non-rejection) of the null hypothesis even though it is actually false