You are on page 1of 39

Survey methodology

Cristina Giudici University La Sapienza Roma

References
Groves R.M. et al. Survey methodology, Wileyinterscience, 2004 Canada national statistical agency: http://www.statcan.gc.ca/ Eurostat methodological publications European Values Study (EVS)

INTRODUCTION : What is a survey ?


A survey is a systematic method of gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of wich the entities are members A survey usually originates when an individual or institution is confronted with an information need and the existing data are insufficient

What is survey methodology ?


Survey methodology is the study of survey methods

How survey work to produce statistics


Characteristics of the population

Inference

Characteristics of a respondent

Statistical computing

Characteristics of the sample

Inference

Respondent answers to quenstions

Chapter I The life cycle of a survey

A survey from a process perspective


Define research objectives Choose mode of collection Choose sampling frame

Construct and pretest a questionnaire Recruit and measure sample

Design and select sample

Code and edit data

Make postsurvey adjustments

Perform analysis

The life cycle of a survey from a design perspective


Target Population Construct Sampling Frame Measurement What is the survey about? Who is the survey about?

Sample

Response

Respondent Edited Response Survey statistics The Measurement dimension describes what data are to be collected about the observational units in the sample The Representational dimention concerns what population are described by the survey

Postsurvey Adjustments

The measurement dimension


Construct Constructs are the elements of information that are sought by the researcher :
How many incidents of crimes with victims there were in the last year; The consumption of beer in the last month; The degree of knowledge of mathematics of childrens

Measurements are ways to gather information about constructs : Measurement


Questions posed to a respondent (During the last 6 month, did you call the police to report something that happened to you that you thought was a crime?) NB: the critical task for maesurement is to design questions that produce answers reflecting perfectly the construct we are trying to measure.

Response

Response could be produced in a variety of means


But in general the nature of the response is determined by the nature of the measurement

Edited Response

Editing of data may examine the full distribution of answers and look for atypical patterns of responses
Edited responses are the data from wich inference is made about the values of the construct for an individual respondent

The representational dimension


Target Population The target population is the set of unit to be studied
The adult population living in households in 2009;

The frame population if the set of target population members that has chance to be selected into the survey sample : Sampling Frame
In a simple case it is a list of all units in the target population, but sometimes it is a set of units imperfectly linked to population members. i.e. a list of telephone numbers when the target population is the adult population

Sample

The sample is the group from wich measurement will be sought. In many case it is a very small fraction of the the sampling frame Respondents are the elements successfully measures.

Respondent

Non respondents is the complement

Postsurvey Adjustments

Postsurvey adjustments consist on weighting up the underrepresented groups in order to improve the survey estimate
Because of mismatches of the sampling frame and the target population (coverage problems) statistics based on the respondents can differ from caracteristics of the target population. Examination of non response patterns may suggest an underrepresentation of some groupes relative to the sampling frame

The life cycle of a survey from a quality perspective


Construct Validity Sampling Frame Measurement Measurement error Response Processing error Edited Response Respondent Adjustments error Postsurvey Adjustments Survey statistics Sample Nonresponse error Sampling error Target Population Coverage error

Coverage of a target population by a frame


Undercoverage Elements in the target population missing from the frame
i.e.:non telephone household, using a telephone frame to cover the full household population

Ineligible units Frame population

Covered population

Ineligible units Elements in the frame that are no member of the target population
i.e.:business telephone numbers, using a telephone frame to cover the full household population

Undercoverage

Target population

Evaluating survey questions:


Are the answers good measures of the intended construct?
Exemple of methods that can be used to evaluate draft survey questions
 Expert reviews
The substantive expert review the wording, the order and the structure of questions, the response alternatives etc. A small number of target population participate in a systematic discussion about the survey topic. The researcher learn about the nomenclature of the concept, the common perspective taken by the target population on key issues etc Researcher test how questions are read and answered. A behaviour coding is often used

 Focus groups

 Questionnaire pretest

Evaluating survey questions:


Exemple of behavior codes for interviewer and respondent behaviors
Interview Questioning Behavior (choose one) 1. Reads questions exacty as worded 2. Reads questions with minor changes 3. Reads questions so that meaning is altering Respondent Behaviors (check as many as apply) 1. Interrupt question reading 2. Asks for clarification of question 3. Gives adequate answer 4. Gives answer qualifies about accuracy 5. Gives answer inadequate for questions 6. Answers dont know 7. Refuses to answer

Chapter II Methods of data collection

Traditional data collection methods


Mailing paper questionnaires to respondents, who fill them out and mail them back Having interviewers call to respondents on the telephone and ask them the question in a telephone interview Sending the interviewers to the respondents home or office to administer the questions in face-to-face (FTF) interviews

Alternatives methods of data collection


OCR/ICR
Optical/intelligent caracter recognition

FAX
Computerised Self Administered Questionnaires

Mail Telephone

Disk by Mail

E-mail

Web

CATI
computer assisted telephone interviewing

TDE
Touchtone data entry

IVR
Interactive voice response

Face to face
SAQ
Self administered questionnaire

CAPI computer assisted personal interviewing


Text CASI Walkman Audio CASI Video CASI

Alternatives methods of data collection (a)


OCR/ICR
Optical/intelligent caracter recognition

FAX

Mail

Disk by Mail

E-mail

Web

Alternatives methods of data collection (b)

Telephone

CATI
computer assisted telephone interviewing

TDE
Touchtone data entry

IVR
Interactive voice response

Alternatives methods of data collection (c)

Face to face
SAQ
Self administered questionnaire

CAPI computer assisted personal interviewing


Text CASI Walkman Audio CASI Video CASI

Chapter III DESIGNING A QUESTIONNAIRE

QUESTIONS AND ANSWERS IN SURVEYS


A questionaire
is a standardised set of questions administered to the respondents in a survey Respondents are required to interpret a preestablished set of questions and to supply the information these questions seek.

The cognitive processes in answering questions


A simple model of the survey response process
Comprehension of the question Retrieval of information Judgment and estimation Reporting an answer

BUT
Respondents often take shortcuts to get through the interview more quickly

OR
they have motives that override their desire to provide accurate information Responses could be biased by acquiescence (the tendency to agree) social desirability (the tendency to present oneself in a favourable light by underreporting undesirable attributes and overreporting desirable one)

The cognitive processes in answering questions (2)


The satisficing model (Krosnik and Alwin, 1987)
Some respondents try to satisfy (to take a low road answering more superficially) whereas others try to optimise (to take an high road by careful answering questions)

Satisficing respondents do not seek to understand the question completely, just well enough to provide a reasonable answer

Problems in answering survey questions


 Failure to encode the information sought  Misinterpretation of the questions  Forgetting and other memory problems  Estimation strategies  Problems in formatting answer  More or less deliberate misreporting  Failure to follow instruction

FORMATTING THE ANSWER


Survey items can take a variety of formats; the most common are: 1) Open-ended qustions that call for numerical answers 2) Closed questions with ordered response scales 3) Closed questions with categorial response options

1 - Open-ended qustions that call for numerical answers


Now, thinking about your physical health, which includes physical illness and injury, for how many days during the past 30 was your physical health not good?
Note that: Open-ended items yield more exact information than closed items

2 - Closed questions with ordered response scales


1 2 3 4 5 Would you say that in general your health is: Excellent Very good The interviewer is instructed to Good please read the answer Fair categories, but not the number Poor attached to them!
Note that : with some type of rating respondents seem to shy away from the negative end of the scale When the scale points have numerical labels, the label can affect the answer (e.g. if respondents are asked to rate their success in life)

3 - Closed questions with categorial response options


1 2 3 4 5 6 Are you: Married Divorced Widowed Separated Never married A member of an unmarried couple

Note that : The respondent may not wait to hear or read all the option; they may select the firs reasonable answer they consider (primacy effect) The opposite coul happen: the last option the interviewer read may be the first one that respondent think about (recency effect)

GUIDELINES FOR WRITING GOOD QUESTIONS (Sudman and Bradburn) Non sensitive questions about behavior
The key problem with many questions about behavios is that respondents may forget some or all of the relevant information, or that their answer may reflect inaccurate estimate In order to reduce memory problems it is essential to play attention to the wording of the question and to provide memory help

Attitude questions
Attitude questions are a very commen class of survey questions. The most frequent problems deals with the wording of questions, the question order and the format of response scales

Non sensitive questions about behavior Play attention to the wording


With closed questions, include all reasonable possibilities as explicit response options Are you: Married Divorced Widowed Separated Never married Are you: Married Single

Non sensitive questions about behavior Play attention to the wording


Make the question as specific as possible
(about who it covers, what time period, which behaviours)

Over the last month, that is .. how often do you read a newspaper in a tipical week?

In a tipical week, how often do you read a newspaper?

Non sensitive questions about behavior Play attention to the wording


Use words that virtually all respondents will understand Have you ever had a heart attack?

Have you ever had a miocardial infarction?

Non sensitive questions about behavior Provide memory help


Uses aided recall (or ask separate questions about subcategories)
Please look cerefully at the following list of volountary organisations: which, if any, do you belong to? A B C D Religious organisations Cultural organisations Political groups Other

To which volountary organisation do you belong to?

Attitude questions Play attention to the wording


Clearly specify the attitude object of interest

Do you think the Government is spending too litte, about the right amount, or too much on higher education?

Do you think the Government is spending too litte, about the right amount, or too much on education?

Attitude questions* play attention to the wording


Measure the strength of the attitute
using a response scale, a separate item or multiple items that can be combined into a scale

Do you agree or disagree with the following statement? Government is spending too little on education 1 Agree strongly 2 Agree 3 Neither agree nor disagree 4 Disagree 5 Disagree strongly
*note that an attitude have generally a direction (agree or disagree) and an intensity (strongly disagree.strongly agree)

Attitude questions reduce impact of question order


When asking general and specific questions about a topic, ask the general question first
(otherwise, the answer to the general question is likely to be affected by the number and content of specific questions)

When asking questions about about multiple items, start with the least popular
(the unpopular questions are likely to seem even less appealing when they follow more popular questions)

When asking general and specific questions about a topic, ask the general question first
Please tell me whether or not you think it should be possible for a pregnant woman to obtain a legal abortion if: the woman wants it for any reason? 1. Yes 2. No 3. Don't know Please tell me whether or not you think it should be possible for a pregnant woman to obtain a legal abortion if: there is a strong chance of a serious defect in the baby? 1. Yes 2. No 3. Don't know
USA GSS

Attitude questions play attention to the response scale


Use closed questions for measuring aptitudes;
(open answers are difficult to code)

Use five-to seven-point response scales and label every scale point
(verbal label ensure that interpret the scale in the same way)

Now is up to you!

You might also like