Survey Errors: Problems and Opportunities

Dr. Jagrook Dawra

Measurement of any quantity is subject to error. Surveys are no exception. Knowledge of these errors
and evolving a strategy to handle these errors are therefore very important to have reliable and valid
results.
The figure below summarizes the errors that a researcher should be aware of in his/her research.

Total survey
error

Sampling error

Non-Sampling
Error

Non response
error

Measurement
Error

Systematic

Random

Sampling Error
In statistics, sampling error or estimation error is the error caused by observing a sample instead of the
whole population.
A good test of whether sampling error is present in my study would be to see if the results obtained
randomly over different samples of the same size are consistent.
So the next time you read the Times of India Poll that says:
Do you think India should resume sporting ties with Pakistan?

Yes 54%
No 41%
1

Undecided 5%

You should ask yourself: would the results be the same if it was a poll conducted by Deccan Chronicle?
By how much would the results vary from paper to paper?
The answer to the second question (in % terms) is an estimate of sampling error.
Question to ponder upon: How can sampling error can be reduced?

Non-sampling error
Non-sampling error is a catch-all term for the deviations from the true value that are not a function of
the sample chosen, including various systematic errors and any random errors that are not due to
sampling. Non-sampling errors are much harder to quantify than sampling errors.
In the above example a non sampling error would be because of the question itself. An incorrectly
framed question or incorrectly framed choices can result in a error that could be more serious than the
sampling bias.
Non-sampling errors are of two types: Non response errors and measurement errors.
Non-response error reduces the size of the sample available for analysis and constitutes a threat to the
representativeness of the final sample. Since sampling error varies inversely with the sample size, any
loss of sample size due to non-response will increase the magnitude of random sampling error realized
relative to the level anticipated if no allowance for non-response is made in planning the study.
More seriously, non-response is likely to cause a systematic bias or error. A telephone company, for
example asked in its survey: What is the monthly average bill. A sample of 352 respondents were
randomly selected using a random telephone number generator.58% people responded to this question.
The mean bill reported was Rs. 2136.00 per month. The company was able to establish from its record
the actual monthly average bill for these respondents as Rs. 2411.00. The true mean was therefore
different from the sample mean.
The problem is aggravated when the sampling process is not truly random, but convenience, as is in
most cases.
To minimize this bias, the researcher should:

Minimize the burden on the respondent by designing questionnaires that are attractive in
appearance, interesting and easy to complete; printing mail questionnaires in booklet format;
placing personal questions at the end; creating a vertical flow of questions; and creating sections
of questions based on their content.
Personalize all communication with the respondent by printing letters and envelopes
individually, using blue ball point pens for signatures and a first class stamp on outgoing and
return envelopes; and constructing a persuasive letter.

Provide information about the survey in a cover letter to respondents, interviewers, and clerical
personnel. If possible, also send out letters in advance informing respondents that a survey is
forthcoming.

Measurement Error
When responding to the question asked by the telephone company, several respondents rounded-off
the figures. The rounding off was both ways over-reporting and under-reporting of the actual figures,
leading to a bias.
The TOI researcher wanted to find out whether the Pakistani players should be allowed to play in IPL
matches in India, but by asking the question above, there is a likelihood of a measurement error
occurring.
An unclear objective, ambiguous question, inadequate response options, incorrect question format, etc.
lead to a measurement error.
The distinction between systematic and random measurement errors are closely related to two very
important properties of measurement instruments reliability and validity. Reliability is concerned with
random measurement error and relates to questions about how stable, consistent or reproducible are
the responses obtained from some measurement instrument. Validity refers to questions about
whether the instrument or procedure measures what it intends to measure, and if so, how adequately
does it do so?
More on reliability and validity is given in a separate note.