Research Methods in

Psychology

UAPG1004
Descriptive Methods:

Survey
Research

Outline:
I.
II.
III.
IV.
V.

OVERVIEW
USES OF SURVEYS
CHARACTERISTICS OF SURVEYS
SAMPLING IN SURVEY RESEARCH
SURVEY METHODS
I.
Mail survey
II. Personal interviews
III. Telephone interview
IV. Internet surveys
VI. SURVEY-RESEARCH DESIGNS
I.
II.
III.

Cross-sectional design
Successive-independent sample design
Longitudinal design

VII. QUESTIONNAIRES
VIII. THINKING CRITICALLY ABOUT SURVEY RESEARCH

Survey Research
Results of surveys are used to describe peoples
opinions, attitudes, and preferences.
Survey results are used to make predictions about
peoples behavior.
In typical survey research, a sample of people
completes a questionnaire(s); responses from the
sample are used to describe the population.
The survey involves using a predetermined set of
questions.

Survey Research

(continued)

Surveys can be specific and limited in their

scope
for example, views about a specific TV program
Other example?

or more global in their goals.

for example, about TV habits in general.
Other examples?

Are surveys always biased?

Best way to determine whether a survey is biased
is to examine the survey procedures and analyses.
Dont assume that because a survey is sponsored
by an organization with a vested interest, that the
survey will be necessarily biased.

Correlational Research
Correlational research: Assess relationships
among naturally occurring variables.
For example: Attitudes, preferences, intelligence,
personality traits, feelings, age, sex

Researchers calculate correlation coefficients to

determine the strength and direction of a
predictive relationship between two variables.
-1.00 to 0 to +1.00
(negative)
(positive)

Correlational Research

(continued)

Do these relationships represent

positive or negative correlations?
As the number of years in which individuals smoke
cigarettes increases, the likelihood of lung cancer
increases.
Positive correlation

As the frequency of participating in volunteer activities

increases, occasions of depressed mood decrease.
Negative correlation

As arousal level increases, the likelihood of retaliation

following an offense increases.
Positive correlation

Other example?
Positive or negative correlation?

Obtaining a Sample
Researchers are not interested simply in the
responses of those surveyed
they seek to describe the larger population
from which the sample was drawn.
Careful selection of a survey sample allows
researchers to generalize the findings from the
sample to the population.

Basic Terms of Sampling

Population: set of all cases of interest. For
example:

current students of UTAR

current residents of Kampar
citizens of Perak
citizens in Malaysia

Sampling Frame: list of the members of a

population.
For example, registrars list of all currently registered
UTAR students

Sample: subset of the population used to

represent the population.
Students in your class as a sample of current students
at your institution (or your city, Malaysia, North
America)

Element: each member of the population.

Obtaining a Sample

(continued)

Goal: Sample should represent the population.

Characteristics of participants in the sample should
be similar to those of the entire population.
Example: Which sample represents a population that
is 30% freshman, 30% sophomore, 20% junior, 20%
senior?
Sample A Sample B
30 freshmen, 30 sophomores,
60 freshmen, 60 sophomores,
20 juniors, 20 seniors
40 juniors, 40 seniors
Both! But note: The samples are representative on one feature only!

Obtaining a Sample

(continued)

A biased sample occurs when the characteristics

of the sample differ systematically from those of
the target population.
A sample may under-represent a segment of the
population, or
over-represent a segment of the population.
For example, most samples in psychology research
overrepresent college students and underrepresent
individuals who are not in college.
Most research underrepresents individuals from diverse
cultures.

Obtaining a Sample

(continued)

Two sources of biased samples:

Selection bias occurs when the

researchers procedures for
selecting a sample result in one
or more segments of the
population being under- or overrepresented.
Example: Researcher places sign-up sheets for a
research study in a Psychology Department. Psychology
students are likely to be over-represented because of
the selection procedure.

Obtaining a Sample

(continued)

Response bias occurs when

individuals selected for the initial
sample do not complete and return
the survey.
Example: People who receive the survey
arent interested, theyre worried about
privacy, have vision or other problems,
dont have time, etc.
Final sample will only represent the
population of people who are interested,
not worried, have good vision, time, etc.

Approaches to Sampling
Sampling refers to the procedures used to obtain
a sample.
Two basic approaches to sampling are

nonprobability sampling, and

probability sampling.

Approaches to Sampling
(continued)
Nonprobability sampling: No guarantee that each
member of the population has an equal chance of
being included in the sample.
Convenience sampling occurs when the researcher
selects individuals who are available and willing to
respond to the survey.

Examples: Magazine surveys,

Internet surveys
Lots of psychological research uses convenience
samples (but this can be OK).

Approaches to Sampling
(continued)
Probability sampling: All members of a population
have an equal chance of being selected for the
survey (this is called a simple random sample).

Need to have a sampling frame

(list) of people in the population,
or
use random-digit dialing (but not
all members of the population
may be included).

Approaches to Sampling
(continued)
Stratified Random Sample: The population is
divided into subpopulations called strata.
Random samples are then drawn from the strata.

For example, strata from a university

population potentially include freshmen,
sophomores, juniors, seniors, graduate
students, faculty, staff, administrators.
Stratified random sampling increases the
likelihood that the sample will represent the
population.

Survey Methods
Four methods for obtaining survey data are:

mail surveys,
personal interviews,
telephone interviews, and
Internet surveys.

Each method has advantages and disadvantages.

Researchers choose depending on the nature of
their research question.

Survey Methods

(continued)

Mail surveys
quick and convenient, self-administered, best for highly
personal or embarrassing topics.
may have the problem of response bias when people
selected for the survey sample dont complete and return
their survey.
Due to response bias, the final sample may not be
representative of the population.
Because mailed surveys are self-administered,
respondents are free to interpret questions as they see
fit, leading to possible differences in how people respond
to questions.

Survey Methods

(continued)

Personal Interviews
are costly, but researchers gain more control over how
the survey is administered, and how people interpret
survey questions.
Interviewers can seek clarification of answers.
potential problem: interviewer bias.
Interviewer bias occurs when interviewer records only
selected portions of respondents answers, or interviewer
words questions differently to fit particular respondents.
Interviewers must be highly motivated, carefully trained,
and supervised.

Survey Methods

(continued)

Telephone Interviews
brief surveys can be completed efficiently and with
greater access to the population.
Random-digit dialing technology allows researchers to
select random samples.
Interviewers can be supervised easily from one location.
Potential problems include selection bias (only people
with phones can be included), response bias (people may
refuse solicitations to complete surveys over the phone),
and interviewer bias.

Survey Methods

(continued)

Internet Surveys
The Internet allows for efficient, low-cost means to
survey very large samples.
Samples can be very diverse and access typically
underrepresented samples.
Potential problems include selection bias (access to
computers and Internet required), response bias,
and lack of control over the research environment.

Survey Methods

(continued)

Ways to increase survey response rate (and

lessen problems associated with response bias):
Questionnaire has a personal touch (e.g.,
respondent are addressed by name and not simply
Dear student)
Responding requires a minimum of effort
Topic of survey is intrinsically interesting to
respondent
Respondent identifies with the organization or
researcher who is sponsoring the survey.

Survey Research Designs

A research design is a plan for
conducting a research project.
Its the research method the psychologist
chooses to best answer his/her research
question.

There are three types of survey research

designs:
Cross-sectional design
Successive independent samples design
Longitudinal design

The survey design researchers choose

depends on their research question.

Survey Research Designs

(continued)
Cross-sectional Survey Design

A sample is selected from one or

more populations at one time.
Researchers choose the
population(s) they would like to
describe.
They use either probability or
convenience sampling
probability sampling will lead to a
more representative sample.
Respondents complete a survey.

Survey Research Designs

(continued)

The responses are used to describe

and make predictions for the
population at that point in time.
Descriptive statistics and correlations
are used.

If two or more samples are drawn

from different populations, the
populations can be compared.
Researchers cannot assess changes
over time with cross-sectional
designs.

Survey Research Designs

(continued)
Successive Independent Samples Design

A series of cross-sectional
surveys over time.
A different sample of people
completes the survey each time.
Each sample is selected from the
same population(s).
Responses from the sample are
used to describe the population
at each point in time.

Survey Research Designs

(continued)
Researchers can compare the survey

responses from each sample to see

how the population changes over time.
Successive independent samples designs
dont tell us whether individuals change
over time (because different individuals
complete the survey each time).

Problem of noncomparable samples:

If different populations are sampled at
each time, we dont know if responses
differ because of true changes over time,
or because different populations were
sampled.

Survey Research Designs

(continued)
Longitudinal Research Designs

The same sample of individuals

completes the survey at different
points in time.
This allows researchers to assess
how individuals change over time.
Responses from the sample of
respondents are generalized to
describe changes over time in the
population from which the sample
was drawn.

Survey Research Designs

(continued)
Problems with longitudinal designs:

Just because people change over time,

surveys cant tell us why people change.
no causal inferences with correlational data

Attrition occurs when people drop out of

the study.
the sample no longer represents the
population from which it was selected

Reactivity: Respondents may strive to be

consistent or become sensitized to the
topic.

Measures in Correlational
Research: Questionnaires
Survey researchers most frequently use
questionnaires to gather information.
Psychologists measure different types of variables:

demographic variables (e.g., age,

sex, race, socioeconomic status)
preferences and attitudes
most often these are measured with
self-report scales
participants respond on rating
scales (assume interval level of
measurement)

Reliability and Validity of SelfReport Measures

All measurement must be reliable and valid.
Reliability refers to the consistency of
measurement.
Test-retest reliability: Administer measure two times
to the same sample. Individuals scores should be
consistent over time.
A high correlation between the two scores indicates
good test-retest reliability (r >.80).
Individuals need only to maintain their relative position
in the distribution; they do not need the same score on
each administration.

Measures in Survey Research

(continued)
Note: We dont expect some measures
to produce consistent scores over time.
When people change on a particular
variable over time, we expect the
measure to have low test-retest
reliability.
Would you expect these measures to have
high or low test-retest reliability?
Depression symptoms
Personality traits
Intelligence

Measures in Survey Research

(continued)
How do we improve reliability?
More items improves reliability.
If I use only one item to measure your research methods
knowledge, the test probably wouldnt be reliable.

There should be great variability on the factor being

measured among the tested individuals.
When the sample is diverse, the measure will more reliably
discriminate high and low individuals.

Reliability improves when the testing situation is free of

distractions and instructions are clear.
Anything that decreases opportunities for errors improves
the reliability of the measure.

Measures in Survey Research

(continued)

Reliable measures make us more confident that we are

consistently measuring a concept within a sample, but
are reliable measures truthful?
I could reliably measure your research methods knowledge
by measuring your height. The taller you are, the better your
score.
Is this a truthful or accurate measure of research methods
knowledge?

Measures in Survey Research

(continued)
Validity refers to the truthfulness of a measure.
A valid measure assesses what it is intended to
measure.

Construct Validity: Does an

instrument measure the theoretical
construct (concept) it was designed
to measure?

Measures in Survey Research

(continued)
This seems like a straightforward question, but consider
widely used measures of intelligence which include items
such as:
comprehension: Why would people use a secret ballot?
vocabulary: What does dilatory mean?
similarities: How are a telephone and a radio alike?

Do these items (and others like them) assess intelligence

in a valid manner?
The construct validity of intelligence measures is a matter of
heated debate.

Measures in Survey Research

(continued)
Establishing the construct validity of a
measure depends on
convergent validity and
discriminant validity.

Convergent validity refers to the extent

to which two measures of the same
construct are correlated (go together).
Discriminant validity refers to the
extent to which two measures of
different constructs are not correlated
(do not go together).

Measures in Survey Research

(continued)
Example of Construct Validity: Suppose you
have developed a new measure of self-esteem
(i.e., a persons sense of self-worth).
Which constructs listed below would you expect
to show convergent validity for your measure
and for which would you expect discriminant
validity?
Measures of:
Depression, well-being, intelligence, extraversion, age, sensationseeking, social anxiety, life satisfaction, grade point average,
reading comprehension, artistic ability

You may see that you need to state a theory that defines
self-esteem and describes how it relates to other
constructs!

Measures in Survey Research

(continued)
Correlations demonstrating construct validity and
reliability are displayed in a correlation matrix:
(1) (2)
(3)
(4)
(1) new self-esteem, Time 1 1.0 ------(2) new self-esteem, Time 2 .85 1.0 ----(3) positive affect
.90 .90 1.0 --(4) artistic ability.10 .10 .15 1.0
Correlation in red indicates test-retest reliability.
Correlations in green indicate convergent validity.
Correlations in yellow indicate discriminant validity.
Correlations of 1.0 indicate the correlation of a measure
with itself.

Constructing a Questionnaire
The best choice for selecting a questionnaire is to
use one that already has been established as
reliable and valid.
If a suitable measure cannot be found,
researchers choose to create their own
questionnaire.
It may seem easy, but a lot goes into developing a
reliable and valid questionnaire.

Constructing a Questionnaire
(continued)
Important steps for preparing a
questionnaire:
1. Decide what information should be sought.
2. Decide what type of questionnaire should be
used (e.g., will it be self-administered?).
3. Write a first draft of the questionnaire.
4. Reexamine and revise the questionnaire after it
is reviewed by experts.
5. Pretest the questionnaire using a sample of
respondents under conditions similar to the
planned administration of the survey.
6. Edit the questionnaire, and specify the
procedures for its use.

Next steps include establishing reliability

and validity of the questionnaire.

Constructing a Questionnaire
(continued)
Guidelines for Writing Survey Questions

(1) Choose how participants will respond:

free-response: Open-ended questions allow greater
flexibility in responses but are difficult to code.
closed-response: Multiple choice and true-false are
quicker to respond to and easier to score, but may not
accurately describe individuals responses.

(2) Use simple, direct, and familiar vocabulary; keep

questions short (20 or fewer words).

Constructing a Questionnaire
(continued)
Guidelines for Writing Survey Questions, continued
(3)

Write clear and specific questions:

Avoid double-barreled questions
(e.g., Do you support capital punishment and
abortion?).

Place any conditional phrases at the beginning of

the question
(e.g., If you were forced to leave your current city,
where would you live?

Avoid leading questions

(e.g., Most people favor gun control; what do you
think?).

Avoid loaded (emotion-laden) questions

(e.g., People who discriminate are racist pigs: T or F).

Constructing a Questionnaire
(continued)
Ordering of Questions
For self-administered questionnaires, place the most
interesting questions first to capture respondents
attention.
For personal and telephone interviews, place
demographic questions first to establish rapport with the
respondent.
Use funnel questions: Start with the most general
questions, and move to more specific questions for a
given topic.
Use filter questions: These questions direct respondents
to the survey questions that apply directly to them.

Thinking Critically About

Survey Research
Correspondence Between Reported and Actual Behavior

Peoples responses on surveys may

not be truthful.
Reactivity: People sometimes dont
report truthful responses, because they
know the information is being recorded.
Social Desirability occurs when people
respond to surveys as they think they
should, rather than how they actually
feel or believe.

Thinking Critically About

Survey Research (continued)

Questionnaires have been developed to

assess the degree to which respondents
endorse socially-desirable responses.

Sample items from the Marlowe-Crowne

(1964) Social Desirability Scale.
For each item, can you tell whether True
or False represents a socially desirable
response?
No matter who Im talking to, Im always a good listener.
T F
I like to gossip at times.
T F
Im always willing to admit it when I make a mistake. T F
I have almost never felt the urge to tell someone off. T F

Thinking Critically About

Survey Research (continued)
Generally, researchers accept peoples
responses as truthful, unless theres
reason to suspect otherwise
For example, responses arent consistent
or visual pattern of responses forms a
picture.

Because behavior doesnt always

match verbal reports of behavior, the
multimethod approach to answering
questions in psychology is best.

Thinking Critically About

Survey Research (continued)
Correlation and Causality

Correlation does not imply causation.

Example: Correlation between being
socially active (outgoing) and life
satisfaction
Three possible causal relationships:
A causes B (being outgoing causes people
to be more satisfied with their life)
B causes A (being more satisfied with life
causes people to be more outgoing)

Thinking Critically About

Survey Research (continued)
Correlation and Causality, continued

C causes A and B
Some third variable may be responsible for
the relationship between social activity
and life satisfaction.
For example, having more friends (a third
variable) may cause people to be more
outgoing and to be more satisfied with
their life.
A correlation that can be explained by a
third variable is called a spurious
relationship.

Thinking Critically About

Survey Research (continued)
Path Analysis is a sophisticated statistical
technique that allows researchers to tease apart
complex correlational relationships among
variables.
Mediator variables are variables that are used to
explain the correlation between two variables.
Moderator variables are variables that affect the
direction or strength of the relationship between
two variables.

Example of Path Analysis

Valentiner et al. (1994) examined students
perceptions of parental support as they
adjusted to college over a two-year period.
They found that students perceptions of positive
support from parents was correlated with greater
well-being and less distress over the two-year period.
Parental support Positive changes in psychological
adjustment
This is called a direct relationship or Path a.

Example of Path Analysis (continued)

Valentiner et al. sought to explain why this relationship
exists. They predicted that the ways students cope
with stress might explain the relationship between
parental support and adjusting to college.
Specifically, they studied the extent to which students
used approach coping strategies (e.g., problem
solving) rather than avoidance coping strategies (e.g.,
trying to forget the problem).
They observed two important correlations:
Perceptions of parental support were correlated with greater
approach coping. (Path b)
Greater approach coping was correlated with psychological
adjustment. (Path c)

These two correlations represent the indirect

relationship between parental support and
psychological adjustment.

Example of Path Analysis (continued)

Greater approach coping
(path b) (path c)
Parental support
adjustment
(path a)

Psychological

Approach coping is the mediating

variable in the relationship between
parental support and psychological
adjustment.

Example of Path Analysis (continued)

This indirect relationship between parental
support, greater approach coping, and
adjustment was not observed for all students.
This is where a moderator variable enters the
story.
The moderator that Valentiner and his
colleagues studied was the controllability of
the most important problem students
confronted with their coping strategies.
Students in the study were divided into two groups
based on whether their most important problem was
Controllable or
Uncontrollable.

Example of Path Analysis (continued)

What happened when students reported
their life event was controllable?
Valentiner observed that the indirect,
mediated relationship described these
students.
For controllable problems, perceptions of
parental support may have helped students to
use more effective approach coping strategies
which was related to better psychological
adjustment.
Greater approach coping

(path b) (path c)
Parental support

Psychological adjustment

Example of Path Analysis (continued)

A different outcome occurred when
students reported about uncontrollable
events.
For uncontrollable events, the direct
relationship between parental support and
adjustment best described students
adjustment
The mediator of approach coping was not an
important factor.
Parental support Positive changes in psychological
adjustment

Example of Path Analysis (continued)

The moderator variable of uncontrollable
vs. controllable events affected the
relationship between parental support and
psychological adjustment.
Controllable events: The mediating variable of
approach coping was significantly related to
parental support and adjustment.
Uncontrollable events: The mediating variable
of approach coping was not related to parental
support and adjustment.

Example of Path Analysis (continued)

How can it be that adaptive approach coping
was not associated with students adjustment
in the face of uncontrollable events?
Valentiner et al. argued that when events are
uncontrollable, it doesnt make sense to use approachcoping strategies such as problem-solving techniques
(this is like hitting your head against the brick wall).
When events are uncontrollable, it may be best not to
try to solve them. In this case, perceptions of parental
support become important for positive adjustment.

Remember: Although path analysis helps us to

understand relationships among variables,
these relationships are still correlational.
We cannot make definitive causal statements.
Other untested variables are related to
students adjustment.