Methodology

Tips for Constructing Instruments

 Matching methods to research paradigm
Method Quantitative
Written Standardized Instrument
Instrument (validity and reliability
established)
Constructed survey or
interview guide (Use some
questions from other surveys;
construct new questions). May
determine validity and
reliability after administration
Observation Structured observation –
predetermined items are
counted in terms of frequency,
duration, occurrence,
permanent product, etc.
Content Structured counting of a pre-
analysis determined phenomena in
terms of frequency, etc.
Qualitative
No predeveloped questions in survey
or interview guide
Predeveloped questions – may be
administered in no established order;
questions may be added depending
on circumstances.
Predeveloped questions – established
order
Open-ended observations – trying to
identify themes and patters
Open-ended analysis; themes and
patterns are identified.
 Quantitative/Descriptive Studies
involve specific Research Designs
• Designs structure how the interviews are
conducted.
• Allow different groups and time periods to
be compared.
• Minimize threats to internal and external
validity.
 Components of Designs
• Sampling (Random; Other probability
sampling or nonprobability – such as
convenience, purposive, or snowball)
• Groups – one group; two, or multiple
group comparisons.
• Time intervals – pre and post tests or
multiple observations.
 Common design types
• Experimental Design (pre-post; random sample,
two or more groups compared.
• Quasi Experimental with nonequivalent
comparisons groups- pre and post.
• Quasi Experiments with post-test only.
• One group - pre and post test only.
• One group – multiple time periods measured
(time series or longitudinal studies)
• One group – one measure – usually called
cross-sectional survey.
 Random designs help ensure
internal/external validity
• Internal validity – attribute of people included or
way the research was conducted may limit our
ability to establish a real connection between the
independent and dependent variables.
• External validity – can we generalize results to
other people, settings, and times.
(Note for some types of studies – qualitative
research and needs assessments and some
types of evaluations – we do not need
generalizability. We are only interested in finding
things out about a particular situation).
 Exceptions to Experimental/Quasi-
experimental Designs: One group studies
• Case studies (combination of methods
looking at one individual, group, family,
organization, or community)
• Survey research
• Consumer satisfaction surveys
• Needs Assessments
 Use these alternative methods
• For descriptive studies
• When you do not have an intervention
• When you want to look at a specific
situation or group and you do not intend to
generalize.
• When it would be difficult or unethical to
create an experimental or control group.
• When you do not have time or money to
conduct an experiment.
Threats to Internal Validity Include:
• History
• Maturation
• Testing
• Instrumentation
• Regression to the mean
• Selection
• Interaction between Selection and the
other threats to Internal Validity.
 Qualitative Research
• Usually one group
• Comparisons may be made after data
analysis – decision to compare made
based on obvious patterns in the data.
• Sampling – nonprobability – purposive,
convenience, snowball
• Uses observation, interviews, and content
analysis.
 Focus groups may also be used to conduct qualitative
research. Focus groups:

• Are used to assess people’s perceptions about programs or

services.
• Are used to generate recommendations for changing programs and
services.
• Are often used in conjunction with needs assessments or to
generate ideas or explore themes that can be used in descriptive
studies.
• Are used to answer questions about how or why programs work.
• May be used in conjunction with other research methods.
• Include six to eight participants who are selected using purposive
sampling methods.
• Utilize a semi-structured interview guide with 6-8 questions.
• Require that a facilitator ask each of the questions and solicit
responses from members.
• Are used to generate a common response or a consensus from
group members.
 In any study:
• A mixture of methods may be used.
• Researchers may seek to increase reliability and
validity by using both quantitative and qualitative
approaches. This is called triangulation!
• Methods used should match the research question:
a) What happens/what is the outcome? quantitative)
b) Why does something happen? How does it happen?
(qualitative)
c) A description of what exists (Descriptive)
 Descriptive studies
Generally involve surveys, standardized
interviews, or the use of pre-existing
(secondary data).
• Include needs assessments.
• Can include both qualitative and
quantitative data collection. In some
cases, one instrument can include a mix of
qualitative and quantitative data collection
methods.
 Survey and Interview Construction
Tips
• Only measure one concept per question.
• Don’t write the question to solicit a specific response
• Response categories (if used) should be mutually
exclusive and exhaustive
• Questions should be tactful and should not invoke a
social stigma.
• May be filter questions – respondents can be directed to
successive questions based on one response.
• Demographic questions are generally put at the
beginning or the end of the instrument.
• Be aware that respondents do not like to answer some
types of questions (age, income, organization budgets,
etc.)
 Example of closed-ended
(response category) questions
1. What is your gender
1. Male 2. Female

2. What is your ethnic identification?

1. African American
2. American Indian
3. Asian/Pacific Islander
4. European American
5. Mexican American/Latino
6. Multicultural or multi-racial
7. Other ____________________
 More examples
3. What is your age?
1. 18 or under
2. 19-29
3. 30-49
4. 50 – 64
5. Over 65

4. How satisfied are you with Social Work 292?

1. Very satisfied
2. Satisfied
3. Neutral
4. Unsatisfied
5. Very Unsatisfied
 Sample question: How would you assess
your own practice ability in the following
areas:
High Medium Low

Write Case 1 2 3
Notes
Write Reports 1 2 3

Use 1 2 3
Computers
Apply 1 2 3
Theories
Evaluate 1 2 3
Practice
 Can you describe how you first became aware of your deafness
1
.
2 How do you see yourself today, in terms of your deafness?
.
3 What does your deafness mean to you?
.
Can you describe any particularly difficult or traumatic experiences
4
in your life related to your deafness?
.

5 Can you describe how you fit into deaf culture?

.
To what extent do you consider yourself active in both the deaf
6
world and the hearing world?
.
 Each type of question corresponds to a level of
measurement. These levels of measurement will
determine what type of data analysis we use:
• All qualitative research is nominal.
• All categorical data in which there is no
difference in value among the categories is
nominal.
• All categorical data in which there is an implied
ranking (for example, high-medium-low) is
ordinal.
• Any question that asks respondents to rank
order responses is also ordinal.
• Any question in which the response is a number
or can be interpreted as a number with equal
values among the data points is ratio.
 For example, ratio data can
include:
• What is your age? _______
• Five point likert scales (1=Very likely, 2 =
likely, etc. Usually likert scales are
illustrated on the survey instrument as 1
2 3 4 5. Respondents are asked to circle
the best response.
• Scores on standardized instruments are
also interpreted as ratio data.