Practice: Get quality data into your evaluator’s hands

Key Action: Use technique to ensure valid and reliable data

TOOL: Making Sure Data Are Valid and Reliable

Purpose: Before you can confidently interpret and analyze your evaluation data, you
must ensure that the data you collect are valid and reliable. Otherwise, they
won’t adequately support your outcomes. Use the questions and suggestions
in this table to ensure the data you collect are valid and reliable.

Instructions: 1. Review the “Questions to Consider” in the table to ensure that the data
you collect for your evaluation are valid and reliable, and that your sample
size is adequate.

2. Consider the suggestions or proactive measures you might take to ensure

valid and reliable data.

3. Based on what you learn, note specific actions you might use in your
district to ensure that your evaluation generates valid and reliable data.

1

Questions to consider
Do the data represent the
outcomes that the instrument
is supposed to measure? Are
the data valid?
A valid measure assesses what
it is designed to measure, which
allows for comparison of results
across studies.
Have we ensured that our
measures are reliable?
A reliable measure produces
stable responses regardless of
the data collector. An unreliable
measure will yield varied
responses depending on
differences between interviewers
or data collectors.
Practice: Get quality data into your evaluator’s hands
Key Action: Use technique to ensure valid and reliable data
Making Sure Data Are Valid and Reliable
Suggestions Actions we’ll consider
First make sure you select instruments that measure the
kinds of outcomes your magnet program is expected to
produce. You can increase measurement validity by using
field-tested instruments that have demonstrated reliability
and validity.
Note: State assessments have been recalibrated or revised
during the years of your study, so you may not be able to
compare data from year to year.
Discuss with your evaluator how you will test for reliability of
your instruments. You may want to borrow from existing
instruments, have an expert panel review and react to new
instruments, or pilot test the instruments in real settings and
among members of your target audience. Build in time and
resources to test for reliability and calculate reliability
coefficients so that you can assure stakeholders that you
have strong instruments.
Note that qualitative data collection (e.g., observations,
open-ended interviews) poses different validity challenges.
Ideally, the instruments you create for these purposes will
require low levels of inference. Provide time and resources
for researcher training in use of the instruments to minimize
differences in participant responses across data collectors.
2
 Practice: Get quality data into your evaluator’s hands
Key Action: Use technique to ensure valid and reliable data

Questions to consider Suggestions Actions we’ll consider

Do we have the right data? You will need an adequate sample size to ensure your data
Do we have enough data? are valid. Make sure you begin with large enough numbers
of students and schools in your evaluation study, taking
To be valid for decision-making, projected attrition into account. Then, as data collection
data must answer your begins, check your data to make sure that subgroup data
questions about program are appropriately coded and that sufficient numbers of
outcomes and include a students in subgroups have taken the tests as planned.
sufficient number of participants Otherwise you may not have enough students in particular
to be representative of your subgroups to make those data valid.
target population and its various
subgroups. Validity of data is
improved when data collection is
“triangulated,” for example,
when various methodologies are
used to measure the same
phenomenon, or multiple
researchers conduct a
structured observation of the
same phenomenon.