Types of Reliability

Reliability of Consistency of What?

 Observers or raters
 Tests over time
 Different versions of the same test
 A test at one point in time
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

Observer 1
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

Observer 1 Observer 2
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

?
=
Observer 1 Observer 2
Inter-Rater or Inter-Observer Reliability

 Are different observers consistent?

 Can establish this outside of your study
in a pilot study.
 Can look at percent of agreement
(especially with category ratings).
 Can use correlation (with continuous
ratings).
Test-Retest Reliability

Time 1 Time 2
Test-Retest Reliability

Test = Test

Time 1 Time 2
Test-Retest Reliability

Stability over time

Test = Test

Time 1 Time 2
 Test-Retest Reliability

 Measure instrument at two times for

multiple persons.
 Compute correlation between the two
measures.
 Assumes there is no change in the
underlying trait between time 1 and time
2.
Parallel-Forms Reliability

Time 1 Time 2
Parallel-Forms Reliability

Form A
=
Form B

Time 1 Time 2
Parallel-Forms Reliability

Form A Stability across forms

=
Form B

Time 1 Time 2
 Parallel-Forms Reliability

 Administer both forms to the same

people.
 Get correlation between the two forms.
 Usually done in educational contexts
where you need alternative forms
because of the frequency of retesting
and where you can sample from lots of
equivalent questions.
Internal Consistency Reliability

Average inter-item correlation

Internal Consistency Reliability

Average Inter-Item Correlation

Test
Internal Consistency Reliability

Item 1 Average Inter-Item correlation

Item 2

Item 3
Test
Item 4

Item 5

Item 6
Internal Consistency Reliability

Item 1 Average inter-item correlation

Item 2 I1 I2 I3 I4 I5 I6

I1 1.00
Item 3 I2 .89 1.00
Test I3 .91 .92 1.00
Item 4 I4 .88 .93 .95 1.00
I5 .84 .86 .92 .85 1.00
I6 .88 .91 .95 .87 .85 1.00
Item 5

Item 6
Internal Consistency Reliability

Item 1 Average inter-item correlation

Item 2 I1 I2 I3 I4 I5 I6

I1 1.00
Item 3 I2 .89 1.00
Test I3 .91 .92 1.00
Item 4 I4 .88 .93 .95 1.00
I5 .84 .86 .92 .85 1.00
I6 .88 .91 .95 .87 .85 1.00
Item 5

Item 6
.90
Internal Consistency
Reliability
Average item-total correlation
Internal Consistency
Reliability
Average item-total correlation

Test
 Internal Consistency
Reliability
Average item-total correlation
Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency
Reliability
Average item-total correlation
Item 1
I1 I2 I3 I4 I5 I6

Item 2 I1 1.00
I2 .89 1.00
Item 3 I3 .91 .92 1.00
Test I4 .88 .93 .95 1.00
I5 .84 .86 .92 .85 1.00
Item 4 I6 .88 .91 .95 .87 .85 1.00
Total .84 .88 .86 .87 .83 .82 1.00
Item 5

Item 6
 Internal Consistency
Reliability
Average item-total correlation
Item 1
I1 I2 I3 I4 I5 I6

Item 2 I1 1.00
I2 .89 1.00
Item 3 I3 .91 .92 1.00
Test I4 .88 .93 .95 1.00
I5 .84 .86 .92 .85 1.00
Item 4 I6 .88 .91 .95 .87 .85 1.00
Total .84 .88 .86 .87 .83 .82 1.00
Item 5

Item 6
.85
Internal Consistency
Reliability
Split-half correlations
Internal Consistency
Reliability
Split-half correlations

Test
 Internal Consistency
Reliability
Split-half correlations
Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency
Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency
Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4

Item 5
Item 2 Item 5 Item 6
Item 6
 Internal Consistency
Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4 .87
Item 5
Item 2 Item 5 Item 6
Item 6
Internal Consistency
Reliability
Cronbach’s alpha ()
Internal Consistency
Reliability
Cronbach’s alpha ()

Test
 Internal Consistency
Reliability
Cronbach’s alpha ()
Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency
Reliability
Cronbach’s alpha ()
Item 1 item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

.87 .87 .87

Item 2
item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency
Reliability
Cronbach’s alpha ()
Item 1 item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

.87 .87 .87

Item 2
item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3 SH1 .87

Test SH2 .85
Item 4 SH3 .91
SH4 .83
SH5 .86
Item 5 ...
SHn .85
Item 6
 Internal Consistency
Reliability
Cronbach’s alpha ()
Item 1 item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

.87 .87 .87

Item 2
item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3 SH1 .87

Test SH2 .85
Item 4 SH3 .91
SH4 .83
SH5 .86
Item 5 ...
SHn .85
Item 6
 = .85
 Internal Consistency
Reliability
Cronbach’s alpha ()
Item 1 item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

.87 .87 .87

Item 2
item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3 SH1 .87

Test SH2 .85
Item 4 SH3 .91 Like the average
SH4 .83 of all possible
SH5 .86
Item 5 ... split-half
SHn .85 correlations
Item 6
 = .85
Internal Consistency Reliability - Summary

 Average inter-item correlation

 Average item-total correlation
 Split-half reliability
 Cronbach’s alpha ()