Professional Documents
Culture Documents
Tools
Payam Kabiri, MD. PhD.
Clinical Epidemiologist
Is it Reliable?
Are the results real or because of chance?
Is it Applicable?
Will the results help locally?
Key Steps to
Effective Critical Appraisal
+ a b
- c d
Sensitivity = a / (a + c)
63
Validity of Screening Tests
True Disease Status
+ -
+ a b
- c d
Specificity = d / (b + d)
64
Two-by-two tables can also be used for
calculating the false positive and false
negative rates.
The false positive rate = false positives /
(false positives + true negatives). It is also
equal to 1- specificity.
The false negative rate = false negatives /
(false negatives + true positives). It is also
equal to 1 – sensitivity.
An ideal test maximizes both sensitivity
and specificity, thereby minimizing the
false positive and false negative rates.
Validity of Screening Tests
Breast Cancer
Physical Exam + -
and Mammo-
graphy + 132 983
- 45 63650
Sensitivity: a / (a + c)
Sensitivity =
Specificity: d / (b + d)
Specificity =
67
Validity of Screening Tests
Breast Cancer
Physical Exam + -
and Mammo-
graphy + 132 983
- 45 63650
Sensitivity: a / (a + c)
Sensitivity = 132 / (132 + 45) = 74.6%
Specificity: d / (b + d)
Specificity = 63650 / (983 + 63650) = 98.5%
68
Probability of Disease
Pre-test probability of disease = disease
prevalence
Post-test probability of disease =
If normal, c/(c+d)
If negative, a/(a+b)
69
Likelihood Ratios (Odds)
This is an alternative way of describing the
performance of a diagnostic test. Similar to S
and S, and can be used to calculate the
probability of disease after a positive or
negative test (predictive value). Advantage of
this is that it can be used at multiple levels of
test results.
The probability of a test result in those with
the disease divided by the probability of the
result in those without the disease. (how
many more times (or less) likely a test result
is to be found in the disease compared with
70 the nondiseased.
“Post-test probability” depends on
the accuracy of the diagnostic test
and the pre-test probability of
disease
Clinicalexperience
Epidemiological data
“Clinical decision rules”
Guess
72
LR+= sensitivity/(1-
specificity)
LR-
= (1-
sensitivity)/specificity
73
Switch to Odds
1000 patients. 100 have disease. 900
healthy. Who will test positive?
74
From pretest to posttest odds
Diseased 100 X.95 =_95
Healthy 900 X.08 = 72
100 = Pretest odds
900
.95 = Sensitivity__ = prob. Of pos test in dis
.08 1-Specificity prob. Of pos test in hlth
95 =Posttest odds. Probability is 95/(95+72)
72
75
Remember to switch back to probability
76
What is this second fraction?
Likelihood Ratio Positive
Multiplied by any patient’s pretest odds
gives you their posttest odds.
Comparing LR+ of different tests is
comparing their ability to “rule in” a
diagnosis.
As specificity increases LR+ increases
and PPV increases (Sp P In)
77
Clinical Interpretation:
Likelihood Ratios
Likelihood ratio
LR+ = Sensitivity/(1-Specificity)
LR- = (1-Sensitivity)/Specificity
Likelihood Ratio Negative
Diseased 100_ X.05 =_5__
Healthy 900 X.92 = 828
100 = Pretest odds
900
.05 = 1-sensitivity = prob. Of neg test in dis
.92 Specificity prob. Of neg test in hlth
(LR-)
Posttest odds= 5/828. Probability=5/833=0.6%
As sensitivity increases LR- decreases and NPV
increases (Sn N Out)
79
Remember to switch to probability and
also to use 1 minus
80
Value of a diagnostic test depends
on the prior probability of disease
Prevalence Prevalence
(Probability) = 5% (Probability) = 90%
Sensitivity = 90% Sensitivity = 90%
Specificity = 85% Specificity = 85%
PV+ = 24%
PV+ = 98%
PV- = 99%
PV- = 49%
Test not as useful
when disease unlikely Test not as useful
when disease likely
81
Clinical interpretation of post-
test probability
0 1
Testing Treatment
threshold threshold
Disease Disease
ruled out ruled in
82
Samples of positive and
negative likelihood ratios
84
Advantages of LRs
Clear separation of test characteristics
from disease probability.
85
Likelihood Ratios - Advantage
Provide a measure of a test’s ability to rule
in or rule out disease independent of
disease probability
Test A LR+ > Test B LR+
Test A PV+ > Test B PV+ always!
Test A LR- < Test B LR-
Test A PV- > Test B PV- always!
86
Using Likelihood Ratios to Determine Post-
Test Disease Probability
Pre-test Pre-test
probability odds of
of disease disease Post-tes t Post-tes t
odds of probability
disease of disease
Likelihood
ratio
87
88
Predictive Values
Alternate formulations:Bayes’ Theorem
PV+ =
Se Pre-test Prevalence
Se Pre-test Prevalence + (1 - Sp) (1 - Pre-test Prevalence)
PV- =
Sp (1 - Pre-test Prevalence)
Sp (1 - Pre-test Prevalence) + (1 - Se) Pre-test Prevalence
89
Clinical Interpretation: Predictive Values
F <40 32 88
F 40-50 46 80
F 50+ 62 68
M <40 62 68
M 40-50 75 54
M 50+ 85 38
90
Clinical Interpretation: Predictive Values
91
If Predictive value is more
useful why not reported?
Should they report it?
Only if everyone is tested.
And even then.
You need sensitivity and specificity from
literature. Add YOUR OWN pretest
probability.
92
So how do you figure pretest
probability?
Start with disease prevalence.
Refine to local population.
Refine to population you serve.
Refine according to patient’s presentation.
Add in results of history and exam (clinical
suspicion).
Also consider your own threshold for testing.
93
Pretest Probability: Clinical
Significance
Expected test result means more than
unexpected.
Same clinical findings have different
meaning in different settings
(e.g.scheduled versus unscheduled visit).
Heart sound, tender area.
Neurosurgeon.
Lupus nephritis.
94
What proportion of all patients
will test positive?
Diseased X sensitivity
+ Healthy X (1-specificity)
Prevalence X sensitivity +
(1-prevalence)(1-specificity)
We call this “test prevalence”
i.e. prevalence according to the test.
Which one of these test is the best
for SLE Dx?
Test Sensitivity Specificity LR(+)
ANA 99 80 4.95
dsDNA 70 95 14
ssDNA 80 50 1.6
Histone 30-80 50 1.1
Nucleoprotein 58 50 1.16
Sm 25 99 25
RNP 50 87-94 3.8-8.3
PCNA 5 95 1
Was it clear enough !
Reliability
Reliability
refers to the repeatability
or reproducibility of a test.
Sensitivity
LR Positive =
1 - Specificity
1 - Sensitivity
LR Negative=
Specificity
Continuous Measurements
Cutoff Value for Positive Test
Healthy
Diseased
Proportion
TN TP
FN FP
SBP
Continuous Measurements
Cutoff Value for Positive Test
Healthy
Diseased
Proportion
TN TP
FP
FN SBP
Continuous Measurements
Cutoff Value for Positive Test
Healthy
Diseased
Proportion
TN TP
FN
FP SBP
Standards for Reporting of
Diagnostic Accuracy
(STARD)
Improve the accuracy and completeness of
research reporting and allow readers to assess
the “potential for bias” in the study reported.
Always use:
Flowchart or Diagram
Checklist
FLOW CHART or Diagram
STARD checklist
METHODS
Participants 3 Describe the study population: The inclusion and exclusion criteria, setting and locations
where the data were collected.
4 Describe participant recruitment: Was recruitment based on presenting symptoms, results
from previous tests, or the fact that the participants had received the index tests or the
reference standard?
6 Describe data collection: Was data collection planned before the index test and
reference standard were performed (prospective study) or after (retrospective
study)?
Test methods 7 Describe the reference standard and its rationale.
8 Describe technical specifications of material and methods involved including how and
when measurements were taken, and/ or cite references for index tests and reference
standard.
9 Describe definition of and rationale for the units, cutoffs and/or categories of the results of the
index tests and the reference standard.
10 Describe the number,training and expertise of the persons executing and reading the
index tests and the reference standard.
11 reference standard were
Describe whether or not the readers of the index tests and
blind (masked) to the results of the other tests and describe any other
clinical information available to the readers.
STARD checklist (METHODS Continued….)
Reviews
Systematic reviews
Drawbacks to systematic
reviews/meta-analyses
Can be done badly
2 systematic reviews on same topic can
have different conclusions
Inappropriate aggregation of studies
A meta-analysis is only as good as the
papers included
Tend to look at ‘broad questions’ that
may not be immediately applicable to
individual patients
Some of the Appraising tools
Appraising systematic reviews
Critical Appraisal Skills Program (CASP):
Systematic Reviews
Systematic Review (of therapy) Worksheet
ARIF (Aggressive Research Intelligence Facility)
Appraising meta-analyses
QUOROM Statement Checklist
CASP
Screening Questions
1. Did the review ask a clearly-focused
question?
2. Did the review include the right type
of study?
– address the review’s question
– have an appropriate study design
Detailed Questions
3. Did the reviewers try to identify all
relevant studies?
– which bibliographic databases were used
– if there was personal contact with experts
– if the reviewers searched for unpublished studies
– if the reviewers searched for non-English-
language
studies
Detailed Questions
4. Did the reviewers assess the quality of
the included studies?
– if a clear, pre-determined strategy was used
to
determine which studies were included.
– a scoring system
– more than one assessor
Detailed Questions
5. If the results of the studies have been
combined, was it reasonable to do so?
– the results of each study are clearly displayed
– the results were similar from study to study
(look for tests of heterogeneity)
– the reasons for any variations in results are
discussed
Detailed Questions
6. How are the results presented and
what is the main result?