Critical Appraisal

How to Use an Article About a Diagnostic Test

Madeleine Grace M. Sosa, MD FPPS,FPNA,FCNSP MS Clinical Epidemiology

General Objective
To apply the various rules of evidence in arriving at decisions regarding the usefulness of a diagnostic test

Specific Objectives
To discuss the rationale for each of the users guides pertaining to the validity of claims on accuracy To define and differentiate the concepts of validity and reliability, as they pertain to the results of a diagnostic test To learn the definitions of and calculations for sensitivity, specificity, positive and negative predictive value, likelihood ration, pretest and post test probability, pre-test and post-test odds

Specific Objectives
To recognize the advantages and disadvantages of using likelihood ratios, sensitivities, specificities and predictive values in measuring the accuracy of diagnostic tests To learn how to estimate posttest probabilities given the pre-test probability and a measure of accuracy

Clinical Scenario
Your nieces classmate Marco was admitted in a hospital while on vacation in the province. Apparently he presented with high grade fever of 5 days duration with associated abdominal pain. He remained well except for generalized body malaise and slightly decreased appetite. His clinical signs are said to be suggestive of typhoid fever or dengue fever. Investigations included a CBC and platelet, which showed normal findings except for a wbc of 4. Chest x-ray and urinalysis were normal. Blood for culture was extracted. Typhidot-M was also extracted and reported to be positive.

Cont
Quite anxious and wanting to quickly arrive at a specific diagnosis and knowing that you are a physician, Marcos father who is a podiatrist asked your help. He wanted to know how sensitive is the Typhidot-M test in the diagnosis of typhoid fever compared to a blood culture, which will take another 5 days to report.

Architecture of a focused question: a 4-part review question

P I C O
+ study design
Richardson et al. The well-built clinical question: a key to evidence-based decisions. ACP Journal Club 1995;A-12 Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med 1997;127:380-7.

CATmaker

Search Strategies

Search Strategies

Search Strategies

 Evaluation of a simple and rapid dipstick assay for the diagnosis of typhoid fever in Indonesia by Hussein Gasem et al, published in the Journal of Medical Microbiology in 2001

Evaluating and applying the results of a diagnostic tests

APPRAISING DIRECTNESS APPRAISING VALIDITY APPRAISING OF RESULTS ASSESSING APPLICABILITY INDIVIDUALIZING RESULTS

I. Evaluating directness
Does the study provide a direct enough answer to your clinical question in terms of P: patient population with a certain disease I/E: the exposure( or treatment) to be administered O: the outcome( or condition ) that the treatment are intended to prevent or promote M: Methodology

Evaluating Directness
Clinical Question Research Question

P
E/I C

O
M

Appraising diagnostic tests: 3 easy steps

Appropriate spectrum of patients?
Does everyone get the gold standard?

1. Are the results valid?

Is there an independent, blind or objective comparison with the gold standard?

2. What are the results?

3. Will they help me look after my patients?

II. Appraising Validity

Validity Questions

1. Was the reference standard and acceptable one ?

2. Was the reference standard interpreted independently from the test in question?

Appraising diagnostic tests: 3 easy steps

1. Are the results valid?

Sensitivity, specificity

2. What are the results?

Likelihood ratios

Predictive values

3. Will they help me look after my patients?

III. Appraising Results

What likelihood ratios were associated with the range of possible test results ?

APPRAISING RESULTS?
What likelihood ratios were associated with the range of possible results?
Measures of accuracy sensitivity, specificity, positive predictive value, negative predictive value Estimate the likelihood ratios for each of the possible results Estimate the pre-test and post-test probability

2 by 2 table: sensitivity

+ +
Test
a

Disease

Proportion of people with the disease who have a positive test result.

True positives c

So, a test with 84% sensitivity.means that the test identifies 84 out of 100 people WITH the disease

False negatives

Sensitivity = a / a + c

2 by 2 table: sensitivity + +
Test
70

Disease

Proportion of people with the disease who have a positive test result.

So, a test with 76.9% sensitivity.means that the test identifies 70 out of 91 people WITH the disease

21

Sensitivity = 70/ 91= 0.769

2 by 2 table: specificity

+ +
Test

Disease
b

Proportion of people without the disease who have a negative test result.

False positives
d

True negatives

Specificity = d / b + d

2 by 2 table: specificity + +
Test Disease
b

Proportion of people without the disease who have a negative test result. So, a test with 75% specificity will be NEGATIVE in 75 out of 100 people without the disease

False positives
d

True negatives

Specificity = d / b + d

2 by 2 table: specificity
Disease

+ +
Test
b

7
d

Proportion of people without the disease who have a negative test result. So, a test with 94% specificity will be NEGATIVE in 109 out of 116 people without the disease

109

Specificity=109/116=0.94

Reference standard
Test result Disease present Disease Present Disease Absent True Positive (a) False Positive (b) Total A+B

Disease absent

False Negative (c)

True Negative (d)

C+D

Sensitivity = a /( a+c )
Specificity = d /( b+d )

LR for positive test result = [ a /(a+c) / [b / ( b+d) ]

LR for negative test result = [ c /(a+c) / [d / ( b+d) ]

Blood Culture
Dipstick test
Positive

Disease Present Disease Absent

True Positive (a) ( 70 ) False Positive (b) (4)

Total
A+B

Negative

False Negative (c ) ( 21 )

True Negative (d) ( 32 )

C+D

Sensitivity = a /( a+c )
Specificity = d /( b+d )

LR for positive test result = [ a /(a+c) / [b / ( b+d) ]

LR for negative test result = [ c /(a+c) / [d / ( b+d) ]

RESULTS ( page 174, 2nd column)

What are the results?

Are the likelihood ratios for the test results presented or data necessary for their calculation provided?
Usefulness of the test = determined by the accuracy with which it identifies the target disease Accuracy measure --- LIKELIHOOD RATIO

Likelihood ratios
Can be used in situations with more than 2 test outcomes Direct link from pre-test probabilities to posttest probabilities

Likelihood ratios
Positive likelihood ratio (LR+)

How much more likely is a positive test to be found in a person with the disease than in a person without it?
LR+ = sens/(1-spec)

Negative likelihood ratio (LR-)

How much more likely is a negative test to be found in a person without the condition than in a person with it? LR- = (1-sens)/(spec)

What do likelihood ratios mean?

LR<0.1 = strong negative test result

LR=1 No diagnostic value

LR>10 = strong positive test result

2 x 2 table: positive likelihood ratio

+ +
Test
SENSI

Disease

How much more often a positive test occurs in people with compared to those without the disease

100-SPECI

LR+ = a/a+c / b/b+d or

100-SENSI

SPECI

LR+ = sens/(1-spec)

BLOOD TEST REST Positive Intermediate Negative Column Total Disease Present A C E ( A+C+E)

CULTURE Disease Absent B D F ( B+D+F) Row Total A+B C+D E+F

Odds ratio of (+) = (A/B) / ( A+C+E)/ ( B+D+F) Odds ratio of intermediate = (C/D)/ ( A+C+E)/ ( B+D+F) Odds ratio of negative = (E/F) ( A+C+E)/ ( B+D+F)

Positive Likelihood Ratios

Dipstick Result With Typhoid Fever Without Typhoid Fever + Likelihood Ratio

Negative
+1 +2

21
18 38

32
1 1

(21/32)/( 91/36)

+3
+4 TOTAL

11
3 91

1
1 36

Appraising diagnostic tests: 3 easy steps

1. Are the results valid?

2. What are the results?

Can I do the test in my setting? Do results apply to the mix of patients I see? Will the result change my management? Costs to patient/health service?

3. Will they help me look after my patients?

IV. ASSESSING APPLICABILITY

Are there biologic issues that may affect accuracy of the test? Are there socio-economic issues that may affect accuracy of the test?

V. INDIVIDUALIZING THE RESULTS

How will the results affect the probability of disease in your patient? ( Estimate the individualized post-test probability of your patient) Is this test useful for your patient?

Determine Applicability Individualizing the results

1. Estimate the Pre-test Probability 2. Calculate the post-test Probability
Step1: Convert the pre-test probability pre-test odds Step2: Multiply the pre-Test odds by the LR of the expected lab results = Post-test Odds Convert the Post-test odds = Post-test probability

Pretest probability and Odds

Pretest Prob : 70% (70/100) Pretest Odds : 2.33 ( 70/30) Odd: X /1-X

Posttest odds and probabiliy

Posttest odds = LR x pretest odds Posttest probability = X/ 1+X x

Will the result change management?

0%

Probability of disease

100%

No action

Test

Action (e.g. treat)

Testing threshold

Action threshold

Will the result change management?

0%

Probability of disease

100%

No action

Test

Action (e.g. treat)

Testing threshold

Action threshold

Will the results help me in caring for my patient?

Will the reproducibility of the test result and its interpretation be satisfactory in my setting? Are the results applicable to my patient? Will the results change my management? Will patients be better off as a result of the test?

 Will the results change my management?

Test threshold Treatment threshold When the probability of the target disorder lies between the test and treatment thresholds, further testing is mandated

Bayesian reasoning
Pre test 5%

Post test 20%

? Appendicitis:

McBurney tenderness LR+ = 3.4

Fagan nomogram
%

Will the result change management?

0%

Probability of disease

100%

No action

Test

Action (e.g. treat)

Testing threshold

Action threshold

Normogram

Pre test

Post test

 Will the reproducibility of the test result and its interpretation be satisfactory in my setting? The value of the test depends on its ability to yield the same results when re-applied to stable patients Poor reproducibility can results from problems with the test itself or in cases where the test will require interpretation

 Are the results applicable to my patient?

Similar setting Patient meets all the study inclusion criteria Patient does not violate any of the exclusion criteria

 Will patients be better off as a result of the test? Does it adds information beyond that otherwise available? Does it leads to change in management that is ultimately beneficial to the patient? The value of an accurate test will be undisputed when the target disorder, if left undiagnosed, is dangerous, the test has acceptable risks and effective treatment
exists