Diagnostic research

Lecture Contents
I.
II. Diagnostics in practice
- Explained with a case

Scientific diagnostic research Design

Data-analysis Reporting

III. IV.

Exercises Summary

Diagnostics in practice
Diagnostics always start with a patient with a complaint/symptom Case: neck stiffness Child, 2 years-old, comes to ER with parents Child turns out to have a very stiff neck What is the physicians aim?

Diagnostics in practice
Aim of the physician Quickly and efficiently determine the correct diagnosis Why diagnose? Basis medical handling Determines treatment choice Gives information about prognosis What are possible diagnoses for neck stiffness?

Diagnostics in practice
Differential diagnosis (DD)
Bacterial meningitis Viral meningitis Pneumonia ENT infection Other (e.g. myalgia)

What is the most important diagnosis? Which one does the physician not want to miss?

Diagnostics in practice
Most important diagnosis Bacterial meningitis (BM) If missed: often fatal

Diagnostics in practice
Suppose: 20% of all children on the ER with neck stiffness has BM 20% with disease in that population =
prevalence Prior-probability

What is your decision for the child in this case?

Diagnostics in practice
Decision for child in case Prior-probability too low to treat Prior-probability too high to send home

Decision: reduce uncertainty diagnostics

What is the best test?

Diagnostics in practice
Best test Lumbal punction (liquor culture)

Diagnostics in practice
Gold standard True disease status; truth
Never 24 karat

Reference standard/test Decisive test with doubt Perform reference test for everybody (=every child on ER with neck stiffness)?

Diagnostics in practice
Reference test for everybody? Unethical too invasive/risky Inefficient too expensive Do not perform unnecessarily
How should we then determine the probability of disease presence and what would be ideal?

Diagnostics in practice
How then? Simpler diagnostics:
Usually anamnesis, physical exam, simple lab tests, imaging, etc. Ideal: diagnosis without reference test

Diagnostic process in practice:

Stepwise process: less more invasive Not one diagnosis based on 1 test Each item: separate test

Diagnostics in practice
Suppose: after anamnesis & PE 10% probability of BM Probability of disease given test results = posteriorprobability The bigger the difference between prior and posterior probability, the better the diagnostic value of the tests Our decision for child in case: probability is too high to send home --> next step?

Diagnostics in practice
Next step
Additional research, e.g. blood tests (leucocytes, CRP, sedimentation, etc.)

Diagnostics in practice
Suppose: 1% posterior-probability after anamnesis, PE+ simple lab tests posterior probability low enough to send home Ideal diagnostic process: simple tests reduce posterior
probability to 0 or 100% (without reference) Most often physician continues testing until sufficiently sure (approximation of 0 or 100%) Choose when sufficiently sure: depends on prognosis of disease if untreated + risks/costs treatment

Diagnostics in practice
Summarizing What does diagnosing involve in practice? Estimation of probability of disease presence based on test results of the patient
When is the probability of disease best estimated? Why is this usually not done?

Diagnostics in practice
Why not all possible tests?
Invasive (for patient and budget) Unnecessary: different test results give same info However: In practice often more tested than necessary!

What diagnostics truly necessary diagnostic research

scientific

Study design
Scientific diagnostic research
What tests truly contribute to probability estimation? Has to serve practice follow practice

Study design
Research question Domain Study population Determinant(s): test(s) to study Endpoint: presence/absence disease (outcome) Study design: design Data analysis, interpretation + reporting

Research question
With as few as possible simple, safe, and cheap tests estimate the probability of the presence/absence of disease. Determinant-outcome relation: probability of disease as a function of test results
outcome = probability of disease = % = prevalence test results = determinants

Research question
Case
What tests contribute to probability estimation of presence or absence of BM in children with neck stiffness at the ER? Or: Determinants of presence/absence disease (BM)? %BM = (age, gender, fever, blood leucocytes, blood CRP, etc)

Research population
Case: All children with neck stiffness in 2002 at ER Utrecht

Domain
For whom domain, generalisation
= type of patient with certain symptom/complaint + setting

Research population = 1 sample from domain Case:

All children (e.g. in Western world) suspected of disease (BM) based on neck stiffness (characteristic) in secondary care (setting)

Determinants
= Tests to study
Diagnostic determinants All possible important tests (in domain) Case Items anamnesis, PE and lab (blood and urine) tests

Endpoint
True presence/absence disease = Diagnostic outcome = Results reference test
NB: reference = not infallible but always best available test in practice at that moment

Case
Positive liquor culture

PICO EBM
Population/ problem Intervention Comparison/ control Outcome Domain Determinant Reference test Outcome

Measure determinants/endpoint
Determinants
Without knowledge (blinded) of the outcome Same method in study and practice never measure more precisely than in practice (overestimation information yield)

Endpoint
Assessment blind for determinants With the best possible test known in practice

Study design
Observational and descriptive
Observational = no manipulation of determinants Descriptive = not causal if the determinant only predicts no hypothesis functional mechanism determinantoutcome

>1 determinant

Study design
Cross-sectional
= Simultaneously measure determinants and outcome

Data-analysis
After data collection, per patient
Value determinants (test results) Diagnostic outcome (reference test)

Data-analysis
Data analysis: 3 steps 1) Estimation a priori probability (without test results) 2) Compare each test result separately with reference = univariate 3) Compare combination of test results with reference = multivariate (via model)
- Following order in practice - Determine added value test result to already collected (previous) test results

Data-analysis
Case
Data scientific research available: 200 patients with neck stiffness at ER Liquor culture positive (BM+) n=40 Liquor culture negative (BM-) n=160 Step 1: A priori probability (prevalence) of BM? = % BM+ = 40/ 200 patients = 20%

Data-analysis
reading 2 by 2 table
Step 2: Analysis per determinant (univariate) Use 2 by 2 table Disease
Presence Absence False positive B D True negative

Test Positive

True positive A Negative C False negative

Data-analysis
reading 2 by 2 table
Horizontally
Positive predictive value (PV+) = probability Disease + if test + PV+ = A / A + B Negative predictive value (PV-) = probability disease - if test PV- = D / C + D
Gold standard Disease + Disease Test +

TP FN

A B C D

FP TN

Vertically
Sensitivity (SE) = probability test + if disease + SE = A / A + C Specificity (SP) = probability test - if disease SP = D / B + D What numbers do you think are most useful in practice (PV+ and PV- or SE and SP)?
Test

Data-analysis
Perfect diagnostic test
False Positive = 0 False Negative = 0 e.g. Fever > 380C as predictor for BM Yes (+) Fever > 380C No (-)

BM+ BM- tot.

20 20 90 110 70 90

40 160 200

Data-analysis
reading 2 by 2 table
Horizontally
probability BM+ if fever+ = 20/110 = 18% PV+ = A / A + B probability BM - if fever- = 70/90 = 78% PV- = D / C + D
Gold standard BM+ BM

20
Fever +

90

Vertically
probability fever+ if BM+ = 20/40 = 50% SE = A / A + C probability fever- if BM- = 70/160 = 44% SP = D / B + D

Fever

TP FN
20

A B
C D

FP TN
70

What numbers do you think are most useful in practice (PV+ and PV- or SE and SP)?

Exercise 1
Mercury thermometer or timpanic membrane infrared meter

Exercise 1
Ad question 1
Research question: Can fever be determined with the TIM? Determinant: test under study = timpanic membrane infrared meter Outcome: fever determined with rectal mercury thermometer (RMT)

Domain: Children in secondary/tertiary care (ER hospital)

Exercise 1
Ad question 2
GS RMT Fever+ Fever

Se = probability TIM+ if RMT+ = 77/96 = 80 % SP = probability TIM- if RMT- = 108/117 = 92%

77
TIM >38

TIM 38

TP FN
19

A B
C D

FP TN
108

Exercise 1
Ad question 3
PV+ = probability RMT+ if TIM+ = 77/86 = 90 % PV- = probability RMT- if TIM- = 108/127 = 85%
GS RMT Fever+ Fever

77
TIM >38

TIM 38

TP FN
19

A B
C D

FP TN
108

Exercise 1
Ad question 4
The prior probability of fever in the general practice is lower, e.g. 20% (X/213=0,2 X=43) For similar Se and SP: GS RMT Fever+ Fever (A/43=0,8 A=34) (D/170=0,92 D=156) 34 14 48 TIM+ PV+ becomes lower (34/48 = 70%) TIM9 156 164 PV becomes higher (156/164 = 95%)
43 170 213

Exercise 1
Ad question 5
In the general practice an unjustly referred or treated child is less of a problem than an unjust reassurance of the parents Especially the negative predictive value must therefore be sufficiently high

Data-analysis: combination of determinants

In practice not one single diagnosis based on 1 test
Tests together distinguish ill/non-ill Method: statistical model

Moreover: diagnostic process is hierarchical

(simple --> invasive/expensive) --> always start with anamnesis model --> see case

Data-analysis
Case:
model with all anamnestic tests gender + age + fever + pain %BM = (gender, age, fever, pain)

Statistical model can be seen as 1 (composed) test

Quantify diagnostic value model with area under ROC curve (Receiver Operating Characteristic =Area Under Curve
(AUC))

Data-analysis
ROC Curve
1,00

,75

Sensitivity

,50

,25

0,00 0,00 ,25 ,50 ,75 1,00

1 - Specificity

Data-analysis
Case: AUC anamnesis model = 0,71
Informal interpretation AUC = % correctly diagnosed

The larger the ROC area the better the model

AUC range: 0,5 1,0
AUC = 0,5 bad (Se = 1- Sp diagonal [coin]) AUC > 0,7 reasonable AUC > 0,8 good AUC > 0,9 excellent AUC = 1,0 perfect (Se=100% & 1-Sp=0%)

Data-analysis
Quantify added value additional tests to previous tests
Extend previous model (follow order practice) Quantify change in AUC Case
Model 1 anamnesis model + physical exam (5 extra tests) --> AUC = 0,72 interpretation? Model 2 anamnesis model + 3 blood tests ---> AUC = 0,90 interpretation?

Data-analysis
ROC Curve
1,00

,75

Sensitivity

,50

,25

Coin flip Patient hisotry

0,00 0,00 ,25 ,50 ,75 1,00

Pat hist + test

1 - Specificity

Data-analysis
The AUC does not directly say anything about individual patients and is therefore not directly applicable

Reporting
Research question Study set-up
Research population, setting, determinants, outcome, design

Results
Predictive values (new) test and/or ROC curve ROC curve combination of tests Added value new test --> ROC curve

Exercise 2
Ad question 1
- Cross-sectional study in patients suspected of a stomach or duodenum ulcer
- For all patients anamnestic data were collected

- For all patients a gastroscopy was done

- Independent diagnostic value of anamnestic factors (determinant) for the diagnosis of ulcer (outcome: gastroscopy) were calculated

Exercise 2
Ad question 2
Adults with stomach complaints referred to a gastroenterology policlinic in a peripheral hospital

Exercise 2
Ad question 3 Score is 5, risk is 57%

Exercise 2
Ad question 4
- Everybody above the cut-off point has the same risk (and the same below the cut-off point) - Of course this is not true and the score loses precision - Preferably predictive values for score-categories and predictive values for more cut-off points

Exercise 2
Ad question 5
PV+ = 20/31 = 65% PV- = 64/69 = 93%
Test + Peptic ulcus +

20

11
A B C D

TP FN
Test -

FP TN
64

Exercise 2
Ad question 6
- Predictive values more favourable and therefore preferred - But it is not about the isolated predictive value but about the added diagnostic value given the results of the anamnestic score

Exercise 2
Ad question 7
Perform the anamnestic score and the breath test for a population from the domain. Subsequently perform the reference test (endoscopy) for everybody Compare the next determinant-outcome relations: P(ulcus) = (age, gender, anamnesis, ...) P(ulcus) = (age, gender, anamnesis, ..., breath test) Then compare the Receiver Operating Characteristic (ROC)-curve of the models

Exercise 2
Ad question 8 - Breath test partially contains the same information as the score - Suppose that the breath test is more often positive with age, then the breath test also measures age and therefore the added value is less than when the breath test would be completely independent of the score

Exercise 2
Ad question 9 - Preferably not, but if the assessor is informed of the data in the score in practice, than it should be the same in the study

Diagnostics Summary (1)

Diagnostics in practice
Uncertainty reduction Determines prognosis & determines policy

Diagnostic research Design

Observational Descriptive

Diagnostics Summary (2)

Cross-sectional Simultaneous measurement determinant and outcome (reference standard) Always study >1 determinant

Design
Assess determinants as in practice

Assess disease status & determinant status with double blinding

Diagnostics Summary (3)

Analysis
Univariate (per determinant) Multivariate: combination of test results in relation to outcome Endpoint = (combination of determinants) Determine added value; first analyse least invasive tests (as in practice)

Reporting
Mainly added value of test