Study Limitations

➲ Retrospective study
 Information bias
 Misclassification bias
➲ Uses uniform criteria for normal WBCs in all
age groups as > 10 cells/mcL
➲ 1:500 rule used in study, but has been
discredited by several studies
➲ Could be Herpes meningoencephalitis or Lyme
disease
➲ Both missed patients were young infants – the
patient sub-group we are most interested in..!
➲ Not useful if patient was pre-treated with Abx
EBM – Clinical Prediction Rules
or
Attempts to quantify Intuition..!

Jegen Kandasamy
PGY-2
Maimonides Medical Centre
 The Problem

➲ 22 day old male newborn, Tmax 101.2F at

home, URI Sx x 2d, otherwise well-appearing

➲ “Full Sepsis w/u” done at the ED, started on

Ampi/Cefotaxime, and admitted to A6

➲ Awaiting Cx results for CSF/Blood/Urine

➲ CSF Analysis
 Protein/Glucose: 70/56
 CSF red, turbid, 100 WBCs and 7500 RBCs per mcL
 85% N, 15% L
 No cells/organisms seen on gram stain
 The Costs

➲ “Awaiting cultures to turn out negative over 2

days”
➲ Typical hospital costs per patient for the 2 – 3
days of stay -
➲ About 1 million admissions like this happen
every year
➲ Social and emotional impact also need to be
appreciated

Vs

➲ Costs of “missing the diagnosis”

 Your Decision

In-patient Admission and Antibiotics

In-patient admission and Observation

Outpatient management with 1 dose of antibiotics

administered in the ED

Outpatient management – Wait and Watch

????????
 Clinical Dilemmas
➲ How likely is a disease ?
 Which patient is most likely to have an ankle fracture..?
 Which patient is most likely to have Strep Pharyngitis..?
 Which patient is most likely to have a head injury that warrants CT..?
 Which patient most likely has conjunctivitis that is bacterial..?

➲ Which test is most helpful ?

➲ What is a patient’s prognosis

 Ambiguity vs. Probability

➲ Ambiguity
 The language that physicians use to describe a patient’s
condition often is ambiguous
 Probable, highly likely, etc.

➲ Probability
 An alternative method of expressing uncertainty
 % or odds
 Probability estimates of various qualitative verbal
expressions
Certain

Likely

Possible

Probable
Low probability

Suggests

High probability
Unlikely

Moderate probability

Pathognomonic

Classic

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
The most important reason we order
diagnostic tests is to Refine Probability
 In the words of Kant...

+ Experience
Priori Test result

Pretest Posttest
Probability Probability

Posteriori
 The effect of test results on the
probability of disease

Pretest Posttest
Probability Probability

Perform Test

0.0 0.5 1.0

Probability of Disease
 Pretest Probability Estimates

➲ Traditional
 Clinical experience / judgment

➲ Objective:
 Prevalence studies
 Subgroup analyses
 Clinical prediction rules
 What is a CPR
➲ A tool that takes the experience of a group of
physicians with hundreds or even thousands of
patients and attempts to distill this experience
into a simple rule

➲ Quantifies contributions of
 History
 Physical Exam
 Basic laboratory tests

➲ Derived from systematic clinical observation

➲ Suggests a diagnostic/therapeutic course of

action, reduces unnecessary actions/costs
and/or maintains quality of care
In an ideal world...
 Making a CPR

Step 1. Derivation
Identification of factors with
predictive power

Step 2. Validation
Evidence of reproducible accuracy
Narrow Validation Broad Validation
Application of a rule Application of a rule
similar clinical setting in multiple clinical
setting and population settings with varying
Step 1 prevalence of disease

Step 3. Impact
Analysis
Evidence that rule changes
physician behavior and
improves patient outcomes
 CPRs Make Pretest Probability
Estimates More Reliable
➲ Often useful in situations with
 Complex decision making
 High clinical stakes
 Opportunity for cost savings w/o compromising
patient care
Methodologic standards -
Derivation
 Methodologic Standards -
Derivation

Were all important predictors included in

deriving the rule – YES

Were all important predictors present in a

significant proportion of the study population
- YES

Does the rule make clinical sense - YES

 Derivation of Rule

➲ Construct & define list of all potential

predictors and outcomes of interest
 Include biological, physiological, symptom
variables
 Items from history, PE, lab testing
 Can include treatment variable
 e.g., effectiveness of treatment, surgical mortality
 Derivation (cont.)
➲ Examine large enough group of patients
with all important predictors
 Determine if candidate clinical predictors are
present
 Determine patient status or outcome of interest
 Blinded to predictors present/absent
➲ Statistical analysis reveals most powerful
predictors
Methodologic standards-
Validation
 Accuracy Generalizability
(External Validity)

➲ Degree to which ➲ Ability of prognostic

predicted outcomes system to provide
match observed accurate predictions in
outcomes new patient sample
 Validation of Prediction Rule

➲ Determine if associations between

predictors and outcome due to chance
➲ Predictors may be idiosyncratic to
 Patient population
 Clinicians using rule
 Other aspects of study design, etc
➲ Test feasibility of implementation
 Results of Validation

➲ Interpret validity of prediction rule via

 Sensitivity, specificity, ROC
 Likelihood ratios, absolute or relative risks
 Depends on prevalence of outcome in population
 Calibration: Predicted vs. actual results
➲ Precision generated from confidence
intervals around quantitative results
 Statistical Validation
➲ Assesses if results due to chance
➲ Split-sample technique
 Divide sample in two
 Develop rule on half
 Test rule on other half
➲ Bootstrap technique
 Remove 1 patient from sample
 Generate rule with remainder of patients
 Test rule on removed patient
 Repeat for every patient
 Generalizability/External
Validity
➲ Reproducibility
 Rule is accurate in patients who were not
included in rule-development population but
come from an identical population
➲ Transportability
 Rule is accurate in patients drawn from a
different but related population or in data
collected differently
 Transportability
➲ Historical
 Accuracy maintained when rule tested at
different calendar times
➲ Geographic
 …when rule tested in data from different
locations
➲ Methodologic
 …when rule tested in data collected using
different methods
 Transportability (cont.)

➲ Spectrum
 Accuracy is maintained in patient sample that,
on average, has more or less advanced disease
or different disease trajectory
 Study Impact of Rule
➲ Time to use prediction rule must be “worth
it” and change clinician management or
behavior to
 Improve patient care
 Reduce costs with same quality of care
➲ Model is valuable if it divides patients into
meaningful groups
➲ Parsimony, not cumbersome
 Summary
➲ Validated prediction rules should be used
when
 Clinical decision is complex
 Clinical stakes are high
➲ Understand
 Patient population
 Clinical setting
 Purpose of the rule
➲ Look for rules that make a difference