You are on page 1of 132

Critical Appraisal

Tools
Payam Kabiri, MD. PhD.
Clinical Epidemiologist

Department of Epidemiology & Biostatistics


School of Public Health
Tehran University of Medical Sciences
What is critical appraisal?

 Critical appraisal is the assessment of


evidence by systematically reviewing its
relevance, validity and results to specific
situations.
Chambers, R. (1998).
Three Basic Questions …
Is it Valid?
Is the methodology appropriate to answer the question.
Is it carried out in a sound way, eliminating bias and
confounding?

Is it Reliable?
Are the results real or because of chance?

Is it Applicable?
Will the results help locally?
Key Steps to
Effective Critical Appraisal

1. What are the results


2. Are the results valid?
3. How will these results help me/my
colleagues do their job/decisions?
Critical Appraisal Tools
 Why do we need them?
 Where we can find them?
Critical Appraisal Tools
 CASP
 Center for Evidence Based Medicine
 International Centre for Allied Health
Evidence
 DISCERN
 AGREE
Reporting Guidelines used as a
Critical Appraisal Tool
 CONSORT checklist and flow diagram
 TREND checklist
 STARD checklist & flow diagram
 STROBE checklists
 PRISMA checklist and flow diagram
 COREQ checklist
 SQUIRE checklist
 REMARK checklist
 ENTREQ
CASP (http://www.casp-uk.net)
 The Critical Appraisal Skills Programme (CASP)
was developed in Oxford in 1993 and has over
the past years helped to develop an evidence
based approach in health and social care.
 The CASP appraisal tools are based on the
guides produced by the Evidence Based
Medicine Working Group, a group of clinicians at
McMaster university, Hamilton, Canada, and
colleagues across North America, published in
the Journal of the American Medical Association.
CASP (http://www.casp-uk.net)…
 Systematic Reviews
 Randomized Controlled Trials (RCTs)
 Qualitative Research
 Economic Evaluation Studies
 Cohort Studies
 Case Control Studies
 Diagnostic Test Studies
Center for Evidence Based
Medicine (CEBM)
 Aviabale at: http://www.cebm.net
 Offers Critical Appraisal Sheets
International Centre for Allied
Health Evidence
http://www.unisa.edu.au/cahe/resources/cat/
 Appraising randomised controlled trials
 Appraising non-randomised controlled trials
 Appraising other forms of quantitative research
 Appraising case studies
 Appraising qualitative research
 Appraising mixed methods research
 Appraising systematic reviews
 Appraising meta-analyses
 Appraising clinical guidelines
 Appraising outcome measures
 Assessing treatment choices
AGREE
 Appraisal of Guidelines Research and
Evaluation (AGREE)
 The AGREE Instrument is developed for
assessment of clinical practice guidelines
and is available online in several
languages at:
http://www.agreecollaboration.org
DISCERN
(http://www.discern.org.uk)
Quality criteria for consumer health information on
treatment choices
 DISCERN is a brief questionnaire which provides users
with a valid and reliable way of assessing the quality of
written information on treatment choices for a health
problem.
 DISCERN can also be used by authors and publishers of
information on treatment choices as a guide to the
standard which users are entitled to expect.
 New DISCERN Genetics site provides a reliable way of
assessing the quality of information on genetic testing
and screening.
Some other fields …
 Economic Evaluation Studies
 Diagnostic Test Studies
 Qualitative Studies
 Therapy
 Diagnosis
 Harm
 Prognosis
 Quantitative Research Studies
 Interventions Addressing the Need for education
 Library Research
 meta-analyses
Do they work?
 Katrak et al. systematic review of the
content of critical appraisal tools. BMC
Medical Research Methodology 2004

 Few critical appraisal tools had


documented evidence of validity of their
items, or reliability of use.
Appraisal Tools for
Observational
Studies
Types of Observational studies
 Cohort
 Case-control
 Cross-sectional
 Ecologic
 Case series
 Case report
STROBE Statement
 Strengthening the Reporting of
Observational studies in Epidemiology
 It is originally a reporting guideline.
 Many journals refer to the STROBE
Statement in their Instructions for Authors.
 Provides recommendation for each section
(22 items)
 Available at: www.strobe-statement.org
Available STROBE check-lists
 STROBE checklist for cohort, case-control,
and cross-sectional studies (combined)
 Checklist for cohort studies

 Checklist for case-control studies

 Checklist for cross-sectional studies


Title and abstract
(a) Indicate the study’s design with a
commonly used term in the title or the
abstract
(b) Provide in the abstract an informative
and balanced summary of what was done
and what was found
Introduction
 Background/rationale:
 Explain the scientific background and
rationale for the investigation being reported
 Objectives:
 State specific objectives, including any pre-
specified hypotheses
Methods
 Study design
 Present key elements of study design early in
the paper
 Setting
 Describe the setting, locations, and relevant
dates, including periods of recruitment,
exposure, follow-up, and data collection
Methods: participants
 Cohort study—Give the eligibility criteria, and the
sources and methods of selection of participants.
Describe methods of follow-up
 Case-control study—Give the eligibility criteria,
and the sources and methods of case
ascertainment and control selection. Give the
rationale for the choice of cases and controls
 Cross-sectional study—Give the eligibility
criteria, and the sources and methods of selection
of participants
Methods: matched studies
 Cohort study—For matched studies, give
matching criteria and number of exposed
and unexposed
 Case-control study—For matched studies,
give matching criteria and the number of
controls per case
Methods: Variables
 Clearly define all outcomes, exposures,
predictors, potential confounders, and
effect modifiers. Give diagnostic criteria, if
applicable
 Quantitative variables
 Explainhow quantitative variables were
handled in the analyses. If applicable,describe
which groupings were chosen and why
Methods: Data sources/
measurement
 For each variable of interest, give sources
of data and details of methods of
assessment (measurement).
 Describe comparability of assessment
methods if there is more than one group
Method: Bias & Study size
 Describe any efforts to address potential
sources of bias
 Explain how the study size was arrived at
Method :Statistical methods
 Describe all statistical methods, including those used to
control for confounding
 Describe any methods used to examine subgroups and
interactions
 Explain how missing data were addressed
 Cohort study—If applicable, explain how loss to follow-up was
addressed
 Case-control study—If applicable, explain how matching of
cases and controls was addressed
 Cross-sectional study—If applicable, describe analytical
methods taking account of sampling strategy
 Describe any sensitivity analyses
Results: Participants
 Report numbers of individuals at each
stage of study—eg numbers potentially
eligible, examined for eligibility, confirmed
eligible, included in the study, completing
follow-up, and analyzed
 Give reasons for non-participation at each
stage
 Consider use of a flow diagram
Results: Descriptive data
 characteristics of study participants (eg
demographic, clinical, social) and
information on exposures and potential
confounders
 number of participants with missing data
for each variable of interest
 Cohort study—Summarise follow-up time
(eg, average and total amount)
Results: Outcome data
 Cohort study—Report numbers of
outcome events or summary measures
over time
 Case-control study—Report numbers in
each exposure category, or summary
measures of exposure
 Cross-sectional study—Report numbers of
outcome events or summary measures
Main results and Other analyses
 Unadjusted estimates and, if applicable,
confounder-adjusted estimates and their precision
(eg, 95% confidence interval). Make clear which
confounders were adjusted for and why they were
included
 Report category boundaries when continuous
variables were categorized
 If relevant, consider translating estimates of relative
risk into absolute risk for a meaningful time period
 Report other analyses done—eg analyses of
subgroups and interactions, and sensitivity analyses
Discussion
 Key results: Summarise key results with reference
to study objectives
 Limitations: Discuss limitations of the study, taking
into account sources of potential bias or imprecision.
Discuss both direction and magnitude of any
potential bias
 Interpretation: Give a cautious overall interpretation
of results considering objectives, limitations,
multiplicity of analyses, results from similar studies,
and other relevant evidence
 Generalisability: Discuss the generalisability
(external validity) of the study results
Other information
 the source of funding and the role of the funders
for the present study and, if applicable, for the
original study on which the present article is
based
CASP: Cohort study

Public Health Resource Unit, Oxford


General comments

 Three broad issues need to be


considered when appraising a cohort
study:

 Are the results of the study valid?


 What are the results?
 Will the results help locally?
screening questions
 The first two questions are screening questions and
can be answered quickly. If the answer to those two is
"yes", it is worth proceeding with the remaining
questions.

1. Did the study address a clearly focused issue?

2. Did the authors use an appropriate method to


answer their question?
Validity: Detailed Questions
3. Was the cohort recruited in an acceptable way?
4. Was the exposure accurately measured to
minimize bias?
5. Was the outcome accurately measured to
minimize bias?
6. Have the authors identified all important
confounding factors? Have they taken account
of the confounding factors in the design and/or
analysis?
7. Was the follow up of subjects complete enough?
Was the follow up of subjects long enough?
What are the results?
8. What are the results of this study?

9. How precise are the results? How


precise is the estimate of the risk?

10. Do you believe the results?


Will the results help me locally?

11. Can the results be applied to the local


population?

12. Do the results of this study fit with


other available evidence?
Appraisal Tools for
Randomized Clinical
Trials
CONSORT Statement

 CONSORT stands for Consolidated Standards


of Reporting Trials.
 It is originally a reporting guidelines.
 It is developed by the CONSORT Group to
alleviate the problems arising from inadequate
reporting of randomized controlled trials (RCTs).
 The website: http://www.consort-statement.org
History of CONSORT
 CONSORT (Consolidated Standards of
 Reporting Trials) statement (In the mid
1990s)
 The revised CONSORT statement (1999,
2000)
CONSORT Statement
The CONSORT statement comprises:
a 22-item checklist pertain to the content of
the Title,
Abstract,
Introduction,
Methods,
Results,
Comment.
a flow diagram depicts information from 4 stages of a
trial enrollment,
intervention allocation,
follow-up,
analysis
Title and abstract
 How participants were allocated to
interventions (e.g., “random allocation,”
“randomized,”or “randomly assigned”).
Introduction: Background
 Scientific background and explanation of
rationale.
Method:
 Participants: Eligibility criteria for
participants and the settings and locations
where the data were collected.
 Interventions: Precise details of the
interventions intended for each group and
how and when they were actually
administered.
 Objectives: Specific objectives and
hypotheses.
Method:
 Outcomes: Clearly defined primary and
secondary outcome measures and, when
applicable, any methods used to enhance the
quality of measurements (e.g., multiple
observations, training of assessors).
 Sample size: How sample size was
determined and, when applicable,
explanation of any interim analyses and
stopping rules.
Method: Randomization
 Sequence generation: Method used to generate
the random allocation sequence, including details
of any restriction (e.g., blocking, stratification).
 Allocation concealment: Method used to
implement the random allocation sequence (e.g.,
numbered containers or central telephone),
clarifying whether the sequence was concealed
until interventions were assigned.
 Implementation: Who generated the allocation
sequence, who enrolled participants, and who
assigned participants to their groups.
Method:
 Blinding (masking): Whether or not participants,
those administering the interventions, and those
assessing the outcomes were blinded to group
assignment. If done, how the success of blinding
was evaluated.
 Statistical methods: Statistical methods used to
compare groups for primary outcome(s); methods
for additional analyses, such as subgroup
analyses and adjusted analyses.
Results
 Participant flow: Flow of participants through each
stage (a diagram is strongly recommended). Specifically,
for each group report the numbers of participants
randomly assigned, receiving intended treatment,
completing the study protocol, and analyzed for the
primary outcome. Describe protocol deviations from
study as planned, together with reasons.
 Recruitment: Dates defining the periods of recruitment
and follow-up.
 Baseline data: Baseline demographic and clinical
characteristics of each group.
Results
 Numbers analyzed: Number of participants
(denominator) in each group included in each
analysis and whether the analysis was by
“intention to treat.” State the results in absolute
numbers when feasible (e.g., 10 of 20, not 50%).
 Outcomes and estimation: For each primary and
secondary outcome, a summary of results for each
group and the estimated effect size and its
precision (e.g., 95% confidence interval).
Results
 Ancillary analyses: Address multiplicity
by reporting any other analyses
performed, including subgroup analyses
and adjusted analyses, indicating those
pre-specified and those exploratory.
 Adverse events: All important adverse
events or side effects in each intervention
group
Discussion
 Interpretation: Interpretation of the results,
taking into account study hypotheses,
sources of potential bias or imprecision, and
the dangers associated with multiplicity of
analyses and outcomes.
 Generalizability: Generalizability (external
validity) of the trial findings.
 Overall evidence: General interpretation of
the results in the context of current evidence.
CASP
Screening Questions
 1. Did the study ask a clearly-focused
question?

 2. Was this a randomised controlled trial


and was it appropriately so?
Detailed Questions
3. Were participants appropriately allocated to intervention and control
groups?
4. Were participants, staff and study personnel ‘blind’ to participants’
study group?
5. Were all of the participants who entered the trial accounted for at its
conclusion?
6. Were the participants in all groups followed up and data collected in
the same way?
7. Did the study have enough participants to minimize the play of
chance?
8. How are the results presented and what is the main result?
9. How precise are these results?
10. Were all important outcomes considered so the results can be
applied?
Appraisal Tools for
Diagnostic Tests
Which one of these test is the best
for SLE Dx?
Test Sensitivity% Specificity%
ANA 99 80
dsDNA 70 95
ssDNA 80 50
Histone 30-80 50
Nucleoprotein 58 50
Sm 25 99
RNP 50 87-94
PCNA 5 95
Diagnostic Tests
Characteristics:
 Sensitivity
 Specificity
 PredictiveValue
 Likelihood Ratio
Validity of Screening Tests
True Disease Status
+ -

+ a b

- c d

Sensitivity: The probability of testing


positive if the disease is truly present

Sensitivity = a / (a + c)
63
Validity of Screening Tests
True Disease Status
+ -

+ a b

- c d

Specificity: The probability of screening


negative if the disease is truly absent

Specificity = d / (b + d)
64
 Two-by-two tables can also be used for
calculating the false positive and false
negative rates.
 The false positive rate = false positives /
(false positives + true negatives). It is also
equal to 1- specificity.
 The false negative rate = false negatives /
(false negatives + true positives). It is also
equal to 1 – sensitivity.
 An ideal test maximizes both sensitivity
and specificity, thereby minimizing the
false positive and false negative rates.
Validity of Screening Tests
Breast Cancer
Physical Exam + -
and Mammo-
graphy + 132 983

- 45 63650

Sensitivity: a / (a + c)
Sensitivity =
Specificity: d / (b + d)
Specificity =
67
Validity of Screening Tests
Breast Cancer
Physical Exam + -
and Mammo-
graphy + 132 983

- 45 63650

Sensitivity: a / (a + c)
Sensitivity = 132 / (132 + 45) = 74.6%

Specificity: d / (b + d)
Specificity = 63650 / (983 + 63650) = 98.5%
68
Probability of Disease
 Pre-test probability of disease = disease
prevalence
 Post-test probability of disease =
 If normal, c/(c+d)
 If negative, a/(a+b)

69
Likelihood Ratios (Odds)
 This is an alternative way of describing the
performance of a diagnostic test. Similar to S
and S, and can be used to calculate the
probability of disease after a positive or
negative test (predictive value). Advantage of
this is that it can be used at multiple levels of
test results.
 The probability of a test result in those with
the disease divided by the probability of the
result in those without the disease. (how
many more times (or less) likely a test result
is to be found in the disease compared with
70 the nondiseased.
“Post-test probability” depends on
the accuracy of the diagnostic test
and the pre-test probability of
disease

A test result cannot be interpreted


without some knowledge of the
pre-test probability
Where does “pre-test
probability” come from?

 Clinicalexperience
 Epidemiological data
 “Clinical decision rules”
 Guess

72
 LR+= sensitivity/(1-
specificity)

 LR-
= (1-
sensitivity)/specificity

73
Switch to Odds
 1000 patients. 100 have disease. 900
healthy. Who will test positive?

 Diseased 100__X.95 =_95


Healthy 900 X.08 = 72
 We will end with 95+72= 167 positive tests
of which 95 will have the disease
 PPV = 95/167

74
From pretest to posttest odds
 Diseased 100 X.95 =_95
Healthy 900 X.08 = 72
 100 = Pretest odds
900
 .95 = Sensitivity__ = prob. Of pos test in dis
.08 1-Specificity prob. Of pos test in hlth
 95 =Posttest odds. Probability is 95/(95+72)
72
75
 Remember to switch back to probability

76
What is this second fraction?
 Likelihood Ratio Positive
 Multiplied by any patient’s pretest odds
gives you their posttest odds.
 Comparing LR+ of different tests is
comparing their ability to “rule in” a
diagnosis.
 As specificity increases LR+ increases
and PPV increases (Sp P In)

77
Clinical Interpretation:
Likelihood Ratios
 Likelihood ratio

 LR+ = Sensitivity/(1-Specificity)

 LR- = (1-Sensitivity)/Specificity
Likelihood Ratio Negative
 Diseased 100_ X.05 =_5__
Healthy 900 X.92 = 828
 100 = Pretest odds
900
 .05 = 1-sensitivity = prob. Of neg test in dis
.92 Specificity prob. Of neg test in hlth
(LR-)
 Posttest odds= 5/828. Probability=5/833=0.6%
 As sensitivity increases LR- decreases and NPV
increases (Sn N Out)
79
 Remember to switch to probability and
also to use 1 minus

80
Value of a diagnostic test depends
on the prior probability of disease

 Prevalence  Prevalence
(Probability) = 5% (Probability) = 90%
 Sensitivity = 90%  Sensitivity = 90%
 Specificity = 85%  Specificity = 85%
 PV+ = 24%
 PV+ = 98%
 PV- = 99%
 PV- = 49%
 Test not as useful
when disease unlikely  Test not as useful
when disease likely
81
Clinical interpretation of post-
test probability

P robability of dis eas e:


Don't Do further
treat for diagnostic Treat for
disease tes ting disease

0 1

Testing Treatment
threshold threshold

Disease Disease
ruled out ruled in
82
Samples of positive and
negative likelihood ratios

Poor-fair Good Excellent


Positive
likelihood 2.1-5 5.1-10 >10
ratio
Negative
likelihood 0.5-0.2 0.19-0.1 <0.1
ratio
Advantages of LRs
 The higher or lower the LR, the higher or lower
the post-test disease probability
 Which test will result in the highest post-test
probability in a given patient?
 The test with the largest LR+
 Which test will result in the lowest post-test
probability in a given patient?
 The test with the smallest LR-

84
Advantages of LRs
 Clear separation of test characteristics
from disease probability.

85
Likelihood Ratios - Advantage
 Provide a measure of a test’s ability to rule
in or rule out disease independent of
disease probability
 Test A LR+ > Test B LR+
 Test A PV+ > Test B PV+ always!
 Test A LR- < Test B LR-
 Test A PV- > Test B PV- always!

86
Using Likelihood Ratios to Determine Post-
Test Disease Probability

Pre-test Pre-test
probability odds of
of disease disease Post-tes t Post-tes t
odds of probability
disease of disease
Likelihood
ratio

87
88
Predictive Values
Alternate formulations:Bayes’ Theorem

PV+ =
Se  Pre-test Prevalence
Se  Pre-test Prevalence + (1 - Sp)  (1 - Pre-test Prevalence)

High specificity to “rule-in” disease

PV- =
Sp  (1 - Pre-test Prevalence)
Sp  (1 - Pre-test Prevalence) + (1 - Se)  Pre-test Prevalence

High sensitivity to “rule-out” disease

89
Clinical Interpretation: Predictive Values

PV+ And PV-1 Of Electrocardiographic Status2


For Angiographically Verified3 Coronary Artery
Disease, By Age And Sex Of Patient
Sex Age PV+ (%) PV- (%)

F <40 32 88
F 40-50 46 80
F 50+ 62 68

M <40 62 68
M 40-50 75 54
M 50+ 85 38

1. Based on statistical smoothing of results from 78 patients referred to NC


Memorial Hospital for chest pain. Each value has a standard error of 6-7%.
2. At least one millivolt horizontal st segment depression.
3. At least 50% stenosis in one or more main coronary vessels.

90
Clinical Interpretation: Predictive Values

91
If Predictive value is more
useful why not reported?
 Should they report it?
 Only if everyone is tested.
 And even then.
 You need sensitivity and specificity from
literature. Add YOUR OWN pretest
probability.

92
So how do you figure pretest
probability?
 Start with disease prevalence.
 Refine to local population.
 Refine to population you serve.
 Refine according to patient’s presentation.
 Add in results of history and exam (clinical
suspicion).
 Also consider your own threshold for testing.

93
Pretest Probability: Clinical
Significance
 Expected test result means more than
unexpected.
 Same clinical findings have different
meaning in different settings
(e.g.scheduled versus unscheduled visit).
Heart sound, tender area.
 Neurosurgeon.
 Lupus nephritis.

94
What proportion of all patients
will test positive?
 Diseased X sensitivity
+ Healthy X (1-specificity)
 Prevalence X sensitivity +
(1-prevalence)(1-specificity)
 We call this “test prevalence”
 i.e. prevalence according to the test.
Which one of these test is the best
for SLE Dx?
Test Sensitivity Specificity LR(+)
ANA 99 80 4.95
dsDNA 70 95 14
ssDNA 80 50 1.6
Histone 30-80 50 1.1
Nucleoprotein 58 50 1.16
Sm 25 99 25
RNP 50 87-94 3.8-8.3
PCNA 5 95 1
Was it clear enough !
Reliability

 Reliability
refers to the repeatability
or reproducibility of a test.

 Itcan be assessed by repeating the


test using the same or different
observers.
Validity
 Relates to whether the test measures what
it purports to measure. Is the result true?

 It can be assessed by comparing the


test results with a Gold Standard.
Likelihood Ratio
Likelihood of (+) test in diseased persons
LR Positive =
Likelihood of (+) test in healthy persons

Sensitivity
LR Positive =
1 - Specificity

Likelihood of (-) test in diseased persons


LR Negative=
Likelihood of (-) test in healthy persons

1 - Sensitivity
LR Negative=
Specificity
Continuous Measurements
Cutoff Value for Positive Test

Healthy
Diseased
Proportion

TN TP

FN FP

SBP
Continuous Measurements
Cutoff Value for Positive Test

Healthy
Diseased
Proportion

TN TP

FP

FN SBP
Continuous Measurements
Cutoff Value for Positive Test

Healthy
Diseased
Proportion

TN TP

FN

FP SBP
Standards for Reporting of
Diagnostic Accuracy
(STARD)
Improve the accuracy and completeness of
research reporting and allow readers to assess
the “potential for bias” in the study reported.

Always use:
 Flowchart or Diagram
 Checklist
FLOW CHART or Diagram
STARD checklist

Section & Topic Item


#
TITLE/ABS…./KEYW 1 Identify the article as a study of diagnostic
ORDS accuracy (recommend MeSH
Heading ‘sensitivity and specificity’).
INTRODUCT… 2 State the research questions or study
aims, such as estimating diagnostic accuracy or
comparing accuracy between tests or across
participant groups.
Section & Item
STARD checklist
Topic #

METHODS
Participants 3 Describe the study population: The inclusion and exclusion criteria, setting and locations
where the data were collected.
4 Describe participant recruitment: Was recruitment based on presenting symptoms, results
from previous tests, or the fact that the participants had received the index tests or the
reference standard?

5 Describe participant sampling: Was the study population a consecutive series of


participants defined by the selection criteria in items 3 and 4? If not, specify how participants
were further selected.

6 Describe data collection: Was data collection planned before the index test and
reference standard were performed (prospective study) or after (retrospective
study)?
Test methods 7 Describe the reference standard and its rationale.
8 Describe technical specifications of material and methods involved including how and
when measurements were taken, and/ or cite references for index tests and reference
standard.

9 Describe definition of and rationale for the units, cutoffs and/or categories of the results of the
index tests and the reference standard.

10 Describe the number,training and expertise of the persons executing and reading the
index tests and the reference standard.
11 reference standard were
Describe whether or not the readers of the index tests and
blind (masked) to the results of the other tests and describe any other
clinical information available to the readers.
STARD checklist (METHODS Continued….)

Section & Item


Topic #
METHODS
Statistical 12 Describe methods for calculating or comparing measures of diagnostic accuracy,
methods and
the statistical methods used to quantity uncertainly (e.g. 95% confidence
intervals).
13 Describe methods for calculating test reproducibility, if done.
STARD checklist
Section & Item
Topic #
RESULTS
Participants 14 Report when study was done, including beginning and ending dates of
recruitment.
15 Report clinical and demographic characteristics of the study
population (e.g. age,sex, spectrum of presenting symptoms, comorbidity,
current treatments, recruitment centers).
16 Report the number of participants satisfying the criteria for inclusion that did
or did not undergo the index tests and/or the reference standard; describe why
participants failed to receive either test ( a flow diagram is strongly
recommended).
Test results 17 Report time interval from the index tests to the reference standard, and any
treatment administered between.
18 Report distribution of severity of disease (define criteria) in those with the
target
condition; other diagnoses in participants without the target condition.
19 Report a cross tabulation of the results of the index tests (including
indeterminate and
missing results) by the results of the reference standard; for continuous
results, the distribution of the test results by the results of the reference
standard.
20 Report any adverse events from performing the index tests or the reference
standard.
Estimates 21 Report estimates of diagnostic accuracy and measures of
statistical uncertainty (e.g. 95% confidence intervals).
22 Report how indeterminate results, missing responses and outliers
of the index tests were handled.
23 Report estimates of variability of diagnostic accuracy between
subgroups of participants, readers or centers, if done.
24 Report estimates of test reproducibility , if done.
STARD checklist

DISCUS….. 25 Discuss the clinical applicability of the study findings.


CASP
Three broad issues

 Are the results of the study valid?


 What are the results?
 Will the results help me and my
patients/population?
Screening Questions
 Was there a clear question for the study to
address?
A question should include information about:
– the population
– the test
– the setting
– the outcomes
 Was there a comparison with an appropriate
reference standard?
 Is this reference test(s) the best available
indicator in the circumstances?
Detailed Questions
3. Did all patients get the diagnostic test and
the reference standard?
4. Could the results of the test of have been
influenced by the results of the reference
standard?
5. Is the disease status of the tested
population clearly described?
6. Were the methods for performing the test
described in sufficient detail?
what are the results?
7. What are the results?
8. How sure are we about these results?
Will the results help me and my
patients/population?
Consider whether you are primarily interested in
the impact on a population or individual level
9. Can the results be applied to your patients the
population of interest?
10. Can the test be applied to your patient or
population of interest?
11. Were all outcomes important to the individual or
population considered?
12. What would be the impact of using this test on
your patients/population?
Appraisal Tools for
Secondary Studies
Secondary Studies

 A secondary study does not generate any


data from direct measurements, instead, it
analyses a set of primary studies and
usually seeks to aggregate the results
from these in order to provide stronger
forms of evidence about a particular
phenomenon.
What is a systematic review?
 A review that has been prepared using
some kind of systematic approach to
minimising biases and random errors,
and that the components of the
approach will be documented in a
materials and methods section
Chalmers et al, 1995
What is a Meta-analysis?

 A statistical analysis of the results from


independent studies, which generally
aims to produce a single estimate of the
treatment effect Egger et al, 2001
What is a systematic review?

Reviews

Systematic reviews
Drawbacks to systematic
reviews/meta-analyses
 Can be done badly
2 systematic reviews on same topic can
have different conclusions
 Inappropriate aggregation of studies
 A meta-analysis is only as good as the
papers included
 Tend to look at ‘broad questions’ that
may not be immediately applicable to
individual patients
Some of the Appraising tools
Appraising systematic reviews
 Critical Appraisal Skills Program (CASP):
Systematic Reviews
 Systematic Review (of therapy) Worksheet
 ARIF (Aggressive Research Intelligence Facility)

Appraising meta-analyses
 QUOROM Statement Checklist
CASP
Screening Questions
1. Did the review ask a clearly-focused
question?
2. Did the review include the right type
of study?
– address the review’s question
– have an appropriate study design
Detailed Questions
 3. Did the reviewers try to identify all
relevant studies?
– which bibliographic databases were used
– if there was personal contact with experts
– if the reviewers searched for unpublished studies
– if the reviewers searched for non-English-
language
studies
Detailed Questions
4. Did the reviewers assess the quality of
the included studies?
– if a clear, pre-determined strategy was used
to
determine which studies were included.
– a scoring system
– more than one assessor
Detailed Questions
5. If the results of the studies have been
combined, was it reasonable to do so?
– the results of each study are clearly displayed
– the results were similar from study to study
(look for tests of heterogeneity)
– the reasons for any variations in results are
discussed
Detailed Questions
6. How are the results presented and
what is the main result?

– how the results are expressed (e.g. odds ratio,


relative risk, etc.)
– how large this size of result is and how meaningful
it is
– how you would sum up the bottom-line result of
the review in one sentence
Detailed Questions
7. How precise are these results?
8. Can the results be applied to the local
9. Were all important outcomes considered?
(individual, policy makers and professionals,
family/caregivers, wider community)
10. Should policy or practice change as a
result of the evidence contained in this
review? (whether any benefit reported outweighs any
harm and/or cost. If this information is not reported can it
be filled in from elsewhere? )
! ‫ بزنيد‬Email ‫اگر ميل داشتيد‬
kabiri@tums.ac.ir

You might also like