HOW TO APPRAISE A PAPER CRITICALLY

The science of trashing a paper

Unimport ant issue Badly written Unoriginal Hypothesis not tested Different type of study required Compromised original protocol Poor statistics Sample size too small

Conflict of interest
Unjustified conclusion

Why should critically Appraise a paper?

Published studies is of variable quality and not always reliable and relevant To improve clinical effectiveness, you need a systematic framework to interpret the studies You need to be sceptical. - Blind & Insightful scepticism Supports sound decision making based on best available evidence Helps us determine: - How rigorous a piece of study is What the results are telling us How relevant it is to our patient

1/25/2010

So, critically Appraise What You Read

What is best evidence?

You need to use evidence from research studies to support your arguments and decisions in caring for your patients One tool to assist you in this is critical appraisal Using critical appraisal skills you can understand the methods and results of a study and then be able to assess the quality of the study.

1/25/2010

The Hierarchy of Evidence

1. Systematic reviews & meta-analyses 2. Randomised controlled trials 3. Cohort studies 4. Case-control studies

5. Cross sectional surveys

6. Case reports

7. Expert opinion
8. Anecdote

The quality of a study depends on

Extrinsic factors: Journal Author Institution Quoted by other authors Quoted in newspapers
1/25/2010

Intrinsic factors: Study design Number of participants Confounders Author bias

Critical appraisal
6

Critical appraisal looks at the intrinsic factors

Critical appraisal is - the process of systematically examining research evidence to assess its validity, reliability and relevance before using it to inform a decision.( Hill H & Spittlehouse C, EBM 2001; 3:2) - the assessment of evidence by systematically reviewing its relevance, validity and results to specific situations. - Chambers, R. (1998).
1/25/2010 7

Key Steps to Effective Critical Appraisal

1. Are the results valid? ( Validity) 2. What are the results? ( Important) 3. How will these results be relevant to the patient? ( Applicability)

V.I.A
1/25/2010 8

Validity, reliability, relevance

Validity: To what extent is the study a close representation of the truth?

Reliability: Are the results credible and (important) repeatable? Relevance: Will the results help me in my own (applicability) study or practice?
1/25/2010 9

Validity and Reliability

A test is valid when it measures what its supposed to. If a test is reliable, it yields consistent results. A test can be both reliable and valid, one or the other, or neither. Reliability is a prerequisite for measurement validity.

1/25/2010

10

Reliable, but Not Valid!

1/25/2010

11

Not Reliable, Not Valid!

1/25/2010

12

Reliable and Valid

1/25/2010

13

1/25/2010

Mark Newman - Middlesex University updated 04/2001

14

Steps to EBM
1. Formulate an answerable focused clinical question - PICO model 2. Search the literature for the best external evidence 3. Critically appraise the evidence for its validity and usefulness/important 4. Implement the useful evidence in clinical practice 5. Evaluate the results
1/25/2010 15

Critical appraisal questions

What is the paper about? Why was the study done? What type of study was done? Was it primary research (experiment, RCT, cohort, case-control, crosssectional, longitudinal, case report /series)?

1/25/2010

16

Critical appraisal questions

Is it of interest? Title, abstract, source. Why was it done? Introduction.
Useful references? Important or novel ideas? Should end with a clear statement of the purpose of the study. The absence of such a statement can imply that the authors had no clear idea of what they were trying to find out. Or they didnt find anything but wanted to publish!

1/25/2010

17

Critical appraisal questions

How was it done? METHODS. Brief but should include enough detail to enable one to judge quality. Must include who was studied and how they were recruited. Basic demographics must be there. What data was collected? Have areas of potential bias been adequately addressed? Is sample size calculation provided? Are assumptions reasonable / referenced? Is study design appropriate to address hypotheses? Was the study ethical?
1/25/2010 18

Specific types of study

Critical appraisal questions

Is study design appropriate to address hypotheses?
Randomised Control Trials (RCTs) Studies in which there are two groups (one treatment one control). Patients are randomly assigned to both groups to reduce the risk of bias and increase the probability that differences between the groups can be attributed to the treatment. Factors that may lead to bias are called confounders. Cohort Studies Studies in which patients (e.g. a group of smokers) are followed over time and compared with a control group (non-smokers) to see what health problems they develop.

Critical appraisal questions

Is study design appropriate to address hypotheses?
Systematic reviews Comprehensive surveys of a topic locating, appraising and synthesising evidence from as many relevant scientific studies as possible. They summarise conclusions about effectiveness and provide a unique collation of the known evidence on a given topic. Meta Analyses Quantitative and statistical studies in which the results are similar enough statistically to be combined and analyzed as if it was one study. Case Control Studies Studies in which patients who already have a certain condition (e.g.lung cancer could be asked how much they smoked in the past), and the answers are compared with a sample of the general population

Critical appraisal questions

Is study design appropriate to address hypotheses?
Observational Studies Collective name given to studies (usually cohort or casecontrol) in which nature is allowed to take its course, seeking to find an association between one factor (e.g. being overweight) and another (e.g. dying from heart disease). Case Series Collections of reports on the treatment of individual patients, or of reports on a single patient no statistical validity because they do not use a control group with which to compare the outcomes
1/25/2010 22

Critical appraisal questions

What has it found? RESULTS. The data should be there not just statistics. Are the aims in the introduction addressed in the results? Are the results clearly presented? Are ALL the results presented? How have statistical results been presented? Are p-values or confidence intervals most appropriate? Are the results correct? Can the reader draw their own conclusions from the results?

1/25/2010

23

Critical appraisal questions

DISCUSSION What does it mean? Effect of methodological limitations Are the results put into context of previous literature? Is there any previous literature? Are similarities / differences with previous findings discussed? Is the impact of methodological limitations discussed? What does it mean for health care? Is it relevant to my patients?

1/25/2010

24

What Makes a study Good

Good research Appropriate study design Well executed procedures Appropriate analysis Adequate sample size . . . and more Discussion of failings Methodological imperfections Potential Bias / Confounding / etc Effect that these may have on results and conclusions
1/25/2010 25

Clinical Scenario
Can furosemide be used for rate control of atrial fibrillation (AF) in a patient with congestive heart failure (CHF)?

1/25/2010

26

Getting Started
1. Formulate research (PICO) question
Population: Patients with atrial fibrillation and CHF Intervention: furosemide Comparison: placebo Outcome: Mortality, effectiveness of rate control

2. Search appropriate sources

3. Find paper (s) 4. Read to suit you! 5. Appraise the paper (s)
1/25/2010 27

PICO Schematic Model

Treatment furosemide

Patient with Atrial Fibrillation in CHF ( P)

Comparison (C)

Effectiveness of rate control (O)

Placebo

1/25/2010

28

Clinical problem

General Search Strategy

Define the search question Try another relevant resource Choose a resource/database Create a search strategy Create a search strategy Summarize the evidence

Poor yield

Summarize the evidence

Apply the evidence

Try another relevant resource

Adapted from: Sackett, D. et al. 2000. Evidence-Based Medicine: How to Practice and Teach EBM. 2nd Edition. Toronto: Churchill Livingstone.
1/25/2010 29

Study Types for Question Types

Diagnosis Prospective cohort study with good quality validation against Gold Standard Randomized controlled clinical trial (RCT) RCT, cohort or case-control study (probably retrospective) Prospective cohort study

Therapy Etiology/ Harm Risk factors/ Prognosis

1/25/2010

30

How are the importance of the study ?

IMPORTANCE of the study Results How large was the treatment effect - Relative risk (RR) - Absolute risk reduction (ARR) - Relative risk reduction (RRR) - An odds ratio - Needed number to treat (NNT) How precise was the estimate of the treatment effect (relates to statistical significance)
1/25/2010 31

Absolute Risk Reduction

Consider a study in which
20% of a control group died but only 15% of the treatment group did

The absolute risk reduction would be:

20% - 15% = 5%

NOTE: An absolute risk reduction of 0% would indicate that there is no difference between the groups.
1/25/2010 32

Risk Ratios
The relative risk is the risk of events among intervention patients as a proportion of the risk among patients in the control group For the same study the relative risk would be:
15% / 20% = 75% or 0.75

Therefore the relative risk reduction would be:

100% - 75% = 25% or 0.25

NOTE: A relative risk of 100% or 1 would indicate that there is no difference between the groups
1/25/2010 33

Odds Ratios
Odds of an event = no. of events / no. of non-events e.g. 51 boys/100 births Odds of boy = 51/49 = 1.04 Odds >1 means event more likely to happen than not Odds of an impossibility are zero Odds of a certainty are infinity Odds ratio = odds in intervention group / odds in control group
12/30/2012 34

Odds Ratios
When events are rare Odds (OR) and Risk (RR) are similar

As prevalence (control event rate/CER) and OR increase, the error in using OR as an approximation for RR becomes unacceptable.

12/30/2012

35

Problem with relative risk (A)

Treatment Control Total
Died 20 30 50 Survived 30 20 50 Total 50 50 100 Risk of Dying 20 / 50 = 0.4 30 / 50 = 0.6

Absolute Risk Reduction = 0.6 - 0.4 = 0.2 = 20% Relative Risk = 0.4 / 0.6 = 0.67 = 67% Relative Risk Reduction = 1 0.67 = 0.33 = 33%

1/25/2010

36

Problem with relative risk (B)

Died Treatment 20 Control 30 Total 50 Survived 4980 4970 9950 Total 5000 5000 10000 Risk of Dying 20 / 4980 = 0.004 30 / 4970 = 0.006

Absolute Risk Reduction = 0.006 - 0.004 = 0.002 Relative Risk = 0.004 / 0.006 = 0.67 = 67% Relative Risk Reduction = 1 0.67 = 0.33 = 33% RRR is the same even though this drug is rubbish!
1/25/2010 37

Number Needed to Treat

Is based on absolute figures rather than relative Can be easier to understand The number of patients who must receive the intervention to produce one additional positive outcome NNT = 1 / Absolute Risk Reduction

For study A, NNT = 1 / 0.2 = 5

For study B, NNT = 1 / 0.002 = 500
1/25/2010 38

Applying the results to your patient

Estimate your patients risk relative to the average participant of the study, as a fraction f You may think that your patient is 50% more likely to die without therapy than the people in the study so f = 1.5 The NNT for your patient will be
NNT / f

For study A your patients NNT is:

5 / 1.5 = 3.3
1/25/2010 39

P value and Confidence Intervals

Two methods of assessing role of chance P values (testing the null hypothesis) If P < 0.05 then the difference between the 2 groups is statistically significant Confidence Intervals (CI) Estimate the range of values (of RR, ARR, RRR) and likely to include the true value If the value corresponding to no effect I.e. RR of 1 or ARR or RRR of 0, falls outside the 95% CI then the result is statistically significant.
1/25/2010 40

How the precise of the result?

p-values aims at measuring the precision of the

results The p-value is calculated using statistical tests that take into account the study design, the variation within the data, and the sample size. a p-value of less than 0.05 is sufficient to say that the observed difference indicates an actual difference or statistical significant.

1/25/2010

41

How much confidence do we have?

CONFIDENCE INTERVALS
telling how close your observed result is to the

actual result. CI can tell whether a result is significant ( statistical and clinical significant) but p-value only statistical significant) A 95% Confidence Interval of 0.1 2.0 tells that if you ran the trial again, there is a 95% chance getting a result somewhere between 0.1 and 2.0 Or if you conducted the trial 100 times you would get a result between 0.1 and 2.0 in 95 of those trials.
1/25/2010 42

Four possible outcomes from any study 1. Difference is clinically important and statistically significant i.e. important and real. 2. Of clinical importance but not statistically significant. sample size too small. 3. Statistically significant but not clinically important i.e. not clinically meaningful. 4. Neither clinically important nor statistically significant.

APPLICABILITY
Applicability of the study Discussion my patient so different from those in the study that the results cannot apply?
- Inclusion and exclusion criteria

Is the treatment feasible in my setting

- Cost - Availability

Will the potential benefits of treatment be outweighed by harms

1/25/2010 44

Case scenario
76 year old male admitted to your clinics due to acute diarrhea for one week. Medication such as kaopectine, imodium had been given but in vain After consultation with a internist, the probiotic was prescribed. Diarrhea improved after 2-3 days.

1/25/2010

45

1/25/2010

46

1/25/2010

47

EBM

1/25/2010

48

How can we do Critical Appraisal?

Use common sense Use simple checklists Use different checklists depending on the different types of studies (i.e., RCTS, systematic reviews etc) Checklists help you focus on the important parts of the article

1/25/2010

49

How to Read a Research Article?

Purpose of reading the paper. Do not read the abstract. Read the title, find out who the authors are and where they work, look for sources of funding and conflicts of interest Look at the tables and figures Is there a diagram to show the flow of participants through each stage of the study Why was the study done and what hypothesis were the authors testing? What is broadly the topic of research? Therapy, Diagnosis, Screening, Prognosis, Causation? What type of study was done? Is this a primary (experimental, clinical trial, survey) or a secondary paper (review, meta-analysis, guideline, economic analysis)? Was the study design appropriate? Now proceed to a critical appraisal of the paper
1/25/2010 50

TELAAH KRITIS JURNAL

12/30/2012

51

No

HAL YANG DINILAI

CHECK LIST PENILAIAN

YA

TIDAK

Judul Makalah

a. Apakah judul tidak terlalu panjang atau terlalu pendek? b. Apakah judul menggambarkan isi utama penilaian? c. Apakah judul cukup menarik? d. Apakah judul menggunakan singkatan selain yang baku?
a. Apakah merupakan abstrak satu paragraf, atau abstrak terstruktur? b. Apakah sudah tercakup komponen IMRAC (Introduction, methods, Results, Conclusion?) c. Apakah secara keseluruhan abstrak informatif? d. Apakah abstrak lebih dari 250 kata?

Terstruktur

Abstrak

Pendahuluan a. Apakah mengemukakan alasan dilakukannya 12/30/2012 penelitian?

52

Metode

a. Apakah disebutkan desain, tempat & waktu penelitian? b. Apakah disebutkan populasi sumber (populasi terjangkau)? c. Apakah kriteria pemilihan (inklusi & eksklusi) dijelaskan? d. Apakah cara pemilihan subjek (teknik sampling) disebutkan? e. Apakah perkiraan besar sampel disebutkan & disebut pula alasannya? f. Apakah perkiraan besar sampel dihitung dengant rumus yang sesuai? g. Apakah observasi, pengukuran, serta intervensi dirinci sehingga orang lain dapat mengulanginya? h. Bila teknik pengukuran tidak dirinci, apakah disebutkan rujukannya? i. Apakah definisi istilah & variabel penting dikemukakan? j. Apakah ethical clearance

53

12/30/2012

Hasil

a. Apakah disertakan tabel deskripsi subjek penelitian? b. Apakah karakteristik subjek yang penting (data awal) dibandingkan kesetaraannya? c. Apakah dilakukan uji hipotesis untuk kesetaraan ini? d. Apakah disebutkan jumlah subjek yang diteliti? e. Apakah dijelaskan subyek yang drop out dengan alasannya? f. Apakah semua hasil di dalam tabel disebutkan dalam naskah? g. Apakah semua outcome yang penting disebutkan dalam hasil? h. Apakah subyek yang drop out diikutkan dalam analisis? i. Apakah disertakan hasil uji statistik (x2,t) derajat kebebasan (degree of freedom), dan nilai p?

54

12/30/2012

Diskusi

a. Apakah semua hal yang relevan dibahas? b. Apakah dibahas keterbatasan penelitian, dan kemungkinan dampaknya terhadap hasil? c. Apakah disebutkan kesulitan penelitian, penyimpangan dari protokol, dan kemungkinan dampaknya terhadap hasil? d. Apakah pembahasan dilakukan dengan meghubungkannya dengan teori dan hasil penelitian terdahulu? e. Apakah dibahas hubungan hasil dengan praktek klinis? f. Apakah disertakan kesimpulan utama penelitian? g. Apakah kesimpulan didasarkan pada data penelitian? h. Apakah efek samping dikemukakan dan dibahas? i. Apakah disebutkan hasil tambahan selama diobservasi? j. Apakah disebutkan generalisasi hasil penelitian? k. Apakah disertakan saran penelitian selanjutnya, dengan anjuran metodologis yang tpat?

12/30/2012

55

KHUSUS - EBM Menilai VIA (Validity, Important, Aplicability) Validitas 1. Apakah awal penelitian didefinisikan dengan jelas? Ya, awal penelitian dan tujuan penelitian dijelaskan secara rinci. 2. Apakah dinyatakan desain penelitian dengan jelas ?

3. Apakah ada pembanding yang jelas ?

12/30/2012 56

4. Apakah pemantauan pasien dilakukan cukup panjang dan lengkap ?

5. Apakah ada identifikasi jelas kelompok dengan prognostik yang berbeda ?

6. Apakah outcome dinilai dengan kriteria objektif?

12/30/2012

57

Important 1. Apakah outcome/hasil dipaparkan secara jelas (hasil uji statistic dengan hasil nilai P)? 2. Seberapa besarkah ketepatan estimasi outcome yang didapat dengan nilai OR,RR,PR dengan nilai korelasi 95% CI ?

12/30/2012

58

Aplicability

1. Apakah pasien kita mirip dengan subjek yang diteliti?

2. Apakah bukti ini akan mempunyai pengaruh yang penting secara klinis terhadap kesembuhan pasien kita tentang apa yang telah ditawarkan/diberikan kepada pasien kita? 12/30/2012 59