Handbook Systematic Reviews

handbook
ABOUT
How to conduct systematic reviews of health policy and systems research in low- and middle-income countries
This is a tool intended to help researchers conducting systematic reviews about health systems issues in low- and
middle-income countries (LMICs). It is targeted to those researchers embarked in synthesizing evidence in this field, but
with a basic knowledge of systematic review methods. It provides a range of information, materials, resources and tools
based on current research on how to address the challenges found in the process of producing systematic reviews of
health policy and systems research. It is based on the experiences of research groups in LMICs conducting reviews
about human resources, financing in health systems, and the role of the non-state sector in health; and on
methodological developments made by leading research groups around the world.
The tool has been developed by the Health Policy and Systems Research Unit at the School of Medicine of the Pontificia
Universidad Catlica de Chile with funding from the Alliance for Health Policy and Systems Research. It is available as
either a Windows Help file or a platform-independent HTML version that can be viewed using a browser. Suggestions for
improvement are welcomed and should be addressed to tpantoja@med.puc.cl.
Release date
This version was released on January 31st 2011.
Commercial statement
The authors and funders are happy for this tool to be used at no charge in free educational programmes or by not for
profit organisations. However, when tuition is charged, or it is used by a for-profit organisation, our royalty fee is 10% of
gross receipts. If you are planning to use it within a for-profit organisation in any way, please contact us to discuss a
donation to our work.
Created with the Freeware Edition of HelpNDoc: Full featured Help generator
INTRODUCTION
This section contains the following:
The relevance of systematic reviews

What is a systematic review?
How the tool is organized?
The people behind this work
References
The relevance of systematic reviews

For many questions that health policy and decision makers might ask (e.g. What is the best way to extend financial
protection to those seeking health care? How have health workers responded to alternative incentive mechanisms?
Which strategies are most effective in terms of improving quality of care?) a substantial body of evidence exists, but this
evidence is often scattered and not available in a form that decision makers find easy to appraise or use. Systematic
reviews of health policy and systems research (HPSR) have the potential to reduce bias in the estimation of the effecs of
a policy option by identifying all relevant studies, selecting those that meet explicit criteria, appraising their quality, and
synthesizing the results using a transparent process.(1) Increasingly, systematic review methodologies have been also
applied to questions other than those about the effectiveness of interventions.(2;3) As a matter of fact, a range of
questions can be addressed, for example regarding the effects of interventions within the public health system or a
health care organization, how to bring about change in an organization, or how to improve accessibility or continuity of
care.(4-6)
While in principle systematic reviews have much to offer to the policy maker, in practice there are many
misunderstandings about what systematic reviews are and how they can make a difference in policymaking.(7) Besides,
although there is a vast literature in the field of systematic reviews and research synthesis, its development and use in
HPSR have been less extensive. Over the last two decades, scientists have established a reasonable consensus
regarding "best practice" for systematic reviews that inform clinical decision-making.(8-10) These methods rely heavily
on the use of hierarchies of evidence, with randomized controlled trials given the greatest weight in the case of reviews
assessing the effectiveness of interventions. However, in the field of HPSR, relatively limited evidence is generated
through randomized controlled trials. In such circumstances, systematic reviews must rely more heavily on quasi-
experimental, non-experimental and/or qualitative research. Although methods are available for synthesizing findings
from such studies, they are less well known and less well developed than those for synthesizing findings from
randomized studies.(11;12)
Therefore, this tool aims to help researchers conducting systematic reviews about health systems issues in low- and
middle-income countries (LMICs) in dealing with the challenges found in the production of such reviews.
What is a systematic review?

Although a variety of terms has been used to describe all or some of the processes involved in review and synthesis
research synthesis, literature review, systematic review, meta-analysis the term systematic review has become more
widely used since the late 1990s making reference to a structured process of review synthesis.(8;13) A systematic
review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific
research question. (9) Systematic reviews adhere to a strict scientific design based on explicit, pre-specified and
reproducible methods. As well as establishing what we know about a particular question, they can also demonstrate
where knowledge is lacking and in this way guide future research.(14) In contrast, a literature review is a more generic
term that refers to an account of what has been published on a topic by accredited scholars and researchers without
making specific reference to the methods used
One of the most known types of systematic review is the Cochrane review.(9) They are produced by the Cochrane
Collaboration, an international, not-for-profit and independent organization, dedicated to produce up-to-date, accurate
information about the effects of healthcare interventions readily available worldwide. Until now they have focused on the
effects of a broad range of healthcare interventions. However in the HPSR field, decision-makers must deal with complex
questions about diverse issues such as the nature and significance of the problem to be addressed, the nature of
proposed interventions, their differential impacts, their cost-effectiveness, and their acceptability. Although Cochrane
reviews in many of these cases could not be appropriate, the systematic reviews methods can still be useful as they
involve comprehensive evaluations of evidence including areas where research exists across all types of evidence. For
example, a systematic review could be conducted about the available qualitative research on a particular topic,(15) or a
combination of quantitative and qualitative research.(4;11)
Conducting a systematic review involves a number of stages extensively described in specialised textbooks and
guides.(8;9;14)They can be summarised as follows:
Formulating the review question. The problem to be addressed has to be specified in the form of a well-structured
question. All other aspects of the review follow directly from this question.
Identifying relevant literature. Comprehensive literature searches have to be conducted to identify potentially relevant
studies that can shed light on the question. This is an essential feature to make a review systematic. Although the
sources to be used will depend on the review question, current guidance suggests the use of at least electronic
databases (such as Medline, Embase and the Cochrane Central Register of Controlled Trials), reference lists of
included studies and contact with authors or experts in the specific field.
Including/excluding studies. A number of criteria to decide which studies should be included in the review need to be
specified. Those criteria in most cases have to be applied by at least two reviewers. Although some reviews
could be focused on the locally available evidence or be restricted to the languages with which the reviewers are
familiar, they should intend to be global in reach.
Assessing quality of the included studies. A judgment about the authors ability to minimize bias in the design,
conduct and analysis of included studies should be done. Reviewers can use a number of available tools for this
purpose.
Summarising the evidence. The findings of the studies included in the review should be summarised through the use
of different approaches (including statistical meta-analysis).
Interpreting the findings. Inferences for policy and practice could be generated by interpreting and exploring the
relevance of the findings.
As mentioned before, although the methods for synthesizing diverse sources of evidence are maybe less well known and
less used that those used in clinical research, our intention was to collect the wide range of methods available to deal
with challenges at the different stages of the conduct of a review.
(Back to top)
How the tool is organized?

The tool is organized in the following sections based on the stages of a typical systematic review:
Scoping the review

Question formulation
Searching for studies
Selecting and appraising the evidence
Synthesis methods
Presenting the results
Interpretation and relevance of the findings
Each section includes a brief introduction, things that the authors thought were relevant to be considered at each stage,
illustrative examples when appropriate, additional resources (documents, further reading, and links to potentially useful
websites), and references. Additionally we include a glossary section at the end where a definition of key terms is
presented.
While every effort has been made to keep valid, reliable and updated information, users are advised to consult the other
relevant sources for more detailed information about specific topics.
The people behind this work

The tool was prepared and is maintained by members of the Methodology Centre for Systematic Reviews of Health
Policy and Systems Research based at the Health Policy and Systems Research Unit of the School of Medicine of the
Pontificia Universidad Catlica de Chile. The challenges being addressed in each section were identified by a network of
Centres conducting systematic reviews in this area under the support and sponsorship of the Alliance for Health Policy
and Systems Research:
Centre for Systematic Reviews on the Non-state sector in health. Health Systems and Infectious Disease Division,
International Centre for Diarrhoeal Disease Research (ICDDR,B), Bangladesh
Centre for Systematic Reviews on Health Financing. Centre for Health Management and Policy, Shandong
University, China
Centre for Systematic Reviews on the Health Workforce. Institute of Public Health, Makerere University, Uganda.
The work of the Centres was supported and shaped by three northern collaborating partners:
The EPOC Satellite at the Norwegian Knowledge Centre for the Health Services, Oslo
The EPPI-Centre at the Institute of Education in London
The Effective Health Care Research Programme Consortium at the Liverpool School of Tropical Medicine in
Liverpool
(Back to top)
References.
(1) Lavis JN, Davies HTO, Oxman AD, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform
health care management and policy-making. J Health Serv Res Policy 2005;10(Suppl 1):S35-S48.
(2) Deeks JJ. Systematic reviews of evaluations of diagnostic and screening tests. BMJ 2001;323:157-62.
(3) Altman DG. Systematic reviews of evaluations of prognostic variables. BMJ 2001;323:224-8.
(4) Popay J. Moving beyond effectiveness in evidence synthesis. Methodological issues in the synthesis of diverse
sources of evidence. London: National Institute for Health and Clinical Excellence; 2006.
(5) Lavis JN, Davies HTO, Gruen RL, Walshe K, Farquhar CM. Working within and beyond the Cochrane
Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthcare
Policy 2006;1(2):21-33.
(6) Grimshaw JM, McAuley LM, Bero L, Grilli R, Oxman AD, Ramsay C, et al. Systematic reviews of the
effectiveness of quality improvement strategies and programmes. Qual Saf Health Care 2003;12:298-303.
(7) Petticrew M. Systematic reviews from astronomy to zoology: myths and misconceptions. BMJ 2001;322:98-101.
(8) Egger M, Davey Smith G, Altman DG. Systematic Reviews in Health Care. Meta-analysis in context. London:
BMJ Books; 2001.
(9) Cochrane Handbook for Systematic Reviews of Interventions. Chichester: Wiley-Blackwell; 2008.
(10) Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-Analysis. Chichester: Wiley; 2009.
(11) Pope C, Mays N, Popay J. Synthesizing qualitative and quantitative health evidence. A guide to methods.
Maidenhead: Open University Press; 2007.
(12) Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management
and policy-making in the health field. J Health Serv Res Policy 2005;10(Suppl 1):S6-S20.
(13) Mulrow CD. Rationale for systematic reviews. BMJ 1994 Sep 3;309(6954):597-9.
(14) Systematic Reviews. CRD's guidance for undertaking reviews in health care. York: Centre for Reviews and
Dissemination, University of York; 2008.
(15) Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res
Methodol 2009;9:59.
(Back to top)
Created with the Freeware Edition of HelpNDoc: Create HTML Help, DOC, PDF and print manuals from 1
single source
SCOPING THE REVIEW
Introduction
Things to consider
Reasons for conducting a scoping study
What are the elements of a scoping study
How to conduct a scoping study
Illustrative examples
Resources
References
Introduction
The complexity of the questions formulated and the types of evidence to be considered in the field of health policy and
system research (HPSR) go far beyond the range usually faced in the clinical field, spreading across a wide range of
disciplines and methodologies(1-3). In such circumstances, reviewers are confronted with the daunting task of identifying
and mapping a broad research domain. A number of terms have been used to label this process (e.g. mapping reviews,
scoping reviews), but scoping studies is the one has won acceptance in the specialized literature(4-6). Although there is
no universal definition for the term, it is considered as one method among many that could be used to review the
literature, with a focus on a comprehensive coverage (breadth) of the available literature underpinning a research or
thematic area. However, the degree of depth covered in a scoping study will depend on its purpose.
Things to consider
1. Reasons for conducting a scoping study
Arksey and OMalley have identified at least four common reasons why a scoping study might be undertaken(4): i) to
examine the extent, range and nature of research activity in a specific area; ii) to determine the value of undertaking a
full systematic review; iii) to summarize and disseminate research findings; iv) to identify research gaps in the existing
literature. These types of scoping studies seem to reflect two different ways of thinking about their purpose: on the one
hand the first two suggest the scoping study might be part of an ongoing process of reviewing with the ultimate aim of
producing a full systematic review. On the other hand, the latter two types suggest it is conceived as a method on its
own right, leading to the publication and dissemination of the findings in a particular field. In the field of research
synthesis in HPSR both approaches seem to be appealing and there are no good reasons to prefer one over the other.
However, it is important to clarify the purpose for conducting the process from the very beginning.
2. What are the elements of a scoping study
Even though scoping studies can contain different elements in order to achieve their different purposes, it is possible to
identify at least one of three major components: a literature map, a conceptual map, and/or a policy map.(2)
2.1 A literature map aims to provide an initial indication of the location of the literature regarding a particular issue
and to estimate its overall size. Its objective is to map out the literature as it stands, without any immediate plan to
review it systematically. This means plotting it out in time, space, source and origin. A good literature map spells out
the origins of work on the specific topic and gives a good feel for its chronological development. Besides it gives a good
account of where this work has been carried out and why, who has done this work and what was the order in which
each group became involved. The Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre)
has been working during the last decade in methodological approaches to the synthesis of evidence in public policy
issues where the mapping and classification of the literature after a systematic and exhaustive search plays a key role in
the review process.(7) They have also carried out a number of reviews where this approach has been used. (8)
2.2 A conceptual map generally explores the terminology in use in relation to a particular topic. It lists what the key
terms are along with any supplementary terms. It is designed to elicit how particular terms are used, by whom and for
what purpose. For instance, there may be differences in the meanings attached to words or phrases by different
disciplines as in the case of the concepts of adherence, compliance and concordance in relation to the ways in which
patients take their medication.(9)
2.3 A policy map identifies the main documents and statements from government agencies and professional bodies that
have a bearing on the nature of practice in a specific area. These policies include those which are disease specific
(diabetes, malaria, etc), those that are client group specific (children, older people), those that are specific to health
service staff groups (nurses, general practitioners) and those that are system wide (mental health, financing).
2.4 Although stakeholder consultations do not constitute an element of a scoping study in their own, they have an
important role to play in identifying research priorities, in helping to target research questions, and in validating the
outcomes of scoping studies through peer- and merit-reviews.
3. How to conduct a scoping study
In an effort to provide guidance to authors undertaking scoping studies Arksey and O'Malley developed a useful six-
stage methodological framework:
i) Identifying the research question

ii) Identifying relevant studies
iii) Study selection
iv) Charting the data
v) Collating, summarizing, and reporting results
vi) Consultation
It has been recently used by other authors in order to clarify and enhance each of its stages.(6) An overview of this
framework is presented in the following Table.
Table. Overview of the Arksey and OMalley methodological framework for conducting a scoping study (adapted from (6))
Stage Description
Identifying the research question Identifying the research question provides the
roadmap for subsequent stages. Relevant
aspects of the question must be clearly defined
as they have ramifications for search strategies.
Research questions are broad in nature as they
seek to provide breadth of coverage.
Identifying relevant studies This stage involves identifying the relevant
studies and developing a decision plan for where
to search, which terms to use, which sources are
to be searched, time span, and language.
Comprehensiveness and breadth is important in
the search. Sources include electronic
databases, reference lists, hand searching of key
journals, and organizations and conferences.
Breadth is important; however, practicalities of
the search are as well. Time, budget and
personnel resources are potential limiting factors
and decisions need to be made upfront about
how these will impact the search.
Study selection Study selection involves post hoc inclusion and
exclusion criteria. These criteria are based on the
specifics of the research question and on new
familiarity with the subject matter through reading
the studies.
Charting the data A data-charting form is developed and used to
extract data from each study. A 'narrative review'
or 'descriptive analytical' method is used to
extract contextual or process oriented information
from each study.
Collating, summarizing, and reporting An analytic framework or thematic construction is
results used to provide an overview of the breadth of the
literature but not a synthesis. A numerical
analysis of the extent and nature of studies using
tables and charts is presented. A thematic
analysis is then presented. Clarity and
consistency are required when reporting results.
Consultation (optional) Provides opportunities for consumer and
stakeholder involvement to suggest additional
references and provide insights beyond those in
the literature.
(Back to top)
Illustrative example.
A scoping review to improve conceptual clarity of interprofessional interventions(10).

The objective of the study was to develop a theoretically based and empirically tested
understanding of interprofessional education (IPE) and interprofessional collaboration (IPC).
Given the lack of clear terms and definitions for interprofessional interventions, authors
employed two main search processes. First, they drew upon a database of over 100 IPE
studies from two previous reviews. Second, they conducted a broad search of Medline.
A total of 104 studies were included for analysis. Studies were examined for their approach to
conceptualization, implementation, and assessment of their interprofessional interventions.
Half of the studies were used for interprofessional framework development and half for
framework testing and refinement. The final framework contains three main types of
interprofessional interventions: IPE; interprofessional practice; and interprofessional
organization; and describes the nature of each type of intervention by stage, participants,
intervention type, interprofessional objectives, and outcomes. The outcomes are delineated as
intermediate, patient, and system outcomes. There was very limited use of theory in the
studies, and thus theoretical aspects could not be incorporated into the framework.
Authors concluded that this study offers an initial step in mapping out the interprofessional
field and outlines possible ways forward for future research and practice.
(Back to top)
Resources.
References.
(1) Pantoja T, Campbell S. Systematic Reviews in Health Policy and Systems Research. Briefing Note 4. 2009.
Alliance for Health Policy and Systems Research.
(2) Anderson S, Allen P, Peckham S, Goodwin N. Asking the right questions: scoping studies in the commissioning
of research on the organisation and delivery of health services. Health Res Policy Syst 2008;6:7.
(3) Remme JH, Adam T, Becerra-Posada F, D'Arcangues C, Devlin M, Gardner C, et al. Defining research to
improve health systems. PLoS Med 2010;7(11):e1001000.
(4) Arksey H, O'Malley L. Scoping Studies: Towards a Methodological Framework. Int J Social Research
Methodology 2005;8(1):19-32.
(5) Davis K, Drey N, Gould D. What are scoping studies? A review of the nursing literature. Int J Nurs Stud 2009
Oct;46(10):1386-400.
(6) Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci 2010;5:69.
(7) Oliver S, Harden A, Rees R, Shepherd J, Brunton G, Garcia J, et al. An emerging framework for including
different types of evidence in systematic reviews for public policy. Evaluation 2005;11(4):428-46.
(8) Rees R, Kavanagh J, Harden A, Shepherd J, Brunton G, Oliver S, et al. Young people and physical activity: a
systematic review matching their views to effective interventions. Health Educ Res 2006 Dec;21(6):806-25.
(9) Horne R, Weinman J, Barber N, Elliot R, Morgan M. Concordance, adherence and compliance in medicine
taking: a scoping exercise. London: NCCSDO; 2005.
(10) Reeves S, Goldman J, Gilbert J, Tepper J, Silver I, Suter E, et al. A scoping review to improve conceptual clarity
of interprofessional interventions. J Interprof Care 2010 Dec 23.
(Back to top)
Created with the Freeware Edition of HelpNDoc: Easy to use tool to create HTML Help files and Help web
sites
QUESTION FORMULATION
Introduction
Things to consider
Dealing whith complexity
The lumping versus splitting debate
Defining eligibility criteria for the review
What study designs should be included in the review
The involvement of other stakeholders in the review process
Resources
References
Introduction
Question formulation is, probably, the most important step in the planning of a systematic review(SR). The review
question will guide the next steps of the review process, including the specification of inclusion criteria, the terms used in
the searching for studies, which data to collect, and the relevant findings to be presented.(1;2)
A typical clinical question can be framed specifying the types of population (participants), types of interventions (and
comparisons), and the types of outcomes that are of interest. The acronym PICO (Participants, Interventions,
Comparisons and Outcomes) helps as a reminder of these.(1;2) Although this framework is a useful starting point, its
application to the Health Policy & System Research (HPSR) field is not always straightforward. Most of the difficulties are
related to the complexity inherent to SR of HPSR. This complexity might be because of multi-component interventions,
diverse study populations, multiplicity of outcomes, mixed study designs utilized and/or the effect of context on
intervention design, implementation and effectiveness. (3)
(Back to top)
Things to consider
1 Dealing with complexity
Over the last two decades, the application of systematic reviews has extended to the evaluation of ever more complex
interventions. The UK Medical Research Council defines complex interventions as those comprising a number of
separate elements which seem essential to the proper functioning of the interventions although the active ingredient of
the intervention that is effective is difficult to specify.(4;5) While most health care interventions have some degree of
complexity, those commonly assessed in the HPSR field are at the most complex end of the spectrum. Examples include
continuity of care and discharge planning, both of which aim to minimize the fragmentation of health care.(6;7)A range of
facets of the complexity of these interventions has been identified. Likewise in the context of systematically reviewing
complex interventions a number of sources of complexity should be identified and considered early in the review
process:
1.1 Characteristics of the intervention

In many cases the interventions are intrinsically complex (multifaceted) or they are a heterogeneous mix of effective and
ineffective components. There are different levels of interdependency between components of multifaceted interventions.
From no interdependency to interventions composed of a combination of interventions where the effect of one
intervention is dependent upon the presence of a contemporaneous action or upon the presence of a preceding action.
The importance of interactions and interdependencies depends on the heterogeneity of interventions across studies,
whether meta-analysis is planned, and the purpose of the review.(3) A multifaceted intervention could be treated as a
relatively simple intervention if the studies included in the review assess identical multifaceted interventions and the
purpose of the review is to estimate the effectiveness of the intervention as a whole and it is assumed that all individual
actions are effective or where the reviewer does not wish to disentangle effective from ineffective actions.
1.2 Contextual factors

The effectiveness of an intervention could be modified by patient, provider and/or institutional factors. For instance, the
effectiveness of the intervention may be modified by the context in which it operates, as in the case of mass media HIV
awareness campaigns and their impact in different jurisdictions. Therefore, information on context within reviews can be
of considerable value to people who are in charge of implementing interventions in the real world.
1.3 Multiple and diverse outcomes

Studies in this field frequently report a variety of outcomes. When there is aconsistency in the way outcomes are
reported across studies it is possible to pool data and report results of meta-analyses for each of the main outcomes
separately. However, this is not always the case, and in many cases there are inconsistencies in the main outcomes
reported, with a variety of outcomes reported in different units.
Often there are multiple sources of complexity in any individual review. Then, reviewers have to decide if those issues
are sufficiently important to be addressed in the review.(8) This requires careful thought at the time of formulating the
review question and writing the review protocol.
2 The lumping versus splitting debate
A key issue faced by reviewers in this area is deciding how broad the scope of the review should be. The rationale for
taking a broad approach (lumping) is that, because systematic reviews aim to identify the common generalisable
features within similar interventions, minor differences in study characteristics may not be crucially important. On the
other hand, the rationale for taking a narrower approach (splitting) is that it is only appropriate to include studies which
are very similar in design, study population, intervention characteristics, and outcome recording.(3)
From the methodological point of view, a lumping approach is preferred in the literature(3;9). Broad systematic reviews
allow the generalisability and consistency of research findings to be assessed across a wider range of different settings,
study populations, and variation of the intervention. This reduces the risk of bias or chance results. Very narrowly
focused reviews are, in effect, subgroup analyses and could suffer all the well recognized potential hazards of such
analyses(10). A more transparent approach could be to lump together all similar interventions and then to carry out
explicit a priori subgroup analyses(3).
(Back to top)
3 Defining eligibility criteria for the review
One of the features that distinguish a systematic from a non-systematic review is the pre-specification of criteria for
including and excluding studies in the review (eligibility criteria). Eligibility criteria are a combination of different aspects
of the review question plus specification of the type of studies that have addressed that question.(2) The participants,
interventions and comparators components of the review question usually translate directly into eligibility criteria. The
criteria for considering types of participants included in studies in a review should be sufficiently broad to encompass the
likely diversity of studies, but sufficiently narrow to ensure that a meaningful answer can be obtained when studies are
considered in aggregate. Sometimes it could be difficult to define the participants in a HPSR review because the subject
of the intervention is a health systems or other group of participants. In that case broad definitions such as health
systems in LMICs or health professional working in underserved areas could be used. Regarding interventions it is not
always straightforward to make a precise definition of them because of their complexity in this area. However, efforts
should be done to obtain an operational definition describing the content (the different ingredients), variation and ways
of delivery of the intervention. Although outcomes are usually not part of the criteria defined for including studies,
reviewers could restrict eligibility to specific outcomes according to the relevance for answering the review question.
4 What study designs should be included in the review
The field of HPSR is characterized by a high degree of methodological pluralism including the use of randomized
controlled trials (RCTs), controlled before and after studies (CBA), uncontrolled studies, interrupted time series (ITS)
designs, surveys and qualitative studies, among others.(11) The study designs to be included in a review of a health
systems issue should be guided by the review question (methodological appropriateness). Although RCTs should be
prioritized where possible for inclusion in systematic reviews of effectiveness, when it is unlikely to find them, reviewers
may choose to include quasi-experimental designs such as interrupted time series. Comparisons with historical controls
or national trends should only be included when this is the sole type of evidence available (e.g. in reviews investigating
the effectiveness of policies) and accompanied by an acknowledgement such evidence is necessarily weaker.
Qualitative research has an important role in systematic reviews to answer questions that go beyond effectiveness,
including appropriateness of interventions to participants and the factors that promote and/or impede the implementation
of the intervention. The answers to these questions are seldom found in quantitative studies of effectiveness. Evidence
on the factors that impinge on the implementation of interventions may be particularly important in the context of complex
multifaceted public health interventions. Moreover, qualitative research can also contribute to the review process in
assisting to identify or refine the questions to be addressed in a systematic review.
5 The involvement of other stakeholders in the review process
Because of the multidisciplinary nature of the HPSR field, reviews will be more relevant and valid if they are informed by
people with a range of experiences, in terms of both the topic and the methodology. Choosing which interventions,
settings/populations, and outcomes to include in a review relies on knowledge of current policy, practice and the views of
the people targeted by the interventions. Therefore, reviewers should consider the incorporation of the views from
different stakeholders in a systematic way.
A number of methods can be used to engage members of the public, including surveys, interviews, focus groups,
requests for submissions, open houses, and public meetings.(REF) One approach that is becoming increasingly
accepted is the use of Advisory Groups whose members are familiar with the topic area. It is usually used to include the
perspectives of policy makers, funders, practitioners and potential users/recipients. The tasks of the Advisory Group may
include(12):
making and refining decisions about the scope of the review, i.e. which interventions, populations (including
subgroups) and outcomes will be included in the review. This decision should be based on the intended use of
results, the complexity of the review, and the time available;
providing important background material that elucidates the issues from different perspectives;
helping to interpret the findings of the review; and
disseminating the review to relevant groups.
Independently of the methods used to involve relevant stakeholders, reviewers should establish a mechanism for making
it happens considering the time and resources available.
(Back to top)
Resources
References.
(1) Systematic Reviews. CRD's guidance for undertaking reviews in health care. York: Centre for Reviews and
Dissemination, University of York; 2008.
(2) O'Connor D, Green S, Higgins JPT. Defining the review question and developing criteria for including studies. In:
Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions.Chichester: Wiley-
Blackwell; 2008. p. 83-94.
(3) Grimshaw JM, McAuley LM, Bero L, Grilli R, Oxman AD, Ramsay C, et al. Systematic reviews of the
effectiveness of quality improvement strategies and programmes. Qual Saf Health Care 2003;12:298-303.
(4) Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter DJ, et al. Framework for design
and evaluation of complex interventions to improve health. BMJ 2000;321:694-7.
(5) Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex
interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655.
(6) Shepperd S, Doll H, Broad J, Gladman J, Iliffe S, Langhorne P, et al. Early discharge hospital at home.
Cochrane Database Syst Rev 2009;(1):CD000356.
(7) Crooks VA, Agarwal G. What are the roles involved in establishing and maintaining informational continuity of
care within family practice? A systematic review. BMC Fam Pract 2008;9:65.
(8) Shepperd S, Lewin S, Straus S, Clarke M, Eccles MP, Fitzpatrick R, et al. Can we systematically review studies
that evaluate complex interventions? PLoS Med 2009 Aug;6(8):e1000086.
(9) Gotzsche PC. Why we need a broad perspective on meta-analysis. It may be crucially important for patients.
BMJ 2000 Sep 9;321(7261):585-6.
(10) Sun X, Briel M, Walter SD, Guyatt GH. Is a subgroup effect believable? Updating criteria to evaluate the
credibility of subgroup analyses. BMJ 2010;340:c117.
(11) Eccles MP, Grimshaw JM, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of
change and improvement strategies. Qual Saf Health Care 2003;12:47-52.
(12) Jackson N, Waters E. Criteria for the systematic review of health promotion and public health interventions.
Health Promot Int 2005 Dec;20(4):367-74.
(Back to top)
Created with the Freeware Edition of HelpNDoc: Easy to use tool to create HTML Help files and Help web
sites
SEARCHING FOR STUDIES
Introduction
Things to consider
Where to find the studies?
How many of these sources should be used?
How to search for studies?
Developing, organizing and reporting a search strategy
Additional considerations
Resources
References
Introduction
Comprehensiveness of the search is one of the key elements that distinguish systematic reviews from traditional reviews.
Their capacity of providing a synthesis of the best available evidence will rely fundamentally on the appropriate planning
of this step.
One of the main characteristics of an optimal search strategy is its ability to avoid bias. We will say bias in this context
each time the sample of studies presented in the review is not representative of the universe of existing studies (for a
reason not attributable to chance). One of the main types of bias to be avoided in systematic reviews is publication bias,
that is, to obtain a sample of studies that is not representative of the universe because only the published studies (and
not the unpublished ones) were found. Even though the inclusion of unpublished literature was controversial some time
ago (based on the theoretical assumption that the quality of published was better than that of unpublished trials), there is
extensive literature showing publication bias may result in an overestimate of effects, since positive studies (those that
show the intervention works) are more frequently published. A similar problem appears if only English literature or
databases are searched for (e.g. searching only in MEDLINE which over represents English literature). (1-3)
As previously mentioned in the section about question formulation, in HPSR reviews it is important to consider non
randomized studies, which are much more difficult to locate. For instance, the Cochrane Collaboration maintains the
CENTRAL database of studies (mainly RCTs) collected during the process of elaboration of systematic reviews, through
manual search of selected journals, among other initiatives. There is no specific wide-scale initiative to provide easy
access to non randomized or HPSR relevant studies.
Additionally, the HPSR field has many particularities that makes search difficult:
Large number of sources: Due to the complexity of the topic, and the interrelation with other areas of knowledge (e.g.
social sciences), the information is widespread across many different databases. For instance, in an evaluation of all
the electronic resources employed in a sample of Cochrane reviews on HPSR interventions, 133 different resources
were identified, with an average of 7.3 by review,(4) against an average of 4 in the whole Cochrane library.(5) Many
of these databases are less familiar to researchers accustomed to clinical reviews, and sometimes less structured
and more difficult to use.
Lack of standardization in terminology: Even though this is also a problem in clinical research, it is more
accentuated in HPSR. Terminology is particularly heterogeneous in the way of describing complex or multifaceted
interventions. An issue that is also particularly troublesome is the terminology used for some study designs different
from RCTs (e.g. interrupted time series) which is not used consistently by researchers and database managers.
Lesser rate of publication in journals: The research community involved in clinical research has an enormous
institutional and academic pressure to publish their results in biomedical journals. This is not necessarily the case for
HPSR, where an emphasis in dissemination through websites or other channels may be more common.
(Back to top)
Things to consider
1. Where to find the studies?
Before providing a brief synthesis of the different sources and the way to look into them, there are a couple of concepts
frequently used in this process that require further clarification. Firstly, the term grey literature is frequently employed to
refer to literature that is not yet published, but it is accessible through some form (sometimes it is used in a wider sense,
including all literature that is not published in biomedical journals). Some examples are conference proceedings, and
thesis. However, the line between published and unpublished is not always clear, and there are many initiatives that tend
to make this distinction even more confuse (e.g. prepublication, research repositories, websites listing research,
publication of proceedings, etc). We think that the term grey literature is not well-defined, and any effort to arrive to a
definition will be challenged by this rapidly evolving area. In the same vein, the distinction between published and
unpublished is also difficult to establish. However and considering that the classical accounts of synthesis methods and
the description of bias (i.e. publication bias) use this nomenclature, we will use this latter distinction highlighting that
there is a continuum where different sources have different proportions of published, grey and unpublished literature. For
instance, MEDLINE and EMBASE index only published literature, but CENTRAL and many other electronic databases
include conference proceedings and other sources. Secondly, many terms frequently used in this area such as
database, electronic database, major database, search engine, interface are blurry and overlapping and we will try to
use a broad definition of them.
The potential sources to be used in order to find relevant studies for responding the review question are presented
below. We did not intend to get into a deep conceptual discussion about each resource but to provide some guidance
about the existing options, and the best way to use them.
1.1. Electronic databases
There are hundreds of resources relevant to HPSR that are usually classified under this category.
The list includes general databases (multidisciplinary coverage), health-related databases (some focused in health policy
and systems) and databases of topics related to HPSR (e.g. social sciences, economics). In some specific topics, other
non-health databases may be useful (e.g. education, transport research). Some databases have a specialty restriction
for example, the Cumulative Index of Nursing and Allied Health- (CINAHL); the Physiotherapy Evidence Database
(PEDro); and AIDSLINE. Others have a geographic restriction, for instance, the Latin American Caribbean Health
Sciences Literature (LILACS), the Australasian Medical Index and the Chinese Biomedical Literature Database.
The magnitude of the contribution of each resource to systematic reviews of HPSR has not been evaluated, so the
decision of which resources to include will depend on the question of the review. In this ( link ) there is a list of some
electronic resources that reviewers may consider to incorporate and the frequency of use in Cochrane reviews of HPSR.
(4)In the developing a search strategy section we provide some examples of relevant resources for different types of
questions.
Even though their contribution to systematic reviews of HPSR has not been formally evaluated, it is reasonable to
include at least MEDLINE, EMBASE and CENTRAL, which cover a large number of areas. MEDLINE and EMBASE are
traditional databases indexing mostly articles published in biomedical journals. Considering the huge amount of
information they contain, a search strategy must be carefully devised. Additional information about MEDLINE and
EMBASE and their relevance for systematic reviews can be found in the Cochrane Handbook .(6)
The Cochrane Central Register of Controlled Trials (CENTRAL) is now the best single source of information about
records of studies that might be eligible for inclusion in Cochrane reviews. Unfortunately, HPSR reviews, usually need to
consider other designs, which are not included, or poorly represented in CENTRAL.
(Back to top)
1.2. Reference lists of other studies and reviews
The vast majority of times, the question of a systematic review have been addressed previously by other researchers.
So, it is reasonable to see if they have found some relevant studies, what can be easily scanned in the reference list.
Other systematic reviews might be particularly helpful, but non-systematic reviews may also be considered. The
reference list of the studies included in the review might be a good source of additional trials too. Other articles, as
guidelines, statements or letters, can be of help in order to address some specific questions, but they are rarely
considered in systematic reviews.
As the number of systematic reviews increases it becomes almost certain that some other review will exist on the same topic, or at
least on a related topic. It is very important to have access to these, in order to incorporate the relevant studies already identified,
to evaluate the sources not searched by previous reviewers and to build upon the search methodology. Nowadays, this is a step
rarely taken seriously by clinical reviewers, since it is perceived that existing methods are enough to capture all the relevant
evidence (which is debatable). However, in HPSR where the challenges to locating the relevant literature are so important, this step
may prove very fruitful.
Places where to look for reviews are presented below in the Resources section.
1.3. Conference and proceedings
It is well known that many studies are first published in conferences, and some of them never get published. So,
searching in these is important in order to avoid publication bias. However, reviewers must ponder carefully this
alternative, since there are a great number of potentially relevant meetings, and many times it is difficult to access this
information.
1.4. Dissertation and thesis
The rationality and difficulties are similar to those related to conference and proceedings.
1.5. Contact with experts and authors
A simple e-mail to identified authors and experts can provide a very good way of retrieving additional trials. A list of
experts can be obtained from an informal search of traditional databases or through personal contacts. If some networks
already exist (e.g. email lists, facebook groups, twitter groups, ning wikis, etc) they can be used with this purpose.
1.6. Other search engines
General search engines, or other health care search engines are sometimes used, especially when addressing a topic
not well covered in electronic databases. Some examples are Google, google scholar, TRIP database, SUM search,
Excelencia Clnica , etc.
1.7. Relevant web sites
A list of relevant web sites can be obtained through contact with experts, browsing the internet or using social networks.
1.8. Handsearching
This term is used for the manual page by page revision of a journal (in paper or online). It might be considered if some
relevant journal is not included in other resources. It may be also used in order to identify articles missed by the
electronic sources. Considering the amount of available information and resources, this strategy is not routinely used by
reviewers.
1.9. Ongoing studies
Identifying ongoing studies may also be important. Many prospective registries exist, and the WHO international Clinical
Trials Registry Platform provides a single interface to access many of them . Unfortunately, prospective registers include
mainly randomized trials, so they will be of little use to identify other study designs.
(Back to top)
2. How many of these sources should be used?
As we mentioned above, in systematic reviews evaluating interventions of health policy and systems, there are many
reasons why information not published in journals, or definitively unpublished must be searched in order to obtain an
unbiased answer to the review question. There is an obvious tension between searching in all existing sources of
published and unpublished literature, and being able to conduct the review in a given timeframe, and with a finite amount
of funding. It is then of cardinal importance to carefully select where, how and how deep will reviewers look for studies.
In our experience, special efforts should be done when conducting reviews addressing: i) Non-randomized evidence; ii)
Questions that have a non-health component (e.g. social sciences, economics, education); and iii) Questions addressed
primarily in LMIC or non-English speaking countries.
2.1. Two illustrative examples
A systematic review evaluating ambulatory versus conventional methods for monitoring blood pressure during
pregnancy(REF)
This is considered a health systems evidence question, since it has to do with a specific health care delivery
arrangement (where care is provided). However, this topic is usually addressed by academic researchers and
published in medical journals. On the other hand, there is a good chance of finding randomized trials on the topic,
measuring all the relevant outcomes (short term and easy to measure), which will provide an unbiased source of
evidence. The vast majority of reviewers will feel comfortable with focusing only in RCTs. In this review, there is not a
strong case in favor of performing a broad search including many resources. Even though it is always desirable to
search in all possible resources, there will be little added value in relation with the effort and cost involved.
A systematic review evaluating different educational strategies to avoid road injuries in low income countries (REF)
This is also a health systems question about delivery arrangements (where and with which technologies health care is
provided). It is very difficult to carry on a randomized trial for such interventions, and when possible, it is not always
possible to measure the relevant outcomes (total road injuries to drivers and pedestrians), or there is a high cost
involved. There are good reasons to look for non randomized trials in this case. On the other hand, these
interventions may be very context specific, so the identification of trials conducted in low income countries and in non-
English language becomes more important. Additionally, this question can be answered by biomedical researchers,
but also by educators, traffic and technology researchers. Moreover, much of the information may have been
commissioned by non academic institutions (e.g. governmental agencies). There are strong reasons to include as
many sources as possible in this review.
(Back to top)
How to search for studies?
Search strategies include different phases (set of terms to represent a concept). For instance, a typical review will have a
phase for defining the intervention of interest, another one to define the population, and a third one to restrict the results
to some specific study design (e.g. RCT).
2.2 Illustrative example

Components (phases) of a search strategy
In a hypothetical review on the effects of telemedicine in primary care, it would be reasonable to include at least a set
of terms to define the intervention (terms 1,2,3), a set for the population or setting (5,6,7) and some terms for the study
design (9,10,11.). The line 12 combines the 3 components.
1- Computerized advice
2- Telemedicine
3- Distant
4- 1 OR 2 OR 3
5- Primary care
6- Ambulatory care
7- Preventive services
8- 5 OR 6 OR 7
9- Randomized
10- Clinical trial
11- Double-blind
12- 9 OR 10 OR 11
13- 4 AND 8 AND 12
(Back to top)
3 Developing, organizing and reporting a search strategy
Developing, organizing and reporting a search strategy for a HPSR review question is in almost every aspect similar to
what is done in clinical reviews. The Cochrane Handbook provides an excellent guidance about this issue.
4 Additional considerations
The search strategy for studies of Health Policy and Systems in Low- and Middle-income countries should consider 3
additional components.
i) Health policy and systems as a topic: Including some terms in order to capture the concept of HPSR is a
tempting approach. However, given the amplitude of the term, it is difficult to think of a one size fits all strategy.
On the other hand, different definitions of this area exists, what makes even more difficult to create such
strategies. In PubMed, there are some hedges for topics that may be used to capture the HPSR concept.
ii) Study designs: The decision about the study design to be included varies depending on the question. The
Cochrane Handbook 6.4.11.1. gives a detailed explanation of the strategy to retrieve randomized trials. However,
in HPSR reviews this design is frequently not enough (see section about question formulation). The Cochrane
Effective Practice and Organisation of Care Group is developing a strategy to include also interrupted time series
and controlled before and after studies. A hedge for qualitative studies is also available in PubMed.
iii) Low and middle income countries: A strategy to restrict the search to specific countries has been used in some
reviews. As far as we know, there is no published evaluation of this approach. The main drawback in doing this is
the uncertainty about the quality of report and indexing of this data in electronic databases.
Including an additional component in the search strategy, may be an effective way of reducing the number of references,
and consequently the associated workload. However, it poses the risk of missing relevant articles. None of the strategies
presented have been extensively evaluated in this area, so it is difficult to anticipate the pros and cons of these
approaches. Reviewers should decide in an individual basis.
(Back to top)
Resources
Where to look for reviews
A number of sources could be used to locate systematic reviews:

In electronic databases using the appropriate filters or hedges for systematic review design (See some hedges for
the main databases)
In databases of systematic reviews as the Cochrane library .
In databases of summaries of systematic reviews, as DARE
In databases of HPSR as Health Systems Evidence
The Rx for Change database. summarizes current research evidence about the effects of strategies to improve drug
prescribing practice and drug use. This database houses summaries of key findings from systematic reviews that
evaluate the effects of strategies targeting professionals, the organization of health care, and consumers.
The health-evidence.ca team regularly searches for, screens, and rates systematic and narrative reviews, and
meta-analyses from a variety of electronic and print sources in order to support evidence-informed decision making
in public health by providing current research evidence in a searchable online registry
In epistemonikos: a new collaborative and multilingual database that compiles the different evidence resources. One
of the particularities of this database is that all the related systematic reviews and their included studies are
connected in the database. So, it is only necessary to find one article about the topic in order to follow the links that
lead to the whole body of related evidence.
References.
(1) Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or
direction of trial results. Cochrane Database Syst Rev 2009;(1):MR000006.
(2) Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta-analyses of randomized trials of health care interventions.
Cochrane Database Syst Rev 2007;(2):MR000010.
(3) Scherer RW, Langenberg P, von EE. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev
2007;(2):MR000005.
(4) Rada G, Neumann I, Herrera C, Manrquez M JJ, Pantoja T. Where do systematic reviews in Health Policy and Systems
Research search for studies? Singapore 2009.
(5) Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews.
PLoS Med 2007 Mar 27;4(3):e78.
(Back to top)
Created with the Freeware Edition of HelpNDoc: Easily create CHM Help documents
SELECTION & APPRAISAL
Introduction
Things to consider
Selecting the studies to be included in the review
Which data to collect
The data extraction form
Assessing the quality of included studies
Resources
References
Introduction
After searching for relevant studies to answer the question of interest, the next step in a SR is to select and extract
relevant information from all studies found and appraise them. In fact, once the search has been run, the researcher
team could be in front of an important volume (pile) of studies. The appropriate selection of studies to be included in the
review and the process of deciding which information should be extracted from those studies need to be planned
carefully to facilitate the next steps of the review.
An additional key step at this moment of the review process is related to the appraisal of the evidence found. In fact, the
judgments about the methodological quality of the included studies will be the basis for judging the confidence that
readers could put on the findings.
(Back to top)
Things to consider
1. Selecting the studies to be included in the review
As in any systematic review, selection of studies should be done according to inclusion and exclusion criteria, as defined
by the review question (see question formulation). Accepting the Cochrane Collaboration standards, this process should
be done by two independent reviewers examining the abstracts and/ or full texts of studies to decide inclusion. Detailed
description of this process is presented in Chapter 7 of the Cochrane Handbook for Systematic Reviews of
Interventions.(1) In the case of non- randomized clinical trials, such as CBA and ITS studies, some authors have
suggested methodology considerations to decide their inclusion in a review, in order to decrease the risk of bias of these
studies. Resources where to find more information about those methodological issues are presented below ( see
selection & appraisal ).
Another challenging issue in selecting studies is the case of reviews assessing complex interventions. Frequently these
interventions are not clearly defined/delimited and their active ingredients could be not completely clear, making difficult
to decide if a specific study is evaluating the intervention of interest to the review. Shepperd et al(2) have proposed
several alternatives for dealing with this problem: i) refining the definition of an intervention through an iterative process
to accommodate previously unseen configurations; ii) contacting study authors for further information; iii) recording the
components of an intervention during data extraction; and iv) being explicit in the review about where disagreement
occurred.
2. Which data to collect
According to the objectives of the review, researchers need to define previously which relevant data to collect, taking into
consideration relevant information that could be useful to both address the review objective, and discuss the applicability
of the results. In fact, as it will be discussed later (See Interpreting the findings), applicability of HPSR issues need to
consider a number of aspects related with the context, human resources, and health system characteristics of where the
primary studies were carried out and of where the results are planned to be used.(3)
Relevant information to be collected from primary studies in HPSR reviews will be related at least with: i) general
information of the studies: country, setting, detailed description of the intervention and comparator, population
characteristics, and Health System description; ii) results of the studies: this could include numerical data of any type of
the effect size of the intervention, number of participants, number of withdrawals; and iii) research methods: details that
allow risk of bias assessment.
(Back to top)
3. The data extraction form
Data extraction could be facilitated by using a pre-design form to collect the relevant information of studies included in
the review. This form could facilitate posterior steps of the review, such as report of the results, analysis and discussion.
It should be easy to fill and ideally in an electronic format. It is recommended to pilot test the designed form with a
representative number of studies before start using it.
The following information should be included in the data extraction form:

Identification of the study
Checklist of inclusion/ exclusion criteria to decide inclusion of study in the review
All relevant information referred above about general information of the study, including context information,
description of population, detailed description of the intervention and its components.
Results reported in the study
Quality assessment of the study, including the tool used to assess it
Detailed considerations about the data form could be consulted in Chapter 7 of the Cochrane Handbook for Systematic
Reviews of Interventions(1) and in the website of the EPOC and Consumer and Communication Groups of the Cochrane
Collaboration (See Resources below).
(Back to top)
4. Assessing the quality of included studies
In the context of a systematic review, the quality of a study refers to methodological quality or the extent to which
methods decrease risk of systematic error or bias in the results. The Cochranes recommendation is to use an individual
quality item approach rather than scores since the specific weight of each item is different and varies depending of the
type of question being addressed.(4)
There are several tools to assess the methodological quality of included studies. The method selected will depend
primarily upon the study design, with different tools for different types of studies. A first distinction could be made
according to the main methodological approach used in the study: quantitative or qualitative. For quantitative
approaches an additional useful distinction is between randomized and non-randomized studies. Table 1 describes the
main items to be considered in quantitative studies.
Table 1. Items used in appraising quality in quantitative studies
Quality Items to assess in an RCT
Randomization
Allocation Concealment
Blinding
Incomplete outcome data
Selective outcome reporting
Baseline comparability
Follow-up
Intention to treat analysis
Quality Items to assess in non-RCTs
Allocation concealment
Blinding
Baseline comparability
Follow-up
Intention to treat analysis
Detailed description and criteria to be applied in each of these items are presented in the chapters 8 and 13 of the
Cochrane Handbook for Systematic Reviews and in the EPOC webpage (See Resources below).
The assessment of the methodological quality of qualitative studies is more controversial.(5) A number of approaches
and tools have been described and the interested reviewer could obtain additional information from the website of the
Cochrane Qualitative Research Methods Group ( link ).
(Back to top)
Resources
Websites
Cochrane Effective Practice and Organisation of Care (EPOC) Group

The Cochrane Effective Practice and Organisation of Care (EPOC) Group is a Review Group of The Cochrane
Collaboration an international network of people helping healthcare providers, policy makers, patients, their advocates
and carers, make well-informed decisions about human health care by preparing and publishing systematic reviews
(SRs). The research focus of the EPOC Group are interventions designed to improve the delivery, practice, and
organisation of health care services. The EPOC editorial base is located in Ottawa, Canada with satellite centres
in Norway, Australia, and England. A number of resources related to the use on non-randomized studies could be found
in its website, including tools for assessing methodological quality (risk of bias) and templates of data extraction forms.
Cochrane Consumers and Communication group

The Cochrane Consumers and Communication Review Group coordinates the preparation and production of systematic
reviews of interventions which affect consumers' interactions with healthcare professionals, services and researchers.
The interventions may relate, for example, to individual use of healthcare services, or to consumer participation in health
planning, policy and research. A number of resources related to non-randomised and qualitative studies could be found
in its website, including tools for assessing methodological quality (risk of bias) and templates of data extraction forms.
Cochrane Qualitative Research Methods Group

The Cochrane Qualitative Research Methods Group develops and supports methodological work on the inclusion in
systematic reviews of findings from studies using qualitative methods and disseminates this work within and beyond the
Collaboration's Review Groups. The Cochrane Qualitative Methods Group focuses on methodological matters arising
from the inclusion of findings from qualitative studies into systematic reviews. A number of resources for assessing the
quality of qualitative studies can be found in its website.
EppiCentre Methods and Tools pages

The Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) is part of the Social Science
Research Unit at the Institute of Education, University of London. Since 1993, they have been at the forefront of carrying
out systematic reviews and developing review methods in social science and public policy. The Centre is dedicated to
making reliable research findings accessible to the people who need them, whether they are making policy, practice or
personal decisions. The EPPI-Centre offers support and expertise to those undertaking systematic reviews. The
methods pages provide a brief description of the methods used at each stage of a systematic review. The Tools page
provides information and links to various data coding and management tools and guidelines.
Further reading
Cochrane Handbook for Systematic Reviews

The Cochrane Handbook for Systematic Reviews of Interventions (the Handbook) provides guidance to authors for the
preparation of Cochrane Intervention reviews (including Cochrane Overviews of reviews). The Handbook is updated
regularly to reflect advances in systematic review methodology and in response to feedback from users. In chapter 7, 8
and 13 it is possible to found detailed guidance about selecting studies, collecting data, and assessing the
methodological quality of studies included in a review.
Systematic Reviews: CRDs guidance for undertaking reviews in health care

A publication of the United Kingdom Centre for Reviews and Dissemination providing practical guidance for those new to
evidence synthesis and a thorough overview of systematic review methods for the more experienced researcher. A
useful general resource where reviewers can found detailed guidance about selecting studies, collecting data, and
assessing the methodological quality of studies included in a review.
(Back to top)
References.
(1) Higgins JPT, Deeks JJ. Selecting studies and collecting data. In: Higgins JPT, Green S, editors. Cochrane Handbook for
Systematic Reviews of Interventions.Chichester: Wiley-Blackwell; 2008. p. 151-85.
(2) Shepperd S, Lewin S, Straus S, Clarke M, Eccles MP, Fitzpatrick R, et al. Can we systematically review studies that evaluate
complex interventions? PLoS Med 2009 Aug;6(8):e1000086.
(3) Lavis JN, Oxman AD, Souza NM, Lewin S, Gruen RL, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking
(STP) 9: Assessing the applicability of the findings of a systematic review. Health Res Policy Syst 2009;7 Suppl 1:S9.
(5) Dixon-Woods M, Shaw RL, Agarwal S, Smith JA. The problem of appraising qualitative research. Qual Saf Health Care
2004;13:223-5.
(Back to top)
single source
SYNTHESIS METHODS
Introduction
Things to consider
What are the available methods for synthesis?
Quantitative approaches
Meta-analysis
Range of effect sizes
Vote-counting
Narrative summary
Qualitative approaches
Meta-ethnography
Thematic synthesis
Narrative synthesis
Mixed approaches
Realist synthesis
The EPPI-Centre Approach
How to choose a specific synthesis approach?
Resources
References
Introduction.
Once relevant studies addressing the review question have been located, selected and appraised they should be in
some way summarized in order to explore and balance the evidence related to the original question. The synthesis
process attempts to integrate the findings from empirical research in order to create some form of generalization.
However, because of the diversity of questions (beyond effectiveness) and study designs (beyond randomized controlled
trials) commonly found in HPSR, different approaches could be used to summarise the available evidence.
(Back to top)
Things to consider.
1. What are the available methods for synthesis?
Methods for the quantitative synthesis of numerical data have been around for the last 35 years and their mathematical
foundations were described a century ago.(1) However, the synthesis of diverse types of evidence seems to have a
relatively shorter history.(2) Even though it may be an emerging area of research there are already a number of
apparently different methods for synthesis available. In 2005 Dixon-Woods et al outlined eleven possible approaches to
syntheses involving qualitative and quantitative evidence,(3) and there are yet more methods described in the
specialised literature.(4)
Therefore rather than attempt to outline all the possible methods currently available for synthesis and following the
framework presented by Pope et al(5) we have organized the description of available methods in three broad areas
according to their primary objective, and the type of studies and data primarily used by the specific synthesis method. In
the quantitative approaches section we describe methods that use numerical data or transform evidence into numbers
to enable different types of statistical or logical analyses. In the qualitative approaches section we describe methods
that use text base data or transform other evidence into this form in order to generate conceptual and theoretical
interpretations or explanations of a body of evidence. Finally in the mixed approaches section we describe methods that
combine the findings of multiple studies that are labeled broadly as using qualitative or quantitative methods. Our
selection of methods is based mainly on i) how useful we think they may be in the HPSR field; ii) the availability of
specific guidance about the method; and iii) the availability of examples illustrating the use of the specific method.
(Back to top)
1.1 Quantitative approaches.

1.1.1. Meta-analysis
It refers to the statistical synthesis of the data from primary studies, where the weights assigned to each study are based
on mathematical criteria that are specified in advance. The formulas used in meta-analysis are extensions of formulas
used in primary studies, and intend to address similar kinds of questions to those addressed in primary studies. Although
the use of meta-analysis has increased in the context of studies assessing the effects of interventions (such as the
randomized controlled trial), this approach may be conducted for a variety of reasons and purposes.(6)
More detailed guidance about how to conduct this kind of synthesis could be found at the websites of the Cochrane
Collaboration and the UK Centre for Reviews and Dissemination.
The effectiveness of lay health workers in primary and community health care(7)
Lewin et al assessed the effects of LHW interventions in primary and community health care on health care
behaviours, patients health and wellbeing, and patients satisfaction with care. A lay health worker was defined as
any health worker carrying out functions related to health care delivery; trained in some way in the context of the
intervention; and having no formal professional or paraprofessional certificated or degreed tertiary education
They searched the Cochrane Effective Practice and Organisation of Care and Consumers and Communication
specialised registers (to August 2001); the Cochrane Central Register of Controlled Trials (to August 2001);
MEDLINE (1966- August 2001); EMBASE (1966-August 2001); Science Citations (to August 2001); CINAHL (1966-
June 2001); Healthstar (1975-2000); AMED(1966-August 2001); the Leeds Health Education Effectiveness
Database and the reference lists of articles.
They found 43 studies that met the inclusion criteria. These showed considerable diversity in the targeted health
issue and the aims, content and outcomes of interventions. Most were conducted in high income countries (n=35),
but nearly half of these focused on low income and minority populations (n=15). Study diversity limited meta-analysis
to outcomes for five subgroups (n=15 studies) (LHW interventions to promote the uptake of breast cancer screening,
immunisation and breastfeeding promotion [before two weeks and between two weeks and six months post partum]
and to improve diagnosis and treatment for selected infectious diseases). Promising benefits in comparison with
usual care were shown for LHW interventions to promote immunization uptake in children and adults (RR=1.30 [95%
CI 1.14, 1.48] p=0.0001) and LHW interventions to improve outcomes for selected infectious diseases (RR=0.74
[95% CI 0.58, 0.93) p=0.01). LHWs also appear promising for breastfeeding promotion. They appear to have a small
effect in promoting breast cancer screening uptake when compared with usual care. For the remaining subgroups
(n=29 studies), the outcomes were too diverse to allow statistical pooling.
Authors concluded that LHWs show promising benefits in promoting immunisation uptake and improving outcomes
for acute respiratory infections and malaria, when compared to usual care. For other health issues, evidence is
insufficient to justify recommendations for policy and practice. There is also insufficient evidence to assess which
LHW training or intervention strategies are likely to be most effective.
1.1..2. Range of effect sizes approach (Median of medians approach)
Using the median of effects reported by eligible studies in the review provides a way of quantifying the effects of
interventions without resorting to numerous assumptions. It involves the use of median in a two-stage process. First, the
median effect across each studys eligible outcomes has to be identified. For instance, in a study reporting 10 process
outcomes (adherence to 10 different behaviours for a specific clinical condition) the median absolute difference in
compliance between the intervention and control group should be calculated. Then, with each study represented by its
median outcome, the median effect and interquartile range (IQR) across all included studies is calculated. A sensitivity
analysis, repeating the procedure using best and worst outcomes from each study, could be carried out. This method
was first proposed and used in a large review of strategies for implementing guidelines,(8) and since then it has been
applied in a number of Cochrane reviews of interventions to improve professional practice(9-13) and other systematic
reviews of quality improvement interventions.(14;15)
More detailed guidance about how to conduct this kind of synthesis could be found at the website of the Cochrane
Effective Practice and Organisation of Care group
The effects of on screen, point of care computer reminders on professional practice and health care outcomes(13)
Shojania et al evaluated the effects on processes and outcomes of care attributable to on-screen computer reminders
delivered to clinicians at the point of care.
They searched the Cochrane EPOC Group Trials register, MEDLINE, EMBASE and CINAHL and CENTRAL to July
2008, and scanned bibliographies from key articles
Two reviewers independently screened studies for eligibility and abstracted data. For each study, they calculated the
median improvement in adherence to target processes of care and also identified the outcome with the largest such
improvement. They then calculated the median absolute improvement in process adherence across all studies using
both the median outcome from each study and the best outcome
Twenty-eight studies (reporting a total of thirty-two comparisons) were included. Computer reminders achieved
amedian improvement in process adherence of 4.2% (interquartile range (IQR): 0.8% to 18.8%) across all reported
process outcomes, 3.3% (IQR: 0.5% to 10.6%) for medication ordering, 3.8% (IQR: 0.5% to 6.6%) for vaccinations,
and 3.8% (IQR: 0.4% to 16.3%) for test ordering. In a sensitivity analysis using the best outcome from each study, the
median improvement was 5.6% (IQR: 2.0% to 19.2%) across all process measures and 6.2% (IQR: 3.0% to 28.0%)
across measures of medication ordering
Authors concluded that point of care computer reminders generally achieve small to modest improvements in provider
behaviour. A minority of interventions showed larger effects, but no specific reminder or contextual features were
significantly associated with effect magnitude.
1.1.3. Vote counting
The conventional procedure of vote counting adds up the number of positive and negative comparisons and concludes
whether the interventions were effective on this basis.(16) Vote- counting could count either the number of comparisons
with a positive direction of effect (irrespective of statistical significance) or the number of comparisons with statistically
significant effects. Despite its intuitive appeal, the method fails to provide an estimate of the effect size of an intervention
(giving equal weight to comparisons showing a 1% change or a 50% change) and ignores the precision of the estimate
from the primary comparison (giving equal weight to comparisons with 100 or 1000 participants). Besides, when using
statistical significance for the counting, comparisons with potential unit of analysis errors need to be excluded because of
the uncertainty about their statistical significance. (17) Therefore, vote-counting does not seem to be an appropriate
method and for some authors it is never a valid approach.(18)
1.1.4. Narrative summary
A narrative summary typically involves the selection, chronicling, and ordering of evidence to produce an account of it.(3)
It is often used in systematic reviews when because of different reasons meta-analysis is not possible or the
reviewers think it would not be appropriate. Its form may vary from the simple recounting and description of findings to a
more interpretive and reflexive approach that includes higher levels of abstraction. The former is the way in which this
approach is currently used in most systematic reviews unable to pool statistically the measures of effect and it is
frequently subject to criticism because of its lack of transparency. The more interpretive form is closer to the narrative
synthesis described below in the section regarding qualitative approaches.Illustrative example
Illustrative example
The effects of changes in the pre-licensure education of health workers on health-worker supply.(19)
Pariyo et al assessed the effect of changes in the pre-licensure education of health professionals on health-worker
supply
Authors searched the Cochrane Central Register of Controlled Trials (CENTRAL) (The Cochrane Library 2007, Issue
3), EMBASE, Ovid (1980 to week 3, October 2007), MEDLINE, Ovid (1950 to week 3, October 2007), CINAHL
(October 2007), LILACS (week 4, November 2007), ERIC (1966 to week 3, February 2008), and Sociological
Abstracts (October 2007).Additionally WHO(WHOLIS) (February 2008), World Bank, Google Scholar, and human
resources on health-related websites were searched to obtain grey literature. Key experts in human resources for
health were contacted to identify unpublished studies. The reference lists of included studies were searched for
additional articles.
They found 2 controlled before-and-after studies on the effects of retention strategies. The heterogeneity among the
studies with regard to study populations precluded meta-analysis; therefore, findings for each study were presented
separately. Both studies (from the same authors) reported that an intervention comprising of a package of student
support activities including social, academic, and career guidance and mentorship resulted in an increase in the
number of minority students who enrolled and graduated from health training institutions.
Authors concluded that the evidence to estimate the likely effects of interventions in pre-licensure education to
increase health-worker supply is generally insufficient or unavailable, particularly in LMICs. However, promising
innovations from a high-income country include providing financial support to health professional students or
introducing mechanisms to identify and encourage potential students and offering support to at risk students
(Back to top)
1.2..Qualitative approaches.
1.2.1 Meta-ethnography
It is a method for synthesizing multiple qualitative research reports that works with metaphors or concepts as the unit for
synthesis, and that was first developed for use in educational research. It seeks to do more than simply collect and
review a series of accounts and instead aims at a novel synthesis, which develops a new theory to explain the range of
research findings encountered. (20) Three different methods of synthesis are used in meta-ethnography. The first one
involves the translation of concepts from individual studies into one another, producing overarching concepts and
metaphors (what is called reciprocal translational analysis). Refutational synthesis involves exploring and explaining
contradictions between individual studies. Lines-of-argument synthesis involves building up a picture of the whole from
studies of its parts.(21) A particular value for policy-makers of this approach is that it could help explain seemingly
divergent quantitative and qualitative study findings.
More detailed guidance about how to conduct this kind of synthesis could be found in the book by Noblit & Hare
Factors associated to patient adherence to tuberculosis (TB) treatment. (22)
Munro et al aimed to understand the factors considered important by patients, caregivers and health care providers
in contributing to TB medication adherence
They searched 19 electronic databases (1966February 2005) for qualitative studies on patients, caregivers, or
health care providers perceptions of adherence to preventive or curative TB treatment with the free text terms
Tuberculosis AND (adherence OR complianceOR concordance).
The study quality of included studies was assessed using a predetermined checklist and data were xtracted
independently onto a standard form. Authors then followed Noblit and Hares method of meta- thnography to
synthesize the findings, using both reciprocal translation and line-of-argument synthesis.
They screened 7,814 citations and selected 44 articles that met the prespecified inclusion criteria. We identified eight
major themes across the studies: organisation of treatment and care; interpretations of illness and wellness; the
financial burden of treatment; knowledge, attitudes, and beliefs about treatment; law and immigration; personal
characteristics and adherence behaviour; side effects; and family, community, and household support. Our
interpretation of the themes across all studies produced a line-of-argument synthesis describing how four major
factors interact to affect adherence to TB treatment: structural factors, including poverty and gender discrimination;
the social context; health service factors; and personal factors.
1.2.2. Thematic synthesis
It combines and adapts approaches from both meta-ethnography and grounded theory (a systematic qualitative research
methodology in the social sciences emphasizing generation of theory from data in the process of conducting research). It
is possible to identify three stages in thematic synthesis: the coding of text line-by-line; the development of descriptive
themes; and the generation of analytical themes. While the development of descriptive themes remains close to the
primary studies, the analytical themes represent a stage of interpretation whereby the reviewer go beyond the primary
studies and generate new interpretive constructs, explanations or hypotheses.(23)
More detailed guidance about how to conduct this kind of synthesis could be found in the Thomas and Hardens paper
previously cited in BMC Medical Research Methodology.
What is known about the barriers to, and facilitators of, healthy eating among children aged 4-10 years?(24)
The aim of the authors for this part of the analysis was to infer barriers to, and facilitators of, healthy eating and ideas
for effective interventions from childrens views.
They found eight qualitative studies that met their prespecified inclusion criteria
Authors examined the findings of each study in turn and assigned codes to describe relevant sentences or
paragraphs. Then they looked for similarities and differences between the codes to organize these into a hierarchical
tree structure centred on childrens understanding of healthy eating and the factors, in their views, that influence the
food they eat .In the next stage, three reviewers independently examined the descriptive themes and their associated
data in the light of the review question to infer barriers, facilitators, and implied recommendations for developing
interventions. The reviewers then met to discuss their findings and to develop a set of more abstract themes.
As an example, one of the themes was that children do not see their personal health as their responsibility but that of
their parents. Children do not regard purchasing fruit for health reasons as a legitimate use of their pocket money.
Again, that is the job of parents. Children prioritise taste over health. This theme suggested that future health
promotion interventions should promote fruit and vegetables as tasty rather than healthy and any emphasis on health
messages should be minimized.
1.2.3 Narrative synthesis
It refers to an approach to the systematic review and synthesis of findings from multiple studies that relies primarily on
the use of words and text to summarise and explain the findings of the synthesis. Whilst narrative synthesis can involve
the manipulation of statistical data, the defining characteristic is that it adopts a textual approach to the process of
synthesis to tell the
story of the findings from the included studies.
More detailed guidance about how to conduct this kind of synthesis could be found in the Guidance on the Conduct of
Narrative Synthesis developed for a group of researchers in the UK funded by the ESRC and in the webpage of the
project
What is known about the barriers to, and facilitators of, healthy eating among children aged 4-10 years?(24)
The aim of the authors for this part of the analysis was to infer barriers to, and facilitators of, healthy eating and ideas
for effective interventions from childrens views.
They found eight qualitative studies that met their prespecified inclusion criteria.
Authors examined the findings of each study in turn and assigned codes to describe relevant sentences or
paragraphs. Then they looked for similarities and differences between the codes to organize these into a hierarchical
tree structure centred on childrens understanding of healthy eating and the factors, in their views, that influence the
food they eat .In the next stage, three reviewers independently examined the descriptive themes and their associated
data in the light of the review question to infer barriers, facilitators, and implied recommendations for developing
interventions. The reviewers then met to discuss their findings and to develop a set of more abstract themes.
As an example, one of the themes was that children do not see their personal health as their responsibility but that of
their parents. Children do not regard purchasing fruit for health reasons as a legitimate use of their pocket money.
Again, that is the job of parents. Children prioritise taste over health. This theme suggested that future health
promotion interventions should promote fruit and vegetables as tasty rather than healthy and any emphasis on health
messages should be minimized.
(Back to top)
1.3. Mixed approaches.
This section describes some approaches to evidence synthesis that do not fit into the categories previously described
as they are capable of accommodating a diversity of types of evidence. In some way they may be also conceived as
global approaches to systematically reviewing evidence about a specific question with their specific methods for each
stage of the review process. However, in this section a focus on the synthesis methods proposed by each of them is
presented.
1.3.1 Realist synthesis
In this approach the primary focus is on the causal mechanisms or theories that underlie types of interventions or
programmes. It aims to build explanations across interventions or programmes which share similar underlying theories
of change as to why they work (or not) for particular groups in particular contexts.(26)
More detailed guidance about how to conduct this kind of synthesis could be found in the book of Ray Pawson
Evidence-Based Policy. A realist perspective'
Disentangling how community health workers (CHW) interventions work(27)
The aim of this study was to explore if randomized controlled trials (RCTs) could yield insight into the working of
interventions involving CHWs for improving child health
Authors searched a number of databases (PubMed, Popline, Embase, CINAHL, Cochrane Database of Systematic
Reviews and CENTRAL), and references of previous reviews. From 1,218 hits initially retrieved, 137 titles and
abstracts were considered potentially relevant. Twenty two of them were relevant for low and middle income
countries and the full text was retrieved and reviewed, ending with 10 studies (6 RCTs and4 cluster RCTs) included.
They found a number of interventions to improve CHW performance including Skills based training of CHW,
Supervision and referral support from public health services, Positioning of CHW in the community. When
interventions were applied in context of CHW programs embedded in local health services, with beneficiaries who
valued services andhad unmet needs, the interventions worked if following mechanisms were triggered: anticipation
of being valued by the community; perception of improvement in social status; sense of relatedness with
beneficiaries and publicservices; increase in self esteem; sense of self efficacy and enactive mastery of tasks; sense
of credibility, legitimacy and assurance that there was a system for back-up support. Studies also showed that if
context differed, even with similar interventions, negative mechanisms could be triggered, compromising CHW
performance
Authors concluded that RCTs could yield some insight, but the hypotheses generated were very general and not
well refined and they need to be tested and refined in further studies.
1.3.2. The EPPI-Centre approach
This approach typically involves a very broad review question from which separate questions are developed. These form
the focus of two or more parallel systematic syntheses. They may, for example, focus on sub-questions about
effectiveness, appropriateness, barriers and enablers to implementation, and the perspectives of the group targeted by
the intervention. The results of the separate syntheses are then combined in a so-called meta-synthesis aimed to
address the review question in its entirety.(28;29)
More detailed guidance about how to conduct this kind of synthesis could be found at the website of the Evidence for
Policy and Practice Information and Coordinating Centre (EPPI-Centre) at the Institute of Education, University of
London.
Young people and physical activity: a systematic review matching their views to effective interventions(30)
The objectives of the study were: i) systematically to locate and characterize existing research literature on the
barriers to, and facilitators of, physical activity among young people, especially those from socially excluded groups;
ii) to prioritize a sub-set of studies to review systematically in-depth; iii)to synthesize what is known from these
studies about the barriers to, and facilitators of, physical activity among young people, and how these can be
addressed; and iv) to identify gaps in existing research evidence.
Authors run highly sensitive searches across a wide range of electronic databases (e.g. The Cochrane Library,
PsycINFO, ERIC and the Social Science Citation Index). A range of controlled and free-text terms for physical activity
was combined with those for health promotion/determinants of health and young people.
Of the total of 7048 citations identified, 96 reports (describing 90 studies) were included in the descriptive map of the
literature. A sub-set of 12 trials and 16 studies of young peoples views entered the in-depth review.
Three types of synthesis were performed: i) narrative synthesis of trials; ii) narrative synthesis of views studies; and
iii) synthesis of trial and views studies together. In the second synthesis, each studys findings were considered in
relation to developing interventions for promoting participation in physical activity, using four separate questions. For
the last synthesis, a matrix was constructed which laid out the barriers and facilitators identified by young people
alongside descriptions of the interventions included in the indepth systematic review of trials.
Authors found that evidence for the effectiveness of the interventions was limited, with some suggestions of
improvements in knowledge and possible differences according to gender. Young women in particular identified
barriers to physical activity associated with certain ways of providing physical education in schools. Young people in
general identified a need for increased choice and facilities within the community and emphasized physical activitys
social side. Some of the barriers and facilitators identified by young people had been addressed by soundly
evaluated effective interventions but significant gaps were identified where no evaluated interventions appear to have
been published (e.g. initiatives explicitly addressing gender issues or the combination of sport and other leisure
activities), or where there were no soundly evaluated interventions.
They concluded that rigorous evaluation is required particularly to assess initiatives that address the limited practical
and material resources that young people identify as barriers to physical activity.
(Back to top)
2. How to choose a specific synthesis approach?.

Choice of a particular approach(es) will depend at least on: i) the underlying aim (i.e. review of the evidence base or
direct contribution to a decision); ii) the specific question(s) to be addressed; and iii) the nature and balance of evidence
available (e.g. whether most of the research is qualitative or quantitative).(20) In most cases the specific question(s) that
the review is addressing will command both the type of evidence selected (see section on selecting the evidence) and
the method used to summarize it. The Table attempts to provide some guidance on which approaches are likely to be
suitable for different sorts of questions.
Table. Choosing a suitable approach to synthesis given the review question and the evidence available (adapted from
Mays et al(16))
Review question Relevant types of evidence (if Likely approach(es) to

available) synthesis
Is this a problem? All types including research Narrative synthesis, meta-

and non-research (e.g.public & ethnography
stakeholder views, opinion
polls, focus groups), qualitative
and quantitative
How big is a problem? Which Quantitative research and Quantitative synthesis, meta-
groups does it affect? routine administrative data. ethnography
Qualitative data on subjective
impact
Is it changing over time? Quantitative research and Quantitative synthesis

routine administrative data
What can be done about it Mostly quantitative research on Meta-analysis of intervention

(what may work)? effectiveness studies
What works, for whom, in Wide range of research and Realist synthesis, narrative
which circumstances? What non-research data review
factors may moderate the
impact of this
policy/programme?
How acceptable will Largely qualitative research Qualitative synthesis

intervention/policy X be? What and non-research data
will the reaction be here?
(Back to top)
Resources.
Websites
Cochrane Effective Practice & Organisation of Care Group. The Cochrane Effective Practice and Organisation of Care
(EPOC) Group is a Review Group of The Cochrane Collaboration an international network of people helping
healthcare providers, policy makers, patients, their advocates and carers, make well-informed decisions about human
health care by preparing and publishing systematic reviews (SRs). The research focus of the EPOC Group are
interventions designed to improve the delivery, practice, and organisation of health care services. The EPOC editorial
base is located in Ottawa, Canada with satellite centres in Norway, Australia, and England. A number of resources
related to specific synthesis methods could be found in its website, including the use of the range of effect sizes
approach.
Cochrane Public Health Group The Cochrane Public Health Group (CPHG), formerly the Health Promotion and
Public Health (HPPH) Field, aims to work with contributors to produce and publish Cochrane reviews of the effects of
population-level public health interventions. A number of resources could be found in its website including detailed
guidance of synthesis methods potentially useful on conducting systematic reviews of public health topics.
Further reading
Cochrane Handbook for Systematic Reviews of Interventions. The Hanbook contains methodological guidance for the
preparation and maintenance of Cochrane intervention systematic reviews. Written in a clear and accessible format, it is
the essential manual for all those preparing, maintaining and reading Cochrane reviews. Edited as a book, its chapters
are also available on line through the Cochrane Collaboration handbook website. In chapter 9 a complete description of
the methods for synthesizing evidence in Cochrane reviews is presented.
Centre for Reviews and Dissemination guidance for undertaking reviews in health care. This publication presents
independent guidance produced by the Centre for Reviews and Dissemination (CRD) for the different steps involved
in the production of systematic reviews.
SCIE systematic research reviews: guidelines
Guidance on the Conduct of Narrative Synthesis in Systematic Reviews
Health Development Agency. Integrative approaches to qualitative and quantitative evidence. This report is an
informal review of the literature on integrating qualitative and quantitative forms of evidence. It explores five key
themes: i) the role of qualitative approaches in traditional trials and experimental studies, beginning with a general
discussion of the rationale for multi-method research; ii) at what point in the development of a field of knowledge it is
appropriate to pull qualitative and quantitative learning together; iii) the complex question of how to determine what
constitutes good evidence from qualitative studies; iv)a brief discussion of whether there are hierarchies of evidence
within the different types of qualitative investigation, and conclude that it is unlikely that consensus can be achieved;
and v)considerations in some detail how the findings of qualitative and quantitative evidence may be synthesized. A
discussion of some of the theoretical and methodological issues that remain to be resolved is presented, and the
report concludes with some directions for research and development.
National Institute for Health and Clinical Excellence. Moving beyond effectiveness in evidence synthesis
(Back to top)
References.
(1) Hunt M. How science takes stock. The story of meta-analysis. New York: Russell Sage Foundations; 1997.
(2) Popay J. Moving beyond effectiveness in evidence synthesis. Methodological issues in the synthesis of diverse sources of
evidence. London: National Institute for Health and Clinical Excellence; 2006.
(3) Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton AJ. Synthesising qualitative and quantitative evidence: a review of
possible methods. J Health Serv Res Policy 2005;10(1):45-53.
(4) Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J
2009 Jun;26(2):91-108.
(5) Pope C, Mays N, Popay J. Synthesizing qualitative and quantitative health evidence. A guide to methods. Maidenhead: Open
University Press; 2007.
(6) Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-Analysis. Chichester: Wiley; 2009.
(7) Lewin SA, Dick J, Pond P, Zwarenstein M, Aja G, van WB, et al. Lay health workers in primary and community health care.
Cochrane Database Syst Rev 2005;(1):CD004015.
(8) Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay C, Vale L, et al. Effectiveness and efficiency of guideline
dissemination and implementation strategies. Health Technol Assess 2004;8(6):1-352.
(9) Jamtvedt G, Young JM, Kristoffersen DT, Thomson O'Brien MA, Oxman AD. Audit and feedback: effects on professional
practice and health care outcomes (Review). The Cochrane Database of Systematic Reviews 2006;(2):CD000259.
(10) Doumit G, Gattellari M, Grimshaw J, O'Brien MA. Local opinion leaders: effects on professional practice and health care
outcomes. Cochrane Database Syst Rev 2007;(1):CD000125.
(11) Farmer AP, Legare F, Turcot L, Grimshaw J, Harvey E, McGowan JL, et al. Printed educational materials: effects on
professional practice and health care outcomes. Cochrane Database Syst Rev 2008;(3):CD004398.
(12) Forsetlund L, Bjorndal A, Rashidian A, Jamtvedt G, O'Brien MA, Wolf F, et al. Continuing education meetings and workshops:
effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2009;(2):CD003030.
(13) Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J. The effects of on-screen, point of care computer
reminders on processes and outcomes of care. Cochrane Database Syst Rev 2009;(3):CD001096.
(14) Walsh JM, McDonald KM, Shojania KG, Sundaram V, Nayak S, Lewis R, et al. Quality improvement strategies for
hypertension management: a systematic review. Med Care 2006 Jul;44(7):646-57.
(15) Ranji SR, Steinman MA, Shojania KG, Gonzales R. Interventions to reduce unnecessary antibiotic prescribing: a systematic
review and quantitative analysis. Med Care 2008 Aug;46(8):847-62.
(16) Bushman B, Wang M. Vote-counting procedures in meta-analysis. In: Cooper H, Hedges LV, Valentine JC, editors. The
Handbook of Research Synthesis and Meta-analysis. Second ed. New York: Russell Sage Foundation; 2009.
(17) Donner A, Klar N. Issues in the meta-analysis of cluster randomized trials. Statistics in Medicine 2002;21:2971-80.
(18) Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Vote counting - A new name for an old problem. Introduction to meta-
analysis.Chichester: John Wiley & Sons Ltd; 2009.
(19) Pariyo G, Kiwanuka S, Rutebemberwa E, Okui O, Ssengooba F. Effects of changes in the pre-licensure education of health
workers on health-worker supply. Cochrane Database Syst Rev 2009;(2):CD007018.
(20) Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policy-
making in the health field. J Health Serv Res Policy 2005;10(Suppl 1):S6-S20.
(21) Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol
2009;9:59.
(22) Munro SA, Lewin SA, Smith HJ, Engel ME, Fretheim A, Volmink J. Patient adherence to tuberculosis treatment: a systematic
review of qualitative research. PLoS Med 2007 Jul 24;4(7):e238.
(23) Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res
Methodol 2008;8:45.
(24) Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, et al. Integrating qualitative research with trials in systematic
reviews. BMJ 2004 Apr 24;328(7446):1010-2.
(25) Arai L, Britten N, Popay J, Roberts H, Petticrew M, Rodgers M, et al. Testing methodological developments in the conduct of
narrative synthesis: a demonstration review of research on the implementation of smoke alarm interventions. Evidence &
Policy 2007;3(3):361-83.
(26) Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review designed for complex
policy interventions. J Health Serv Res Policy 2005 Jul;10 Suppl 1:21-34.
(27) Kane SS, Gerretsen B, Scherpbier R, Dal Poz M., Dieleman M. A realist synthesis of randomised control trials involving use
of community health workers for delivering child health interventions in low and middle income countries. BMC Health Serv
Res 2010;10:286.
(28) Oliver S, Harden A, Rees R, Shepherd J, Brunton G, Garcia J, et al. An emerging framework for including different types of
evidence in systematic reviews for public policy. Evaluation 2005;11(4):428-46.
(29) Harden A, Thomas J. Methodological issues in combining diverse study types in systematic reviews. Int J Social Research
Methodology 2005;8(3):257-71.
(30) Rees R, Kavanagh J, Harden A, Shepherd J, Brunton G, Oliver S, et al. Young people and physical activity: a systematic
review matching their views to effective interventions. Health Educ Res 2006 Dec;21(6):806-25.
(Back to top)
single source
PRESENTING THE RESULTS
Introduction
Things to consider
Information about included/ excluded studies
How to report the results according to the method of synthesis
How to communicate information to audiences beyond the research community
Resources
References
Introduction
As in any systematic review, the findings of a HPSR review include reporting the number and describing the studies
found; and describing the main results that could answer the review question. The way in which those findings will be
presented will depend mainly on the type of evidence included in the review (qualitative or quantitative) and on the target
audience for the review. In fact, authors should consider at least two groups of audiences: the more academic sector and
decision/policymakers.
(Back to top)
Things to consider
1 Information about included/ excluded studies
A list of included studies should be provided including information related with the main characteristic of settings,
populations, interventions, comparators and outcomes for each study. The amount of information to be included will
depend on how useful is for understanding the intervention and for making judgments about the applicability of results. It
is also recommended to provide a list of excluded studies with the specific reasons for exclusion.
(Back to top)
2. How to report the results according to the method of synthesis
2.1 Quantitative data
If quantitative data are found and meta-analysis is done, the forest plot is the graphic way to present it.(1) If meta-
analysis is not possible, a narrative presentation of the findings should be done, describing consistently the information
from each study in a way that could be more understandable for the reader. Methodological details and
recommendations of these are offered in the chapter 11 of the Cochrane Handbook for Systematic Reviews of
Interventions.
An emerging tool for synthesizing and presenting quantitative results in a transparent and tabular way is the use of
Summary of Finding tables.(2;3) These tables provide key information about the quality of evidence, the magnitude of
effect of the intervention examined and the sum of available data for each main outcome. A table is presented for each
relevant comparison and includes:
A list of all important outcomes, both desirable and undesirable.
A measure of the typical burden of these outcomes (e.g. illustrative risk, or illustrative mean, on the control group).
Absolute and relative magnitude of effect (if both are appropriate).

Numbers of participants and studies addressing these outcomes.
A rating of the overall quality of evidence for each outcome (which may vary by outcome).
Space for comments.

The planning for the Summary of Findings table should come early in the systematic review, with the selection of the
comparisons and outcomes to be included both in the review and in the table. More specific guidance about how to build
a Summary of Finding table is presented in Chapter 11 of the Cochrane Handbook for Systematic Reviews.(4)
2.2- Qualitative data
When qualitative data are included in a review, data presentation will depend on the specific method chosen for
synthesizing the evidence (See Section Synthesis Methods). The objective is to demonstrate the process by which
themes were synthesized and to present the models resulting from synthesis. Some examples of presentation of
qualitative data can be found in the web page of Cochrane Qualitative Research Group (See resources)
(Back to top)
3 How to communicate information to audiences beyond the research community
The Cochrane Collaboration recommendation to review authors is that they consider the use of the abstract and a plain
language summary to communicate in a simpler and shorter way the relevant information from the review to healthcare
decision-makers and consumers of health care respectively. In this sense, authors should consider from the very
beginning of the review process the main issues they want to communicate to those different audiences.
(Back to top)
(Back to top)
Resources
Websites
Cochrane Qualitative Research Methods Group
The Cochrane Qualitative Research Methods Group develops and supports methodological work on the inclusion in
systematic reviews of findings from studies using qualitative methods and disseminates this work within and beyond the
Collaboration's Review Groups. The Cochrane Qualitative Methods Group focuses on methodological matters arising
from the inclusion of findings from qualitative studies into systematic reviews. Examples of how to present qualitative
data in systematic reviews can be found in its website.
Further reading
Cochrane Handbook for Systematic Reviews
The Cochrane Handbook for Systematic Reviews of Interventions (the Handbook) provides guidance to authors for the
preparation of Cochrane Intervention reviews (including Cochrane Overviews of reviews). The Handbook is updated
regularly to reflect advances in systematic review methodology and in response to feedback from users. In chapter 11 it
is possible to found detailed guidance about how to present the results of a review and the Summary of Findings tables.
(Back to top)
References.
(1) Lewis S, Clarke M. Forest plots: trying to see the wood and the trees. BMJ 2001 Jun 16;322(7300):1479-80.
(2) Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in Cochrane reviews improved understanding and rapid
retrieval of key information. J Clin Epidemiol 2010 Jun;63(6):620-6.
(3) Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. User testing and stakeholder feedback contributed to the development of
understandable and useful Summary of Findings tables for Cochrane reviews. J Clin Epidemiol 2010 Jun;63(6):607-19.
(Back to top)
Created with the Freeware Edition of HelpNDoc: Easily create Web Help sites
INTERPRETING THE FINDINGS
Introduction
Things to consider
Applicability
Equity
Economic considerations
Monitoring and evaluation
Resources
References
Introduction
In contrast with clinical medicine where the results of a determined intervention (e.g., the effectiveness of a drug) will
almost always work in a similar and predictable way, in the health systems field the same policy or program may have
completely different outcomes in different settings. This could be explained by differences in the health systems
mechanisms, as well as by a number of social, political, and economic factors of each society. These differences may
explain why the same intervention that it is used in one setting is not feasible or acceptable in another (e.g. abortion
policies), or why a program was successful in one country but was less effective in another (e.g. implementation of user
fees).
By the word 'setting', we will understand the political/country jurisdiction (e.g. Chile or Bangladesh, or their constituent
provinces), as well as the sectors (e.g. primary care or hospital care), or locales (e.g. urban or rural) where the studies
were carried out and/or the policies/programmes are intended to be implemented.
This section is focused on the things that reviewers should consider on making judgments about the relevance of the
findings and their potential translation to different settings. Normally those judgments will be included in the discussion
section of the review. Although in many cases local decision-makers will make those assessments for their local context,
reviewers are in a better position to make an initial global interpretation of the review findings.
(Back to top)
Things to consider
In order to translate the findings of a systematic review of health policy & systems research to different settings a number
of issues should be considered: applicability, equity, economic considerations, and needs for monitoring & evaluation.
1. Applicability
Applicability is the process of considering the different factors that will influence the decision of translating the findings of
a given study or review to a specific population, intervention, or setting. Transferability or the potential for translation are
similar and appropriate terms and they are all closely related to integrity, context, and sustainability(1).
The following questions can guide reviewers judgments about the applicability of findings(2).
1. The findings of the studies included in the systematic review were consistent across the different settings or time
periods?
If the findings of the systematic review have been shown to be consistent across settings or time periods then similar
impacts might be expected when a decision maker thinks about applying the same policy or program into their own
setting. In contrast, if the variability of the findings is large (e.g. high heterogeneity in a meta-analysis) then the
interpretation about applicability should be more cautious and describing the situations when it worked better and worse
becomes important.
2. What on-the-ground realities and constraints should be considered because they might substantially alter the
feasibility and acceptability of the findings?
The reviewers should consider the resources and capacity facilitators or constraints that were found in the settings of the
studies included in the review. For example, if the intervention consists of an important participation of midwives, then
the setting where this intervention is considered to be implemented must have the necessary number of these
professionals to carry out the tasks. Also, the perspectives and political influence amongst health system stakeholders
may be important to consider, because an option can be not feasible or acceptable in a specific setting. For instance, the
introduction of a demand side financing scheme may be strongly rejected by civil society groups making impossible its
implementation.
3. What health system mechanisms may mean that the findings could not work in the same way?
Health system mechanism could be divided into governance, financial and delivery arrangements. Governance
arrangements address the effects of changes in or interventions to improve policy authority, organizational authority,
commercial authority, professional authority, and user and stakeholder involvement. Financial arrangements concentrate
on the effects of changes in or interventions to improve financing, funding, remuneration, financial incentives and
resource allocation. Delivery arrangements deal with the effects of changes in or interventions to improve to whom care
is provided, by whom, where, with what information or technology and how the quality and safety of care is monitored(3).
Reviewers should describe the health systems arrangements where the studies included in the systematic review were
carried out, addressing if these could be critical factors affecting the findings. Deciding whether health system
arrangements might alter an option's impacts requires an understanding of how and why an option might work and
reviewers can give, for example, a framework, more evidence or features of the settings that will help to identify those
modifying factors.
4. What insights can be drawn about the constraints that may be found in the implementation of the intervention(s)?
When it comes to complex interventions in health systems, implementation of them is something that constitutes a crucial
step for their success. That is why reviewers can orientate about some key issues found in the review that might help to
understand the process of implementation of the intervention.
An assessment of the local applicability of a systematic review on lay health worker interventions from the perspective of a South African
policymaker (adapted from Lavis et al(2))
1. Were the studies included in the systematic review conducted in the same setting or were the findings consistent across settings or time
periods?
48 studies were included in the review
25 from the Unites States (US)
3 from the United Kingdom (UK)
2 each from Brazil, South Africa and Tanzania
1 each from Bangladesh, Canada, Ethiopia, Ghana, India, Ireland, Mexico, Nepal, New Zealand, Pakistan, Philippines, Thailand, Turkey
and Vietnam
Findings were not always consistent across settings
Most studies were published from 1995 onwards although one study was published in 1980. It is not clear from the review whether the
findings were consistent over time periods
2. Are there important differences in on-the-ground realities and constraints that might substantially alter the feasibility and acceptability of an
option?
In South Africa, concerns have been expressed about the capacity of the health system and non-government organisations (NGOs) to
provide clinical and managerial support for a very large scale-up of LHW programmes, particularly in currently under-resourced areas where,
it could be argued, they are most needed. Capacity may be different from the high-income settings (US, UK) in which many of the studies
were conducted
In South Africa, there is some resistance among nurses, and within nursing professional associations, to extending the scope of practice of
LHWs. This may restrict the range of tasks that LHWs are able to take on. While the acceptability of LHWs to consumers seems reasonable,
based on observations from existing programmes, this is likely to vary across settings in the country and for different tasks (e.g.
immunisation, breastfeeding promotion)
In South Africa, most LHWs are currently involved in providing home-based care to people living with HIV/AIDS and treatment support to
this group and to people with TB. It is not clear how feasible it would be to extend their roles to include the areas shown to be effective in the
review (immunisation promotion, treatment of childhood infections, breastfeeding promotion). Furthermore, the LHW interventions shown to
be effective in the review were focused on very specific health issues, such as the promotion of breastfeeding or immunisation uptake. Little
evidence was identified regarding the effectiveness of more 'generalist' LHWs who are given responsibility for delivering a range of primary
healthcare interventions
In South Africa, norms and traditions regarding breastfeeding as well as differing baseline levels of breastfeeding and high rates of
HIV/AIDS among mothers may alter the applicability of the review findings on LHWs for breastfeeding promotion
3. Are there important differences in health system arrangements that may mean an option could not work in the same way?
In South Africa, LHWs are not licensed to dispense antibiotics for the treatment of acute respiratory infections in children or to dispense
anti-malarial drugs. It may therefore be difficult in the short- to medium-term to extend their scope of practice in this way, even if shown to be
effective in a review
In South Africa, most LHWs are employed by NGOs, who receive funding from the government for the LHWs' salaries. It is not clear how
secure this funding mechanism is
4. What insights can be drawn about options, implementation, and monitoring and evaluation?
Most of the LHW interventions shown to be effective were focused on single tasks. The effectiveness of 'generalist' LHWs who deliver a
range of primary healthcare interventions needs evaluation.
(Back to top)
2. Equity
The World Health Organizations Commission on the Social Determinants of Health (CSDH) has defined health equity as
the absence of unfair and avoidable or remediable differences in health among social groups(4). Moreover, Whitehead
clarifies the difference between health inequalities, which are the measurable differences in health experience and
health outcomes between different population groupsaccording to socioeconomic status, geographical area, age,
disability, gender or ethnic group and health inequities, that she defines as differences in opportunity for different
population groups which result in, for example, unequal life chances, access to health services, nutritious food, adequate
housing, etc. These differences may be measurable; they are also judged to be unfair and unjust(5).
The following questions can guide reviewers judgments about the potential impacts of a policy or program
intervention(s) over disadvantaged populations and equity(6):
1. Which groups or settings are likely to be disadvantaged in relation to the intervention being considered?
Disadvantage may be related to economic status, employment or occupation, education, place of residence, gender,
ethnicity, or combinations of these characteristics. The relevance of these may vary depending on the policy or program
of interest. While there may be good reasons for prioritizing particular groups or settings generally, for specific policies or
programs it is often important to consider inequities in relation to a range of potentially disadvantaged groups or settings.
Subsequent attention by reviewers should focus on those groups or settings for which there is a reason to anticipate
significant differential effects.
2. Are there plausible reasons for anticipating differences in the relative impact of the intervention for disadvantaged
groups or settings?
If plausible reasons for differential impacts are identified the overall findings should be interpreted looking at the specific
impacts on relevant disadvantaged groups. In exploring these subgroup analyses, reviewers should be aware that they
can be misleading and then their interpretation should be done carefully. This is because studies may be too small to
reliably detect differences in effects, resulting in false negative conclusions. On the other hand, testing multiple
hypotheses regarding factors that might moderate the effects of an intervention may results in false positive conclusions.
General guidelines for interpreting subgroup analyses should be applied with a healthy skepticism whenever subgroup
analyses are considered.(7;8)
3. Are there important considerations that should be made when implementing the intervention(s) in order to ensure that
inequities are reduced, if possible, and that they are not increased?
In general, disadvantaged populations have poorer access and quality of care, and the implementation strategies can
help to reach them in a better way. Such mechanisms may include different delivery, financial and governance
strategies, or the investment of additional resources. Reviewers may have identified some of these strategies while
conducting the systematic review and they could be in a good position to give some guidance about this issue.
An example of important equity considerations regarding implementation (adapted from Oxman et al(6))
likelihood that disadvantaged children compared to more advantaged children will be exposed to greater health risks, have less resistance to
her mortality rates. These inequities are compounded by reduced access to health services. Even public subsidies for health frequently benefit rich
menting interventions to reduce child mortality will not necessarily reduce these inequities and may, in some cases, even increase them. Considerat
designed to reduce inequities, such as the provision of more affordable and accessible health services. These strategies may target poor peop
rsally. Situations in which targeting or universal coverage might be more appropriate include(9):
Targeting more likely to be appropriate Universal coverage more likely to be appropriate
asy to identify High risk groups hard to identify
eeded by children at risk Intervention needed by everyone
otects those who receive it Intervention has a spill-over effect
ely provided through the public sector Intervention is widely provided through the private sector
and for the intervention is low Spontaneous demand for the intervention is high
e unable to cover the whole population Health services are able to cover the whole population
e may be a more appropriate strategy for vaccines, which are needed by everyone and which have spill-over effects (decreasing the risk of infection
d others). However, in order to also reduce inequities in coverage, additional targeted strategies may be needed such as those that address proble
h service accessibility or to a lack of demand for vaccinations in disadvantaged populations.
(Back to top)
3. Economic considerations
We understand as economic considerations those implications of cost and cost-effectiveness that can be extrapolated
from the findings of a systematic review and that affect the use of resources. Every policy or program intervention has
an impact in the economic sphere, related directly with its consequences over health or other public policy areas. For
example, from an economic perspective, some potentially important resource consequences are:
Changes in use of health care resources
Changes in use of non-healthcare resources
Changes in use of patient and informal caregiver time
Changes in productivity
The following questions can guide reviewers judgments about the potential impacts of a policy or program
intervention(s) on the use of resources(10):
1. What are the most important impacts of the findings on resource use?
According to the findings, reviewers can identify the consequences on resource use of implementing the intervention.
They should consider both the resources used to implement the option (i.e. resource inputs such as drugs, equipment
and care) and subsequent resource use arising from the impacts of the option on health or other outcomes (e.g.
increases or decreases in healthcare utilization due to the impacts of the option). Also, changes in the productivity of
patients (e.g. because they get healthier) or the health organizations may be important.
2. How confident is it possible to be in the findings for impacts on resource use?
If the studies included in the systematic review yielded findings about resource use, the quality of this evidence must be
assessed for each important resource consequence. This is because the quality of evidence may be better for some
consequences (e.g. drug use) than for others (e.g. personnel time). The criteria for assessing the quality of evidence for
resource use are largely the same as those for health outcomes(11;12). According to this assessment, reviewers can
comment on the different aspects of the interventions for which the findings on resource use are more reliable.
Although published cost-effectiveness analyses can be helpful, particularly for developing a model, they are often of
limited value because the assumptions made and the unit costs used may not be transferable from the setting where the
analysis was done to the one where a decision must be made.
Identifying potentially important resource consequences for a national programme of outreach visits to improve prescribing for
hypertension (adapted from Oxman et al(10))
Systematic reviews have found that educational outreach visits (i.e. personal visits to healthcare professionals in their own settings by trained
outreach visitors) have relatively consistent and small, but potentially worthwhile, effects on prescribing(13). In a randomised trial in Norway,
these visits were found to increase the use of thiazides, in adherence with clinical practice guidelines, from 11% to 17% among patients with
newly diagnosed hypertension(14). To determine whether this improvement was worthwhile (in relation to the cost of a national outreach
programme), the following uses of resources were considered(15):
Development of software (used to audit medical records and provide feedback to physicians)
Training outreach visitors (pharmacists)
Printed materials
Travel for the pharmacists doing the outreach visits
Pharmacists' time
Administrative time (e.g. making appointments for the outreach visits)
Physicians' time (for the outreach visits)
Technical support
Drug expenditure
Patient visits
Laboratory tests
(Back to top)
4. Monitoring and Evaluation
The process of conducting a systematic review can give information about the need of performing monitoring and/or
evaluation (M&E) of the intervention or some particular aspect of it. Although these terms are not used consistently and
may mean different things to different people, evaluation has a particular focus on the achievement of results(16).
Monitoring could be defined as a continuing function that uses systematic collection of data on specified indicators to
provide management and the main stakeholders of an ongoing development intervention with indications of the extent of
progress and achievement of objectives and progress in the use of allocated funds. On the other hand, evaluation could
be defined as the systematic and objective assessment of an on-going or completed project, program or policy, its
design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development
efficiency, effectiveness, impact and sustainability(17). Impact evaluation is frequently used when an attempt is made to
assess whether observed changes in outcomes (or 'impacts') can be attributed to a particular policy or program.
The following questions can guide reviewers judgments about the need of additional M&E(16):
1. Is monitoring necessary?
The reviewers, by looking at the evidence provided by the systematic review, can judge if monitoring is necessary to
facilitate and improve the implementation of the intervention, helping the stakeholders to know more about what is
happening 'on the ground' during the implementation process.
2. What should be measured?
Reviewers can identify which parts of the intervention may require monitoring. There are various parts of the 'results
chain' (i.e. on inputs, activities, outputs, outcomes or impacts) that are typically used to monitor the implementation of a
program or policy option.(17) What part to monitor depends on the setting where it will be implemented, but evidence
from the systematic review can orientate where to focus on.
3. Should an impact evaluation be conducted?
When research evidence is scarce, reviewers can comment about the need of conduct a formal impact evaluation when
the intervention is implemented. This can help to establish a causal relationship between the program or policy and
changes in outcomes. On the other side, if the evidence is robust and consistent across different settings, then an impact
evaluation might not be necessary.
4. How should the impact evaluation be done?
Reviewers can comment about the most desirable type of study that can be used to evaluate the policy or program. To
evaluate the effectiveness of a health policy or health systems intervention, the designs that are recommended are
randomized controlled trials, controlled before-after, and interrupted time-series.(18)
Example of a monitoring system in the healthcare system. Scaling up provision of antiretroviral (ART) in Malawi(19).
When Malawian health authorities decided to make ART available to a large proportion of the HIV-positive population, a system was put in
place to monitor the implementation of this new policy. The principles of the system are based on the WHO's approach to the monitoring of
national tuberculosis programmes. Each patient who starts on ART is given an identity card with a unique identity number, and this is kept at
the clinic. The information collected from new patients includes their name, address, age, height, the name of their guardian, and the reason
for starting ART. Patients are asked to attend each month to collect their medication. During their visit, their weight is recorded and they are
asked about their general health, ambulatory status, work, and any drug side effects. Pill counts are also undertaken and recorded as a way
of ensuring drug adherence. In addition, the following standardised monthly outcomes are recorded using the following categories:
Alive: Patient is alive and has collected his/her own 30-day supply of drugs
Dead: Patient has died while on ART
Defaulted: Patient has not been seen at all for a period of 3 months
Stopped: Patient has stopped treatment completely either due to side effects or for other reasons
Transfer-out: Patient has transferred out permanently to another treatment
Data collected as part of the Malawian monitoring system of the ART rollout may be analysed and used in a variety of ways. Comparisons
can be made of treatment outcomes for patients who were recruited at different times. If, for example, the rate of switching from first-
second-line regimens increases, or rates of mortality do likewise, an increase in drug resistance to the first-line regimen could be the cause. If
the rate of deaths or defaulters declines, this could indicate that the management of the ART treatment programme is improving. If outcomes
are particularly poor in certain geographic areas or clinics, action may need to be taken to address this.
Example of an impact evaluation. Home-based antiretroviral therapy (ART) in Uganda(20;21).
Shortages of clinical staff and difficulties with accessing care due to transportation costs are major obstacles to scaling up the delivery of ART
in developing countries. One proposed solution is home-based HIV care, where drug delivery, the monitoring of health status, and the
support of patients is carried out at the home of the patient by non-clinically qualified staff. It is highly uncertain, however, whether this
strategy is able to provide care of sufficient quality, including timely referrals for medical care, or whether such a system is cost-effective.
Therefore, before implementing home-based care programmes widely it is important that they are evaluated for their (cost-) effectiveness.
To ensure a fair comparison between home-based and facility-based ART, researchers in Uganda conducted a randomised trial. The study
area was divided into 44 distinct geographical sub-areas. In some of these, home care was implemented, while in others a conventional
facility-based system continued to be used. The selection and allocation of areas to receive, and not to receive, the home-based care system,
was randomly determined. This reduced the likelihood of important differences between the comparisons groups which might otherwise have
influenced the study if, for example, the districts themselves had decided whether to implement home-based care, or if decisions had been
based on an existing preparedness to implement home-based care. The random allocation system used was also the fairest way of deciding
where to start home-based care since each district had an equal chance of being chosen.
The researchers found that the home-based care model using trained layworkers was as effective as nurse- and doctor-led clinic-based care.
(Back to top)
3. Resources
Useful documents and further reading
Haynes RB. Can it work? Does it work? Is it worth it? The testing of healthcare interventions is evolving. BMJ
1999;319:652-3.
SUPPORT Tools for evidence-informed health Policymaking (STP). These are a series of documents intended to
help policymakers and other relevant stakeholders in knowing how to find and use research evidence in the
policymaking process. They address four broad areas: supporting evidence-informed policymaking; identifying needs
for research evidence; finding and assessing both systematic reviews and other types of evidence; and going from
research evidence to decisions.
Dans AM, Dans L, Oxman AD, Robinson V, Acuin J, Tugwell P, Dennis R, Kang D. Assessing equity in clinical
practice guidelines. J Clin Epidemiol 2007;60:540-6.
Tugwell P, de SD, Hawker G, Robinson V. Applying clinical epidemiological methods to health equity: the equity
effectiveness loop. BMJ 2006; 332(7537):358-361.
Tugwell P et al. Assessing equity in systematic reviews: realising the recommendation of the Commission on Social
Determinants of Health. BMJ 2010;341:c4739.
Segone M (ed). Bridging the gap: The role of monitoring and evaluation in evidence-based policy making. UNICEF:
Geneva, 2008.
The World Bank. Monitoring and evaluation (M & E): Some Tools, Methods and Approaches. The World Bank:
Washington DC, 2004.
Barber S. Health systems strengthening interventions: Making the case for impact evaluation. Alliance for Health
Policy and Systems Research: Geneva, 2007.
Links to websites
Campbell and Cochrane Equity Methods Group
The Campbell and Cochrane Equity Methods Group is registered with the Campbell and the Cochrane Collaborations.
Both the Cochrane and Campbell Collaborations are international, not-for-profit, and independent organizations. The
Cochrane Collaboration's purpose is to ensure that relevant, accurate, and current research about health interventions is
available worldwide. To meet this objective, Cochrane contributors conduct and distribute systematic reviews. Similarly,
the Campbell Collaboration produces reviews with an aim to "help people make well-informed decisions about the effects
of interventions in the social, behavioral and educational arenas." The Equity Methods Group encourages authors of both
Campbell and Cochrane reviews to include explicit descriptions of the effect of the interventions not only on the whole
population but to describe their effect upon the disadvantaged and/or their ability to reduce socio-economic inequalities
in health and to promote their use to the wider community. Ultimately, this will help build the evidence base on such
interventions and increase our capacity to act on the health gap between rich and poor.
WHO Commission on Social Determinants of Health
The final reports on the WHO Commission on Social Determinants of Health are available here. They are intended to
support countries and global health partners to address the social factors leading to ill health and inequities. These
reports draw attention to the social determinants of health that are known to be among the worst causes of poor health
and inequalities between and within countries. The determinants include unemployment, unsafe workplaces, urban
slums, globalization and a lack of access to health systems.
EQUINET Africa
EQUINET, the Regional Network on Equity in Health in Southern Africa, is a network of professionals, civil society
members, policy makers, state officials and others within the region who have come together as an equity catalyst, to
promote and realise shared values of equity and social justice in health.
Global Equity Gauge Alliance
The Global Equity Gauge Alliance was created to participate in and support an active approach to monitoring health
inequalities and promoting equity within and between societies. The Alliance currently includes 11 member-teams, called
Equity Gauges, located in 10 countries in the Americas, Africa and Asia.
Campbell & Cochrane Economics Methods Group
The Campbell & Cochrane Economics Methods Group (CCEMG) is an international network of individuals with an
interest and expertise in approaches to evidence synthesis that combine economics and systematic review methods.
CCEMG strives to promote the inclusion of economic perspectives and evidence in systematic reviews of health care,
social welfare, education and criminal justice interventions. This is acheived through our development and provision of
methods guidance, training, peer review and advisory support for economics components of reviews.
International Health Economics Association
The International Health Economics Association was formed to increase communication among health economists,
foster a higher standard of debate in the application of economics to health and health care systems, and assist young
researchers at the start of their careers.
NHS Economic Evaluation Database (NHS EED)
NHS EED contains 24,000 abstracts of health economics papers including over 7,000 quality assessed economic
evaluations. The database aims to assist decision-makers by systematically identifying and describing economic
evaluations, appraising their quality and highlighting their relative strengths and weaknesses.
International Initiative for Impact Evaluation (3ie)
3ie seeks to improve the lives of poor people in low- and middle-income countries by providing and summarizing
evidence related to what policy options work, as well and when and why, and the costs involved.
Independent Evaluation Group (IEG) at the World Bank.
The IEG is an independent unit within the World Bank that assesses what works, and what does not; how a borrower
plans to run and maintain a project; and the lasting contribution of the Bank to a country's overall development. The
goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank's
work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and
disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings.
(Back to top)
References
(1) Armstrong R, Waters E, Doyle J. Reviews in public health and health promotion. In: Higgins JPT, Green S,
editors. Cochrane Handbook for Systematic Reviews of Interventions.Chichester: Wiley-Blackwell; 2008. p. 593-
606.
(2) Lavis JN, Oxman AD, Souza NM, Lewin S, Gruen RL, Fretheim A. SUPPORT Tools for evidence-informed
health Policymaking (STP) 9: Assessing the applicability of the findings of a systematic review. Health Res Policy
Syst 2009;7 Suppl 1:S9.
(3) Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, et al. Examining the role of health
services research in public policymaking. Milbank Q 2002;80(1):125-54.
(4) Solar O, Irwin A. Towards a conceptual framework for analysis and action on the social determinants of health.
WHO Commission on Social Determinants of Health; 2007.
(5) Whitehead M. The concepts and principles of equity and health. International Journal of Health Services
1992;22(3):429-45.
(6) Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP)
10: Taking equity into consideration when assessing the findings of a systematic review. Health Res Policy Syst
2009;7 Suppl 1:S10.
(7) Fletcher J. Subgroup analyses: how to avoid being misled. BMJ 2007 Jul 14;335(7610):96-7.
(8) Sun X, Briel M, Walter SD, Guyatt GH. Is a subgroup effect believable? Updating criteria to evaluate the
credibility of subgroup analyses. BMJ 2010;340:c117.
(9) Victora CG, Wagstaff A, Schellenberg JA, Gwatkin D, Claeson M, Habicht JP. Applying an equity lens to child
health and mortality: more of the same is not enough. Lancet 2003 Jul 19;362(9379):233-41.
(10) Oxman AD, Fretheim A, Lavis JN, Lewin S. SUPPORT Tools for evidence-informed health Policymaking (STP)
12: Finding and using research evidence about resource use and costs. Health Res Policy Syst 2009;7 Suppl
1:S12.
(11) Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ. What is "quality of evidence" and why
is it important to clinicians? BMJ 2008 May 3;336(7651):995-8.
(12) Guyatt GH, Oxman AD, Kunz R, Jaeschke R, Helfand M, Liberati A, et al. GRADE: Incorporating considerations
of resources use into grading recommendations. BMJ 2008 May 24;336(7654):1170-3.
(13) O'Brien MA, Rogers S, Jamtvedt G, Oxman AD, Odgaard-Jensen J, Kristoffersen DT, et al. Educational outreach
visits: effects on professional practice and health care outcomes. Cochrane Database Syst Rev
2007;(4):CD000409.
(14) Fretheim A, Oxman AD, Treweek S, Bjorndal A. Rational Prescribing in Primary Care (RaPP-trial). A randomised
trial of a tailored intervention to improve prescribing of antihypertensive and cholesterol-lowering drugs in general
practice [ISRCTN48751230. BMC Health Serv Res 2003 Feb 27;3(1):5, 2003.
(15) Fretheim A, Aaserud M, Oxman AD. Rational prescribing in primary care (RaPP): Economic evaluation of an
intervention to improve professional practice. Plos Medicine 2006;3(6):0792-9.
(16) Fretheim A, Oxman AD, Lavis JN, Lewin S. SUPPORT tools for evidence-informed policymaking in health 18:
Planning monitoring and evaluation of policies. Health Res Policy Syst 2009;7 Suppl 1:S18.
(17) OECD Development Assistance Committee Working Party on Aid Evaluation. Glossary of key terms in
evaluation and results based management. OECD Publications 2002Available from: URL ( link )
(18) Eccles MP, Grimshaw JM, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of
change and improvement strategies. Qual Saf Health Care 2003;12:47-52.
(19) Harries AD, Gomani P, Teck R, de Teck OA, Bakali E, Zachariah R, et al. Monitoring the response to
antiretroviral therapy in resource-poor settings: the Malawi model. Trans R Soc Trop Med Hyg 2004
Dec;98(12):695-701.
(20) Amuron B, Coutinho A, Grosskurth H, Nabiryo C, Birungi J, Namara G, et al. A cluster-randomised trial to
compare home-based with health facility-based antiretroviral treatment in Uganda: study design and baseline
findings. Open AIDS J 2007;1:21-7.
(21) Jaffar S, Amuron B, Birungi J, Namara G, Nabiryo C, Coutinho A, et al. Integrating research into routine service
delivery in an antiretroviral treatment programme: lessons learnt from a cluster randomized trial comparing
strategies of HIV care in Jinja, Uganda. Trop Med Int Health 2008 Jun;13(6):795-800.
(Back to top)
Created with the Freeware Edition of HelpNDoc: Full featured Documentation generator
GLOSSARY
Term Definition
Absolute risk reduction See Risk difference.
A solution designed for a specific problem or task that is not generalisable to other situations
Ad hoc (Latin).
An adverse event for which the causal relation between the drug/intervention and the event is at
Adverse effect least a reasonable possibility.
An adverse outcome that occurs during or after exposure to a drug or other intervention and
Adverse event
which may or may not be caused by the intervention.
Bias resulting from a systematic difference (other than the intervention) between experimental
Allocation bias and control groups in a clinical trial. Allocation bias can be avoided by randomisation.
See Concealment of allocation.

Allocation concealment
Formed or conceived beforehand (Latin).

A priori
Collection of material made at the end of a project and preserved to assist in answering queries
Archive about the review and to facilitate any update.
A graphical summary of overall diagnostic accuracy; the closer the curve is to the upper left
Area under the curve (AUC) hand corner of the graph, the better the diagnostic performance. The AUC ranges from 0 to 1,
with 0.5 indicating a poor test where the accuracy is equivalent to chance.
Bias resulting from systematic differences between comparison groups as a result of differential
Attrition bias withdrawals or exclusions of participants.
An approach to statistical analysis that can be used in single studies or meta-analysis. A prior
Bayesian analysis probability distribution based on objective evidence and subjective opinion is defined at the
outset. Bayes theorem is then used to update the prior distribution in light of the results of a
study, producing a posterior distribution from which point estimates of effect and credible
intervals (equivalent to confidence intervals) can be defined.
Participant characteristics that are collected at the beginning of a study prior to receiving the
Baseline characteristics intervention. Characteristics may include demographic details such as age and gender and
clinical characteristics such as stage of disease or presence of co-morbidites.
Before-and-after study A study design where a group is studied before and after an
intervention.
A systematic error or deviation in results or inferences from the underlying truth. See also
Bias selection bias; performance bias; attrition bias; detection bias and reporting bias.
Databases that provide descriptive records of items such as books and articles.
Bibliographic databases
Computer software that assists with the organisation of bibliographic references. There are
Bibliographic software many different packages (e.g. EndNote, Reference Manager), but most will allow for the import
of references from bibliographic databases and the automated production of reference lists.
Keeping knowledge of which comparison group a participant belongs (e.g. to intervention or

Blinding control) from the study participants, investigators or outcome assessors. This reduces the risk of
bias.
Boolean operators are used to combine terms when conducting electronic searches. Examples
Boolean operator include AND (used to narrow a search), OR (used to broaden a search) and NOT (used to
exclude terms from a search).
A statistical approach for examining the uncertainty in cost- effectiveness analysis. It involves
Bootstrapping drawing many random sub-samples from the original data set and computing the statistic of
interest from each in the same way. After sampling, each subsample is returned to the data set,
a process known as sampling with replacement.
An observational study that compares people with a specific disease or outcome of interest
Case-control study (cases) with a suitable control group of people without that disease or outcome, and which seeks
to find associations between the outcome and prior exposure to particular risk factors.
A study reporting observations on a series of individuals, usually all receiving the same
Case series intervention, with no control group.
Formal process for coding data from qualitative cases into a quantitative form for statistical
Case survey analysis.
See Heterogeneity.
Clinical heterogeneity
A trial where randomisation is of clusters of people (e.g. general practices, schools) rather than
Cluster randomised trial individuals themselves.
An international organisation that aims to help people make well-informed decisions about
Cochrane Collaboration health care by preparing, maintaining, updating and ensuring the accessibility of systematic
reviews of the effects of healthcare interventions
An observational study in which a defined group of participants is observed over time and a
Cohort study comparison made between those who did and those who did not receive the intervention.
Co-intervention An additional diagnostic or therapeutic procedure given to people receiving a particular

intervention.
Information provided by an organisation wishing to commission a systematic review to assist

Commissioning brief researchers in preparing proposals to undertake the work.
Any medium used to convey a message to an audience or audiences.

Communication channel
Co-morbidity The presence of one or more diseases or conditions other than those of primary interest.
In a controlled trial, the intervention (which could include placebo, usual care, another
Comparator intervention or no treatment) with which the intervention of interest is compared.
An intervention involving a number of separate elements that seem essential to the proper
Complex intervention
functioning of the intervention although the active ingredient of the intervention that is effective
is difficult to specify.
The process used to prevent foreknowledge of which comparison group an individual will be
Concealment of allocation assigned to in a randomised controlled trial. Inadequate concealment of allocation may lead to
selection bias.
In narrative synthesis, the use of visual methods to help to construct groupings of, and
Conceptual mapping relationships between, ideas and/or concepts. Closely related to idea webbing.
In narrative synthesis, the use of a combination of different perspectives and/or methods to

Conceptual triangulation study a particular concept.
A measure of uncertainty around the results of a statistical analysis that describes the range of
Confidence interval values within which we can be reasonably sure that the true effect lies. For example a 95%
confidence interval is based on the notion that if a study were repeated many times in other
samples from the same population, 95% of the confidence intervals from those studies would
include the true value of the effect being measured. Wider intervals indicate lower precision;
narrow intervals, greater precision.
A situation in which a measure of the effect is distorted because of an association between the
Confounding intervention (or exposure) with other factor(s) that influence the outcome under investigation.
For example, if the control group includes patients with more advanced stages of cancer than in
the intervention group, then an analysis of survival will be confounded by tumour stage.
A set of procedures for collecting and organizing non-structured information. This approach
Content analysis makes it easier to systematically and objectively analyze the data and make inferences about
the population of interest.
Continuous outcomes Outcomes related to variables with a potentially infinite number of possible values within a given
range, for example weight and blood pressure.
A system of publication credit in which all those who contributed to a publication are listed with
Contributorship details of their contribution, including those who did not meet the standard criteria to be listed
as authors.
The group that acts as a comparator for one or more experimental interventions in a controlled
Control group trial. The group without the disease or outcome of interest in a case control study.
A clinical trial with a control group.

Controlled trial
An economic analysis that converts effects into health terms and describes the costs for some
Cost-effectiveness analysis additional health gain (e.g. cost per additional stroke prevented).
In economic analyses, an adjustment applied to charges in order to better reflect the true costs
Cost-to-charge ratios of the technology being evaluated.
Critical interpretive synthesis A form of review which, while sensitive to the issues involved in conducting reviews that
conventional systematic review methodology has identified, draws on a distinctive tradition of
qualitative inquiry, including interpretive approaches to review, enabling the generation of
theory with strong explanatory
power.
A type of clinical trial comparing two or more interventions in which all the participants receive
Cross-over trial all the interventions but the order of receipt is determined by randomisation.
A study that examines the relationship between diseases (or other health related characteristics)
Cross-sectional study
and other variables of interest as they exist in a defined population at a particular time.
In bibliographic software, a field for which the type of content is not pre-specified by the
Custom field software and which can therefore be customised by the individual. Often used in systematic
reviews for keeping track of decisions or articles ordered.
The threshold or value at which continuous data are divided into dichotomous categories. If used
Cutpoint it is important that the cutpoint is not determined by a data dependent process.
The specific platform through which a database is accessed. Examples include OVID and Dialog.
Database provider Many databases are available via more than one provider. See also Search interface.
A theoretical construct (often using a mathematical framework) that allows the comparison of
Decision modelling the relationship between costs and outcomes of alternative health care interventions by
incorporating evidence from a variety of sources.
Bias caused by systematic differences between comparison groups in how outcomes are
Detection bias ascertained, diagnosed or verified.
An overall indicator of diagnostic performance, calculated as the odds of a positive test result
Diagnostic odds ratio among those with the target condition, divided by the odds of a positive test result among those
without the condition.
Point at which diagnostic test results are classified as positive or negative.

Diagnostic threshold
Data that can take one of two possible values for example dead or alive, myocardial infarction or
Dichotomous data no myocardial infarction. Also known as binary data.
A type of bias that occurs when a diagnosis is verified using different reference standards,
Differential verification bias depending upon the result of the index test.
In health economics, a reduction applied to future costs and benefits to reflect the fact that
Discounting costs and benefits available today have a higher value than those occurring in the future.
In a controlled trial, this is the process by which the participants and the investigators (outcome
Double-blind
assessors) are prevented from knowing which intervention the participants have been given. See
also Blinding.
The extent to which a specific intervention, applied under usual circumstances, does what it is
Effectiveness intended to do.
The extent to which an intervention produces a beneficial effect under ideal conditions.
Efficacy
The observed relationship between an intervention and an outcome expressed as, for example
Estimate of effect odds ratio, risk difference, risk ratio, hazard ratio, standardised mean difference, weighted mean
difference, number needed to treat.
An approach to the practice of medicine that involves integrating individual clinical expertise
Evidence-based practice with the best available external clinical evidence from systematic research.
An intervention under evaluation.

Experimental intervention
A study in which investigators determine (by randomisation or another method) to which

Experimental study intervention group an individual will be allocated.
The degree to which the results of a study hold true in other settings (generalisability). See also
External validity Validity.
A method of calculating a pooled effect that assumes all variation of estimates of effect between
Fixed-effect model studies is assumed to be due to random error (the play of chance). See also Random-effects
model.
Forest plot A graphical representation of the individual results of each study included in a meta-analysis
together with or without the pooled meta-analysis result. The plot provides an at a glance
indication of the variability between studies and highlights any studies with outlying results.
Each study is shown as squares centred on its estimate of effect with a horizontal line indicating
the confidence interval. A vertical line is drawn through the value that indicates no difference
between the interventions being compared (e.g. 1.0 for an odds ratio). If shown, the overall
pooled estimate is represented as a diamond at the bottom of the plot. The centre of the
diamond represents the pooled point estimate, and its horizontal extremities represent the
confidence interval.
In literature searching, the use of everyday words and phrases, as opposed to index terms, to
Free text terms search bibliographic databases.
A graphical display of study precision such as the standard error plotted against effect size that
Funnel plot can be used to investigate biases associated with small trials (including publication bias).
See External validity.

Generalisability
A general term for the kind of material that is not published in an easily accessible form or listed
Grey literature in standard bibliographic databases, for example conference proceedings, internal
reports, theses and some books.
A set of methods for sampling, data collection and analysis.

Grounded theory
The process of searching a journal page by page to identify relevant articles.
Handsearching
A measure of effect calculated by time-to-event analyses that represents how many times more
Hazard ratio or less one group is likely to experience the outcome of interest.
A broad term that covers any method used by health professionals to promote health, to prevent
Health technology and treat disease, or to foster and improve rehabilitation and long-term care.
In systematic reviews heterogeneity refers to variability or differences between studies. A

Heterogeneity
distinction is sometimes made between: statistical heterogeneity - differences in the effect
estimates methodological heterogeneity - differences in study design clinical heterogeneity -
differences in participants, interventions or outcome measures See also Homogeneity.
A hierarchy of study designs based on their internal validity,or risk of bias, with well-designed
Hierarchy of evidence systematic reviews and randomised trials at the top and observational studies and case series
lower down.
The degree of similarity between the studies included in areview.

Homogeneous
Idea webbing In narrative synthesis, the use of visual methods to help toconstruct groupings of, and
relationships between, ideas and/or concepts. Closely related to Conceptual mapping.
A type of bias that occurs when the result of the index test isused in establishing the final
Incorporation bias diagnosis (i.e. it forms part of the reference standard).
The difference in the mean costs of two interventions in the population of interest divided by the
Incremental cost-effectiveness ratio difference in the mean outcomes in the population of interest.
(ICER)
The test whose performance is being evaluated.

Index test
A word or words used to describe the subject of, for example, a journal article. They are
Indexing term(s) designed to make searching easier and more effective. Ideally these terms will be assigned from
a controlled vocabulary, for example MeSH. See also MeSH.
A specific type of systematic review that uses the original individual participant data obtained
Individual patient data (IPD) meta- from those responsible for included studies. Data are centrally collected, checked and re-
analysis
analysed.
True intention to treat analysis captures two criteria: (i) participants should be analysed
Intention to treat (ITT) analysis irrespective of whether or not they received their allocated intervention and irrespective of what
occurred subsequently, for example, participants with protocol violations or those subsequently
judged ineligible should be included in the analysis; (ii) all participants should be included
irrespective of whether outcomes were collected. Although the first criterion is generally
accepted, there is no clear consensus on the second as it involves including participants in the
analyses whose outcomes are unknown, and therefore requires imputation of data. Many
authors describe their analyses as ITT when only the first criterion has been met.
See Validity.
Internal validity
The degree of agreement exhibited when a measurement is repeated under identical conditions
Inter-rater agreement by different raters.
A quasi-experimental study design involving multiple observations over time that are
Interrupted time series interrupted, usually by an intervention, to permit separation of real intervention effects from
other long-term trends.
A group of participants in a study receiving a particular intervention.

Intervention group
A widely used and easy to implement method of pooling study results which is very flexible and
Inverse variance method can be used to combine any type of effect measure provided that standard errors are available.
A fixed effect meta-analysis using the generic inverse variance method calculates a weighted
average of study effect estimates by summating individual effect estimates and weighting these
by the reciprocal of their squared standard errors.
Investigator triangulation In narrative synthesis, a method of exploring the extent to which heterogeneity in study results
may be attributable to the diverse approaches taken by different researchers.
A measure of inter-rater agreement.

Kappa statistic
Interactions between decision-makers and researchers through the process of planning,

Knowledge transfer disseminating and applying existing or new research in decision-making.
A scatter plot of the risk in the experimental group against the risk in the control group, used to
LAbb plot
explore possible heterogeneity in systematic reviews.
Bias in a systematic review resulting from limiting inclusion to the exclusion of items not written
Language bias in a particular language or languages.
The deliberate restriction of search results to particular languages. Results in language bias
Language restrictions
A measure of accuracy of a diagnostic test. The likelihood ratio expresses the odds that a
Likelihood ratio (LR)
diagnostic test will give the correct result in a patient with the target disorder.
Two way communications and partnerships between producers and users of research.
Linkage and exchange
For a systematic review this should be a systematic search for information on a given topic. Can
Literature search include searches of bibliographic databases, websites, handsearching of journals and books,
citation searching and reference checking.
A method for combining studies that uses an alternative weighting scheme to the inverse
Mantel-Haenszel variance method. It has better statistical properties than the inverse variance method when
events are few.
An analytical method particularly suited to modelling repeated events (e.g. headache) or the
Markov model progression of a chronic disease (e.g. dementia) over time.
See Blinding.
Masking
A standard statistic that measures the absolute difference between the mean value of two
Mean difference groups, estimating the amount by which on average the intervention changes the outcome
compared to the control.
MeSH is the controlled vocabulary indexing system used by the National Library of
Medical Subject Heading (MeSH) Medicine for indexing articles on Medline. It is also used in some other electronic bibliographic
databases. See also Indexing term; Keyword.
Statistical techniques used to combine the results of two or more studies and obtain a combined
Meta-analysis estimate of effect.
A set of techniques for synthesising qualitative studies. It involves the selection, comparison and
Meta-ethnography analysis of studies to create new interpretations or concepts.
A statistical technique used to explore the relationship between study characteristics and study
. Meta-regression results in a systematic review.
See Multi-level synthesis.

Meta-synthesis
The use of qualitative and quantitative methods together.

Mixed-method
A variable that alters the effect of an explanatory variable on a dependent variable.

Moderator variable
A method of quantifying the contribution of each of a series of tests to the diagnostic process by
Multivariable prediction modelling modelling the occurrence of the target condition as a function of the different test results.
Where a review of qualitative studies is undertaken alongside a review of quantitative studies

Multi-level synthesis and the results of the two syntheses are combined. (Also referred to as meta-synthesis,
sequenced, or cross-design synthesis).
Predominantly a textual approach that provides an analysis of the relationships within and
Narrative synthesis between studies and an overall assessment of the robustness of the evidence.
The probability of non-disease among persons with a negative test result.

Negative predictive value
Number needed to treat/harm An estimate of how many people need to receive an intevention before one more person would
(NNT/NNH) experience a beneficial or a harmful outcome, respectively. Also referred to as number needed
to treat for benefit (NNTB) and number needed to treat for harm (NNTH).
A study in which the investigators observe and measure but do not seek to intervene.
Observational study
Odds describe the ratio of the probability that an event will happen to the probability that it will
Odds not happen.
The ratio of the odds of an event in one group (e.g. the experimental (intervention) group) to
Odds ratio the odds of an event in another (e.g. the control group).
In economic evaluations, one-way simple sensitivity analysis varies each parameter individually
One-way or multi-way sensitivity in order to isolate the consequences of each parameter on the results of the study. Multi-way
analyses
simple sensitivity analysis varies two or more parameters at the same time and the overall
effect on the results is evaluated.
The cost of foregone outcomes that could have been achieved through alternative investments.
Opportunity costs
An aspect of a participants clinical or functional status that we seek to change through

Outcome intervention, for example survival, tumour recurrence, conception, live birth, level of anxiety,
frequency of asthma attacks. See also Primary outcome and Secondary outcome.
The proportion of people correctly classified by the test.

Overall accuracy
Parallel synthesis Where a review of qualitative studies is undertaken alongside a review of quantitative studies
and the formal qualitative synthesis is used to interpret the findings of the quantitative
synthesis.
A type of bias that occurs when only a selected sample of participants undergoing the index test
Partial verification bias also receive the reference standard (e.g. only those who had a positive index test result).
An analysis restricted to those participants in a study who followed the trial protocol closely
Per protocol analysis
enough to ensure that their data would be likely to show an effect of treatment if it existed. Per
protocol analysis may be subject to bias because the reasons for not following the protocol may
be related to treatment.
Bias resulting from systematic differences in care provided to those in each intervention group
Performance bias (other than the intervention being evaluated) that arise because carers or participants act
differently because they know which intervention is being delivered.
Calculates odds based on the difference between the observed number of events and the
Peto odds ratio number of events that would be expected if there was no difference between experimental and
control interventions. The method performs better than alternative approaches when events are
very rare. It can also be used to combine time to event data by pooling log rank observed-
expected (O - E) events and associated variance. It can give biased estimates when treatment
effects are very large, or where there is a lack of balance in treatment allocation within the
individual studies.
The process of testing a procedure on a small scale before introducing it into practice, e.g.
Piloting testing a data extraction form on a small sample of studies to identify any problems and
inconsistencies between reviewers.
An intervention without specific biological activity in the condition being treated, usually
Placebo
administered to compare its effects with those of an active intervention. Placebos are used
because the act of intervention (rather than the intervention itself) may bring about some
benefit for psychological or other reasons.
See Meta-analysis
Pooling
The probability of disease among persons with a positive test result.

Positive predictive value
A measure of the likelihood of random errors in the results of a study, meta-analysis or

Precision measurement. The proportion of articles identified by a search strategy that are relevant.
The main or outcome of greatest importance. See also Outcome and Secondary outcome.
Primary outcome
Primary study The original study in which data were collected. The term is sometimes used to distinguish such
studies from secondary studies that re-examine previously collected data (e.g. systematic
reviews).
In economic evaluations, probabilistic sensitivity analysis attributes distributions of probabilities

Probabilistic sensitivity analyses to the uncertain variables which are incorporated into evaluation models based on decision
analytical techniques (e.g. Monte-Carlo simulation). This method can only be used to deal with
uncertainty in data input.
Prognostic markers (biomarkers) Characteristics that help to identify or categorise people with different risks of specific
future outcomes. They may be simple clinical measures such as body mass index, but are more
often pathological, biochemical, molecular or genetic measures or attributes.
Tests conducted to assess a patients risk of a particular outcome.

Prognostic tests
A study in which participants are identified and then followed forward in time to observe whether
Prospective study particular outcomes do or do not occur.
Bias arising from the fact that studies with statistically significant results are more likely to be
Publication bias published than those with inconclusive results. As a result, systematic reviews
that fail to include unpublished studies may omit relevant research and are likely to be biased
towards the positive and overestimate the effect of an intervention.
Facilitating demand (user)-led access to research findings.

Pull strategy
Researcher-led distribution of new research findings.

Push strategy
The probability of obtaining the observed effect (or larger) under the null hypothesis which for
p-value systematic reviews will commonly be the assumption that there is no effect of the experimental
intervention. A very small p-value means that it is very unlikely that the observed effect has
arisen purely by chance and provides evidence against the null hypothesis.
The point on the ROC curve that intersects with the line of symmetry (where sensitivity is equal
Q* to specificity). Sometimes used as an indicator of overall test performance where there is no
clinical preference for maximising either sensitivity or specificity.
A method for summarising and comparing data from case studies using Boolean logic.
Qualitative comparative analysis
A set of techniques for aggregating qualitative research findings.

Qualitative meta-summary
A set of techniques for the interpretive integration of qualitative findings.

Qualitative meta-synthesis
Research that adopts an interpretive, naturalistic approach and studies things in their natural
Qualitative research settings.
Quality of life An individuals emotional, social, and physical well-being, and their ability to function in the
ordinary tasks of living.
In systematic reviews, restricting inclusion to studies that meet predefined criteria related to
Quality threshold quality (validity).
In economic evaluations, a measure of health gain in which survival duration is weighted or

adjusted by the patients (health-related) quality of life during the survival period.
Quality-adjusted life year (QALY)
Research that concentrates on describing and analysing phenomena by using numerical data
Quantitative research and empirical models.
See Meta-analysis
Quantitative synthesis
The process of allocating participants to one of the groups of a randomised controlled trial using
Randomisation (i) a means of generating a random sequence and (ii) a means of concealing the sequence, such
that those entering participants to a trial are unaware of which intervention a participant will
receive. This should ensure that intervention groups are balanced for both known and unknown
factors.
An experiment in which investigators use randomisation to allocate participants into the groups
Randomised controlled trial (RCT) that are being compared. Usually allocation is made at the level of individuals, but sometimes it
is done at group level e.g. by schools or clinics.
A graph used to display the trade-offs between sensitivity and specificity as a result of varying
Receiver Operating Characteristic (ROC) the diagnostic threshold.
curve
The best currently available diagnostic test, against which the index test is compared.
Reference standard
A statistical modelling technique used to estimate or predict the influence of one or more
Regression analysis independent variables on a dependent variable e.g. the influence of stage of disease and tumour
size on survival.
A statistical phenomenon by which extreme examples from any set of data are likely to be
Regression to the mean followed by examples which are less extreme; a tendency towards the average of any sample.
For example, the offspring of two very tall individuals tend to be tall, but closer to the average
(mean) than either of their parents.
Developing long-term relationships with customers in order to retain them; relationship

Relationship marketing marketing techniques focus on customer retention and satisfaction.
See Risk ratio.

Relative risk
A bias caused by only a subset of all relevant data being available for inclusion. For example
Reporting bias through not all trials being published or not all outcomes being reported.
The combination and evaluation of separate studies to provide a coherent overall understanding
Research synthesis to a research question.
Resources A general term covering the staff, time, money, equipment and consumables required to, for
example, implement an intervention or conduct a systematic review.
A study in which the outcomes have occurred to the participants before the study commenced.
Retrospective study
The probability with which an outcome (usually adverse) will occur. For example, if out of 100
Risk participants 20 have a myocardial infarction, the risk of infarction is 0.2.
The difference in size of risk between two groups. For example, if the control group has a 30%
Risk difference risk of experiencing a particular event and the intervention group has a 20% risk of experiencing
the event, the risk different is 10%. Also known as Absolute risk reduction.
An aspect of an individuals genetic, physiological, environmental, or socioeconomic state that

Risk factor affects the probability of them experiencing a particular disease or outcome. For example people
with high body mass index are at increased risk of developing diabetes.
The ratio of the risk of an event in one group (e.g. the experimental (intervention) group) to the
Risk ratio risk of an event in another (e.g. the control group).
A calculation performed when planning a clinical study to determine the number of participants
Sample size calculation needed to ensure a given probability of detecting an effect of a given magnitude if it exists.
A test used to detect possible disease in people without symptoms.

Screening test
The means by which a user can interrogate a database. Interfaces vary in complexity. Some
Search interface consist merely of a text box in which a limited number of words can be entered. Others are
more complex and allow the searcher to create complex searches. See also Database host.
A term used to combine words within a search. For example, many search interfaces allow
Search operator searches for terms occurring within so many words of each other (known as adjacency
searching). See also Boolean operator.
The exact terms and their combinations used to search a bibliographic database.
Search strategy
An outcome of lesser importance than the primary outcome. See also Outcome and Primary
Secondary outcome outcome.
A relatively long-term trend in a community or country.

Secular trend
1. Bias caused by systematic differences between comparisongroups in prognosis or

Selection bias responsiveness to treatment.
2. Bias caused by systematic differences between those who are selected for a study and those
who are not. This affects the generalisability (external validity) of a study but not its (internal)
validity or risk of bias.
3. Bias arising from the way in which studies were selected for inclusion in a systematic review,
for example, publication bias.
In diagnostic/screening tests, a measure of a tests ability to correctly identify people with the
Sensitivity disease or condition of interest. In literature searching, the proportion of relevant articles that
are retrieved using a specific search strategy.
An analysis used to test the robustness of findings and determine how sensitive results are to
Sensitivity analysis the data that were included and/or the way that analyses were done.
An activity that makes the recipient believe they have received the actual intervention when
Sham (surgery/device) they have not; e.g. sham surgery involves an anaesthetic, an incision and suturing, but without
the actual surgical intervention being performed
In diagnostic/screening tests, a measure of a tests ability to correctly identify people who do

Specificity not have the disease or condition of interest. In literature searching, the proportion of non
relevant articles that are not retrieved.
In systematic reviews a person or group with an interest in or potentially affected by the results
Stakeholder of the review.
Standardised mean difference The difference between two estimated means divided by an estimate of the within-group
standard deviation. It is used to standardise and combine results from studies that have used
different ways of measuring the same concept, e.g. mental health.
See Heterogeneity.
Statistical heterogeneity
The probability of rejecting the null hypothesis when a specific alternative hypothesis is true. In
Statistical power comparative studies the chance of detecting a real effect as statistically significant, given that
the effect actually exists. For a given size of effect, studies with more participants have greater
power. Studies with a given number of participants have more power to detect large effects than
to detect small effect.
An adjective describing a random or probabilistic event or process.

Stochastic
In a clinical study or systematic review, an analysis in which the effect of the intervention is
Sub-group analysis evaluated in a defined subset or subsets of participants.
Data that have been aggregated for presentation or analysis, for example numbers of events in
Summary data each group in a clinical trial.
Surrogate outcome An outcome measure that is not of direct practical importance but is believed to be an indicator
or predictor of outcomes that are clinically important. These are often physiological or
biochemical markers that can be obtained much more quickly compared to the clinical outcome
of interest. To be valid, a surrogate outcome must have been shown to correlate with and
accurately predict the outcome of interest.
A one-sided comparison between the results of an index test and those of a reference standard.
Any discrepancy is assumed to arise from error in the index test.
Test accuracy study
A method used in the analysis of qualitative data to systematically identify the main, recurrent
Thematic analysis/synthesis and/or most important themes and/or concepts across multiple responses.
In economic evaluations, threshold analysis identifies the critical values of the parameters above
Threshold analyses or below which the results of a study vary. This method is usually used together with simple
sensitivity analysis.
The time span that reflects the period over which the main differences between interventions in
Time horizon
health effects and use of health care resources are expected to be experienced.
The predilection of an individual (or a society) for the use of resources in the present rather than
Time preferences in the future.
Data that reflect not just whether an event occurs but the time at which it happens. For example
Time-to-event data time to death or survival analysis. Each data item is represented by a state variable indicating
whether or not an event has occurred and an elapsed time at which the state was assessed.
Individuals who have not (yet) experienced the event at a particular point in time are censored
and contribute their event-free time to the analysis.
Treatment received analysis See Per-protocol analysis
A research strategy in which the researcher observes the same variable or phenomenon with
Triangulation multiple sources, measures, and methods.
A symbol used when searching electronic databases to retrieve all words that begin with a
Truncation symbol particular stem. For example, a search for child$ on Ovid MEDLINE will find any words that
begin with child, including child, children and childhood.
See Discounting.
Undiscounted
The re-running of a literature search to capture material that has become available since the
Update searching original search was conducted. May involve re-writing search strategies to take account of
changes in terminology and database indexing.
The degree to which a measurement truly measures what it purports to measure.

Validity (of a measurement)
Validity (of a study) The degree to which a result of a study is likely to be true and free of bias (systematic errors),
and hence the degree to which inferences drawn from the study are likely to be justified. Validity
in this sense is synonymous with internal validity. See also External validity.
In meta-analysis, the relative contribution of each individual study to the overall result and/or
Weighting the method used to determine this. Studies are often weighted by the inverse of the variance of
their measure of effect so that studies with more statistical information make a relatively greater
contribution.
In economic evaluations, a sensitivity analysis using extreme values for the input data to
Worst/best case analysis investigate the outcome of the economic evaluation in the extreme case. A pessimistic or
optimistic outcome is generated. Also known as Analysis of extremes.
A table presenting the results of a test accuracy study, showing the number of true positive,
2x2 contingency table false positive, false negative and true negative results.
Created with the Freeware Edition of HelpNDoc: Free Web Help generator
DOWNLOAD
- Doc file (MS WORD) <Download>
- Handbook Offline <Download>
Created with the Freeware Edition of HelpNDoc: Easily create Web Help sites

Handbook Systematic Reviews

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Handbook Systematic Reviews

Uploaded by

Copyright:

Available Formats

handbook

The relevance of systematic reviews

The relevance of systematic reviews

What is a systematic review?

How the tool is organized?

Scoping the review

The people behind this work

2. What are the elements of a scoping study

3. How to conduct a scoping study

i) Identifying the research question

A scoping review to improve conceptual clarity of interprofessional interventions(10).

1.1 Characteristics of the intervention

1.2 Contextual factors

1.3 Multiple and diverse outcomes

2 The lumping versus splitting debate

3 Defining eligibility criteria for the review

4 What study designs should be included in the review

5 The involvement of other stakeholders in the review process

1.1. Electronic databases

1.2. Reference lists of other studies and reviews

1.3. Conference and proceedings

1.4. Dissertation and thesis

1.5. Contact with experts and authors

1.6. Other search engines

1.7. Relevant web sites

1.9. Ongoing studies

2.1. Two illustrative examples

How to search for studies?

2.2 Illustrative example

10- Clinical trial

13- 4 AND 8 AND 12

3 Developing, organizing and reporting a search strategy

Where to look for reviews

A number of sources could be used to locate systematic reviews:

In databases of systematic reviews as the Cochrane library .

In databases of summaries of systematic reviews, as DARE

In databases of HPSR as Health Systems Evidence

2. Which data to collect

The following information should be included in the data extraction form:

4. Assessing the quality of included studies

Table 1. Items used in appraising quality in quantitative studies

Quality Items to assess in an RCT

Cochrane Effective Practice and Organisation of Care (EPOC) Group

Cochrane Consumers and Communication group

Cochrane Qualitative Research Methods Group

EppiCentre Methods and Tools pages

Cochrane Handbook for Systematic Reviews

Systematic Reviews: CRDs guidance for undertaking reviews in health care

1.1 Quantitative approaches.

1.1..2. Range of effect sizes approach (Median of medians approach)

1.1.3. Vote counting

1.1.4. Narrative summary

1.2.2. Thematic synthesis

1.2.3 Narrative synthesis

1.3. Mixed approaches.

1.3.1 Realist synthesis

Disentangling how community health workers (CHW) interventions work(27)

1.3.2. The EPPI-Centre approach

2. How to choose a specific synthesis approach?.

Review question Relevant types of evidence (if Likely approach(es) to

Is this a problem? All types including research Narrative synthesis, meta-

Is it changing over time? Quantitative research and Quantitative synthesis

What can be done about it Mostly quantitative research on Meta-analysis of intervention

How acceptable will Largely qualitative research Qualitative synthesis