Professional Documents
Culture Documents
Review Article
Blackwell Publishing Ltd
Abstract
Background and objectives: The expansion of evidence-based practice across
sectors has lead to an increasing variety of review types. However, the diversity
of terminology used means that the full potential of these review types may be
lost amongst a confusion of indistinct and misapplied terms. The objective
of this study is to provide descriptive insight into the most common types of
reviews, with illustrative examples from health and health information domains.
Methods: Following scoping searches, an examination was made of the
vocabulary associated with the literature of review and synthesis (literary
warrant). A simple analytical framework—Search, AppraisaL, Synthesis and
Analysis (SALSA)—was used to examine the main review types.
Results: Fourteen review types and associated methodologies were analysed
against the SALSA framework, illustrating the inputs and processes of each
review type. A description of the key characteristics is given, together with
perceived strengths and weaknesses. A limited number of review types are
currently utilized within the health information domain.
Conclusions: Few review types possess prescribed and explicit methodologies
and many fall short of being mutually exclusive. Notwithstanding such limitations,
this typology provides a valuable reference point for those commissioning, con-
ducting, supporting or interpreting reviews, both within health information
and the wider health care domain.
information. Consequently, they tended to reach available to most policymakers. The need to trade
conclusions that were biased and wrong.2 off rigour and relevance has become a central
Archie Cochrane, a famous British epidemiologist, theme to recent methodological developments and
noted: has led to a bewildering plethora of review designs
developed to meet a variety of demands from the
‘It is surely a great criticism of our profession that domains of research and policy.
we have not organized a critical summary, by An early example of a review was published in
specialty or subspecialty, adapted periodically, of the non-health library and information sector,7
all relevant randomized controlled trials.’3 whilst the early 1980s saw the first of many manu-
scripts seeking to answer the eternal question of
In answer to this challenge, the worldwide how best to undertake bibliographic instruction in
Cochrane Collaboration was formed in 1992 to the health sciences.8
provide an expanding resource of updateable Coinciding with the first Evidence-Based
systematic reviews of randomized controlled trials Library and Information Practice (EBLIP) con-
(RCTs) relating to health care. Thus began the ference in 2001, Booth proposed that the library
modern incarnation of the review article, a tool and information science (LIS) sector could follow
that had for many centuries been the mainstay for the trend within medical EBP by graduating to the
updating scientific knowledge. development of more systematic reviews once a
critical mass of rigorous studies has been
attained.9 A recent evaluation of the evidence base
Rise of the review
has continued to sound such a call for establishing
While it is well established that, in the 18th century, a solid evidence base within the LIS sector.10
James Lind was the instigator of the first reported To ‘review’ has been defined as: ‘To view,
RCT, a lesser-known fact records that he was probably inspect, or examine a second time or again’.11
the first to describe the systematic review method. Six This definition broadly characterizes all review
years after his first RCT was published,4 Lind wrote: types now in existence. What remain largely un-
acknowledged are the subtle variations in the degree
‘As it is no easy matter to root out prejudices ... it of process and rigour within the multifarious
became requisite to exhibit a full and impartial review types. Such variations are most clearly
view of what had hitherto been published on the evidenced in the structures and methodologies
scurvy ... by which the sources of these mistakes that distinguish one review type from another. As
may be detected. Indeed, before the subject could more professions have adopted and adapted the
be set in a clear and proper light, it was necessary systematic review method, the LIS sector, as
to remove a great deal of rubbish.’5 elsewhere, has been afforded access to an ever-
increasing variety of methods and techniques for
Gathering research, getting rid of rubbish and summarizing the evidence base. The model of the
summarizing the best of what remains captures the systematic review of randomized controlled trials
essence of the science of systematic review. has limited potential within the LIS literature,
Nevertheless, although the need to synthesize given that no more than a score of such studies had
research evidence has been recognized for well been identified. This has necessitated the identifi-
over two centuries, it was not until the 20th cation of a greater range of review types, opening
century that researchers began to develop explicit up the prospect of summarizing case studies,
methods for this form of research. A fuller account qualitative research and even theoretical and con-
of the rise of the discipline of research synthesis ceptual published and unpublished outputs.
has been published by Iain Chalmers and other
distinguished proponents.6 Recent years have seen
Different types of reviews
recognition that the typical timescale commanded
by the rigour of the systematic review process may Analysis of download statistics from the Health
be unsuited to the decision-making windows Information and Libraries Journal electronic archive
since 2006 indicates that reviews figure prominently type was analysed, its characteristics were described
amongst the most highly sought articles. This and its perceived strengths and weaknesses were
suggests that, as with medicine before it, the LIS outlined. An example of each type of review was
sector values the opportunity to access already identified and selected, primarily for its usefulness
synthesized evidence in informing its practice. in illustrating review characteristics. No judgement
Indeed, in 2008, Ankem noted that there was of quality is implied by each selection.
evidence of systematic reviews making a ‘substantial
contribution to medical library and information
Results: characterizing types of review
literature’.12
Ankem, in an evaluation of methods in systematic Fourteen review types and associated methodologies
reviews and meta-analyses published between were analysed against the SALSA framework
1996 and 2006 in the library and information science (see Table 1). To inform the subsequent analysis,
sector, identified a total of eight manuscripts. the same framework had been previously applied
These manuscripts, combined with reviews published to 17 review manuscripts identified from combining
in Health Information and Libraries Journal those figuring in the Health Information and Libraries
following an editorial commitment in 2007 to seek Journal review feature between 2007 and 2009 with
to publish a review in each future issue,13 provide papers cited in Ankem’s Review of Reviews (see
an illuminating insight into the flourishing Table 2). A descriptive summary appears below.
terminology synonymous with this genre. This
terminology includes such terms or phrases as
Critical review
review of the evidence,14,15 comprehensive review,16
literature review,17 overview18 and systematic Description. A critical review aims to
review.19–30 Given the importance evidence-based demonstrate that the writer has extensively
practice places upon the retrieval of appropriate researched the literature and critically evaluated
information, such diverse terminology could, if its quality. It goes beyond mere description of
unchecked, perpetuate a confusion of indistinct identified articles and includes a degree of analysis
and misapplied terms. and conceptual innovation. An effective critical
review presents, analyses and synthesizes material
from diverse sources. Its product perhaps most
Objective
easily identifies it—typically manifest in a hypothesis
The objective of this study is to provide a descriptive or a model, not an answer. The resultant model
insight into the most common examples of review, may constitute a synthesis of existing models or
illustrated by examples from health and health schools of thought or it may be a completely new
information domains. interpretation of the existing data.
Critical review Aims to demonstrate writer has extensively Seeks to identify No formal quality Typically narrative, Significant component: seeks to
researched literature and critically evaluated its most significant items assessment. Attempts perhaps conceptual identify conceptual contribution
quality. Goes beyond mere description to include in the field to evaluate according or chronological to embody existing or derive
degree of analysis and conceptual innovation. to contribution new theory
Typically results in hypothesis or model
Literature review Generic term: published materials that provide May or may not May or may not Typically narrative Analysis may be chronological,
examination of recent or current literature. include comprehensive include quality conceptual, thematic, etc.
Can cover wide range of subjects at various searching assessment
levels of completeness and comprehensiveness.
May include research findings
Mapping review/ Map out and categorize existing literature Completeness of No formal quality May be graphical Characterizes quantity and
systematic map from which to commission further reviews searching determined assessment and tabular quality of literature, perhaps by
and/or primary research by identifying by time/scope study design and other key
gaps in research literature constraints features. May identify need for
primary or secondary research
Meta-analysis Technique that statistically combines the Aims for exhaustive, Quality assessment may Graphical and Numerical analysis of measures
results of quantitative studies to provide a comprehensive searching. determine inclusion/ tabular with of effect assuming absence of
more precise effect of the results May use funnel plot to exclusion and/or narrative commentary heterogeneity
assess completeness sensitivity analyses
Mixed studies Refers to any combination of methods where Requires either very Requires either a generic
Typically both Analysis may characterise both
review/mixed one significant component is a literature sensitive search to retrieve appraisal instrument or
components will be literatures and look for
methods review review (usually systematic). Within a review all studies or separately separate appraisal presented as narrative correlations between
context it refers to a combination of review conceived quantitative processes with and in tables. May also characteristics or use gap analysis
approaches for example combining and qualitative strategies corresponding checklists
employ graphical means to identify aspects absent in one
quantitative with qualitative research or of integrating quantitative literature but missing in the other
outcome with process studies and qualitative studies
Overview Generic term: summary of the [medical] May or may not include May or may not include Synthesis depends on Analysis may be chronological,
literature that attempts to survey the comprehensive searching quality assessment (depends whether systematic or not. conceptual, thematic, etc.
© 2009 The authors
literature and describe its characteristics (depends whether whether systematic Typically narrative but may
systematic overview or not) overview or not) include tabular features
Qualitative systematic Method for integrating or comparing the May employ selective Quality assessment Qualitative, Thematic analysis, may
review/qualitative findings from qualitative studies. It looks for or purposive sampling typically used to narrative synthesis include conceptual models
evidence synthesis ‘themes’ or ‘constructs’ that lie in or across mediate messages not
individual qualitative studies for inclusion/exclusion
Journal compilation © 2009 Health Libraries Group. Health Information and Libraries Journal, 26, pp.91–108
© 2009 The authors
Table 1 Continued
Rapid review Assessment of what is already known Completeness of Time-limited formal Typically narrative Quantities of literature and
about a policy or practice issue, by using searching determined quality assessment and tabular overall quality/direction of
systematic review methods to search and by time constraints effect of literature
critically appraise existing research
Scoping review Preliminary assessment of potential size and Completeness of searching No formal quality Typically tabular Characterizes quantity and quality
scope of available research literature. Aims to determined by time/scope assessment with some narrative of literature, perhaps by study
identify nature and extent of research constraints. May include commentary design and other key features.
evidence (usually including ongoing research) research in progress Attempts to specify a viable review
State-of-the-art Tend to address more current matters in Aims for comprehensive No formal quality Typically narrative, Current state of knowledge
review contrast to other combined retrospective and searching of current assessment may have tabular and priorities for future
current approaches. May offer new perspectives literature accompaniment investigation and research
on issue or point out area for further research
Systematic review Seeks to systematically search for, appraise Aims for exhaustive, Quality assessment Typically narrative What is known; recommendations
and synthesis research evidence, often comprehensive may determine with tabular for practice. What remains
95
96
Table 2 Reviews from Health Information and Libraries Journal review (2007–2009) feature or Ankem’s review of reviews (2008)
Authors (year) Description No. of included studies Search Appraisal Synthesis Analysis
Journal compilation © 2009 Health Libraries Group. Health Information and Libraries Journal, 26, pp.91–108
Ankem (2006)19 Systematic review 110 studies 3 databases None Narrative Meta-analysis and
of the research literature and tabular descriptive statistics
Booth et al. (2009)21 Systematic review 29 14 databases Standard checklists of quality assessment Qualitative Thematic using32
criteria for different study designs
Boulos et al. (2007)18 Overview Not specified Not specified None Narrative Descriptive
Brettle (2003)22 Systematic review of 24 3 databases Instrument developed by Health Care Narrative Descriptive
the literature Practice R&D Unit (University of Salford) and tabular
Brettle (2007)23 Systematic review 54 7 databases None Narrative Thematic and
and tabular descriptive statistics
Brown (2008)24 Systematic review 20 peer reviewed, 19 23 databases Articles from popular press, magazine Narrative Chronological
magazine, 146 newspaper and newspaper articles reviewed for and tabular and thematic
and 141 university types of information published
newspaper articles
Childs et al. (2005)25 Systematic review 57 8 databases None Narrative Descriptive
of the literature
Davies (2007)14 Review of the evidence Not specified (34 from table) 3 databases None Narrative Descriptive
and tabular
Fanner & Urquhart Systematic review Not specified 9 databases None Narrative Descriptive
(2008)26
Grant (2007)27 Systematic review 13 LISA None Narrative Thematic
Hall & Walton (2004)17 Literature review 23 7 databases None Narrative Descriptive
Koufogiannakis & Systematic review 55 15 databases Glasgow checklist Narrative Meta-analysis and
Wiebe (2006)28 and meta-analysis framework analysis
Rossall et al. (2008)15 Review of the evidence Not specified Not specified None Narrative Descriptive
Wagner & Byrd (2004)29 Systematic review 35 5 databases Criteria for medical informatics Narrative Descriptive
evaluative studies plus additional criteria and tabular
Ward et al. (2008)16 Comprehensive review 79 12 databases None Narrative Thematic
of the research literature
© 2009 The authors
Weightman & Systematic review 28 7 databases Internationally accepted criteria from Narrative Descriptive
Williamson (2005)30 previously published literature and tabular
Beverley & Winning Systematic review of Seventeen (16 unique) 16 databases CriSTAL: Critical Skills Training in Narrative Descriptive
(2003)20 the literature evaluative and 33 Appraisal for Librarians Checklist and tabular
descriptive studies
A typology of reviews, Maria J. Grant & Andrew Booth 97
more structured approaches to the literature. Perceived weaknesses. Literature reviews lack an
While there is considerable value in trying to explicit intent to maximize scope or analyse data
identify all the available literature on a topic under collected. Any conclusions they may reach are
review, there is no formal requirement to present therefore open to bias from the potential to omit,
methods of the search, synthesis and analysis perhaps inadvertently, significant sections of the
explicitly. The emphasis is on the conceptual literature or by not questioning the validity of
contribution of each item of included literature, statements made. Additionally, authors may only
not on formal quality assessment. While such a select literature that supports their world view,
review does serve to aggregate the literature on a lending undue credence to a preferred hypothesis.
topic, the interpretative elements are necessarily
subjective and the resulting product is the starting Example. Hall, A. & Walton, G. Information
point for further evaluation, not an endpoint in overload within the health care system: a literature
itself. review. Health Information and Libraries Journal
2004, 21(2), 102–8.
Example. Kulviwat, S., Guo, C. & Engchanil,
N. Determinants of online information search: a
Mapping review/systematic map
critical review and assessment. Internet Research:
Electronic Networking Applications and Policy Description. This type of review has been
2004, 14(3), 245–53. developed and refined by the Evidence for Policy
and Practice Information and Co-ordinating
Centre (EPPI-Centre), Institute of Education,
Literature review
London, to map out and categorize existing
Description. According to the Medical Subject literature on a particular topic,34 identifying gaps
Headings (MeSH) scope note, a literature review in research literature from which to commission
describes ‘Published materials which provide an further reviews and/or primary research. Mapping
examination of recent or current literature. reviews can be distinguished from scoping reviews
Review articles can cover a wide range of subject (see below) because the subsequent outcome may
matter at various levels of completeness and involve either further review work or primary
comprehensiveness based on analyses of literature research and this outcome is not known beforehand.
that may include research findings’.33 This is
necessarily a very broad description making it difficult Perceived strengths. Mapping reviews enable the
to generalize. However, common characteristics contextualization of in-depth systematic literature
are that a literature review reviews published reviews within broader literature and identification
literature, implying that included materials possess of gaps in the evidence base. They are a valuable
some degree of permanence and, possibly, have tool in offering policymakers, practitioners and
been subject to a peer-review process. Generally, researchers an explicit and transparent means of
a literature review involves some process for identifying narrower policy and practice-relevant
identifying materials for potential inclusion— review questions. Systematic maps may characterize
whether or not requiring a formal literature studies in other ways such as according to
search—for selecting included materials, for theoretical perspective, population group or the
synthesizing them in textual, tabular or graphical setting within which studies were undertaken. In
form and for making some analysis of their addition to describing the research field, a
contribution or value. systematic map can also provide the basis for an
informed decision about whether to undertake the
Perceived strengths. The literature review method in-depth review and synthesis on all of the studies
seeks to identify what has been accomplished or just a subset. The map can show whether the
previously, allowing for consolidation, for building total population of studies is sufficiently similar
on previous work, for summation, for avoiding for a coherent synthesis. It can also establish
duplication and for identifying omissions or gaps. whether these studies will help answer the review
question and address pragmatic considerations about compared with the time taken to review scattered
the resources available to complete the review. individual studies,
For policy makers the attraction of being able to attempts to survey the literature and describe its
arrive at a more holistic understanding of a characteristics. As such, it can be used for many
particular intervention or condition is compelling. different types of literature review, with differing
This does however depend upon the review team degrees of systematicity. In the early days of
being able to demonstrate the added value of the systematic reviews, the term ‘overview’ was used
combined approach and also being equipped to synonymously with ‘systematic review’ to describe
meet the challenge of delivering a larger review that particular approach. As a consequence, the
enterprise within the same meaningful decision- value of the term within a typology is debatable,
making timeframe. Unlike single method reviews, notwithstanding the fact that the overall intent
such as most of those in this typology, mixed conveyed by the term continues to have
methods reviews also provide a potentially more considerable appeal to readers.
complete picture of the research landscape in a
specific topic area. Perceived strengths. Overviews can provide a
broad and often comprehensive summation of
Perceived weaknesses. Mixed Methods reviews a topic area and, as such, have value for those
may compound the methodological challenges of coming to a subject for the first time.
appraising and synthesising both quantitative and
qualitative research with the added difficulty of Perceived weaknesses. As mentioned above, the
integrating the resultant products. Currently no term ‘overview’ is frequently used as a non-
consensus exists on the point at which quantitative discriminant word for reviews of varying rigour
and qualitative components should be integrated. and quality. For this reason, the Cochrane
For example should the qualitative component Collaboration has chosen to differentiate
precede the qualitative thus informing the resultant ‘systematic overview’, used as a synonym for
question? Alternatively should the qualitative ‘systematic review’ (see below), from other types
component follow the quantitative to increase of overview that typically lack both systematic
understanding of how the intervention works or of methods and explicit reporting.
issues relating to its implementation or to
adherence? Clearly if both components proceed in Example. Boulos, M., Kamel, N., Hetherington, L.
parallel there are issues as to when they should & Wheeler, S. Second Life: an overview of the
opportunely be brought together—either at a potential of 3-D virtual worlds in medical and
significant summative point or iteratively to help health education. Health Information and Libraries
inform the ongoing conduct of both components. Journal 2007, 24(4), 233–45.
More significant than such pragmatic decisions
are more complex issues regarding the theoretical
Qualitative systematic review/qualitative
and methodological challenges of bringing
evidence synthesis
together differently structured studies, addressing
different though related questions and conducted Description. Qualitative systematic review is ‘a
within different paradigms. method for integrating or comparing the findings
from qualitative studies. The accumulated
Example. Shepherd, J., Harden, A., Rees, R., knowledge resulting from this process may lead to
Brunton, G., Garcia, J., Oliver, S. & Oakley, A. the development of a new theory, an overarching
Young people and healthy eating: a systematic “narrative”, a wider generalization or an
review of research on barriers and facilitators. “interpretative translation” ’. It ‘looks for “themes”
Health Education Research 2006, 21(2), 239–57. or “constructs” that lie in or across individual
qualitative studies. The goal ... is not aggregative
in the sense of “adding studies together,” as with a
Overview
meta-analysis. On the contrary, it is interpretative
Description. An overview is a generic term used in broadening understanding of a particular
for ‘any summary of the [medical] literature’37 that phenomenon’.38
There remains considerable confusion regarding acceptable as long as the method of sampling
the phrase ‘qualitative systematic review’. This is a papers for inclusion is appropriate. Such debates
historical legacy of the systematic review move- centre on whether the dominant model for
ment whereby ‘when the results of primary studies qualitative evidence synthesis is the classic
are summarized but not statistically combined, the systematic review method or whether it is more
review may be called a qualitative systematic appropriate to adapt and adopt concepts from
review’.39 This definition (i.e. that of a systematic primary qualitative research (e.g. grounded
review where meta-analysis is not possible) theory, theoretical saturation, purposive sampling
continues to persist, particularly in the literature etc.). Nevertheless, emerging guidance, now
of analgesia and pain management. For this, and included in the Cochrane Collaboration’s handbook42
other reasons, the Cochrane Collaboration’s and Centre for Reviews and Dissemination
Qualitative Research Methods Group promotes methodologies is gradually moving towards
‘qualitative evidence synthesis’ as the terminology greater consensus.43
of choice. Other terms encountered in the literature
include the tautological ‘qualitative meta-synthesis’ Example. Duggan, F. & Banwell, L. Constructing
and the misleading ‘meta-ethnography’ (describing a model of effective information dissemination in
a method that can be adapted to interpreting a crisis. Information Research 2004, 9(3). Available
many types of qualitative research, not simply from: http://InformationR.net/ir/9-3/paper178.html.
ethnographies).
Rapid review
Perceived strengths. Qualitative systematic reviews
can be used: to explore barriers and facilitators to Description. The methods of rapid review, seen
the delivery and uptake of services; for an initially by some as an unwelcome concession to
exploration of user views; to investigate perceptions the need for evidence-based decisions within a
of new roles, from the point of view of either those policymaker’s time frame, have recently gained
filling the roles or those with whom the post holder legitimacy in the form of Rapid Evidence
interacts; and to inform the prioritization of Assessments. This method is now proposed by the
services where evidence on effectiveness is equivocal Government Social Research website as a means
and preferences and attitudes thus become the of providing an ‘assessment of what is already
determining factors.40 This type of review therefore known about a policy or practice issue, by using
possesses a considerable strength in complementing systematic review methods to search and critically
the research evidence with the other two essential appraise existing research’.44
components of evidence practice, i.e. user-reported
and practitioner-observed considerations.41 Provided Perceived strengths. Rapid reviews and rapid
such insights are generalizable, findings from evidence assessments seek to be ‘Quick but Not
qualitative research may be more powerful than Dirty’: ‘They aim to be rigorous and explicit in
isolated comments from local questionnaires or method and thus systematic but make
surveys. concessions to the breadth or depth of the process
by limiting particular aspects of the systematic
Perceived weaknesses. Methods for qualitative review process’.45 The methodology identifies
systematic review are still in their infancy and several legitimate techniques that may be used to
there is considerable debate about when specific shorten the timescale. These include carefully focusing
methods or approaches are appropriate. For example, the question, using broader or less sophisticated
opinions differ as to whether comprehensive search strategies, conducting a review of reviews,
search strategies are required, identifying as many restricting the amount of grey literature,
relevant qualitative research studies as possible, or extracting only key variables and performing only
whether what is being sought is a holistic ‘simple’ quality appraisal. The reviewer chooses
interpretation of a phenomenon. If the latter is the which stages to limit and then explicitly reports
case, then a more selective search approach may be the likely effect of such a method.
Perceived weaknesses. Curtailing the duration of quality is used as the basis for conclusions. As a
the review process runs the risk of introducing consequence, their findings cannot be used to
bias. This is true for any review process, but this recommend policy/practice.
risk is accentuated when measures are fast-tracked
or even sidestepped. Limiting the time taken to Example. Weeks, L. C. & Strudsholm, T. A scoping
search may result in publication bias, limiting review of research on complementary and alternative
appraisal or quality assessment may place a medicine (CAM) and the mass media: looking
disproportionate emphasis on poorer quality back, moving forward. BMC Complementary and
research, while a lack of attention to synthesis may Alternative Medicine 2008, 19(8), 43.
overlook inconsistencies or contradictions.
Producing the evidence within a rapid timescale
State-of-the-art review
has to be offset against this risk of increased bias.
Documenting the methodology and highlighting Description. State-of-the-art reviews are specifically
its limitations is one way of militating against such mentioned by scope notes of the MeSH database
biases. Furthermore, inadequate attention to the for the entries under Review, Literature as a
question to be addressed or the quantity and Topic33 and Review [Publication Type].46 As such
quality of literature that exists in a topic may result they represent a subtype of the more generic
in a very precise answer to the wrong question or ‘Literature Review’. They are characterized as
an inconclusive answer to an ill-conceived follows: ‘State-of-the-art reviews tend to address
question. more current matters’47 in contrast to the
combined retrospective and current approaches of
Example. Lacey Bryant, S. & Gray, A. the “literature review” ’. The review may offer new
Demonstrating the positive impact of information perspectives on an issue or highlight an area in
support on patient care in primary care: a rapid need of further research.
literature review. Health Information and Libraries
Journal 2006, 23(2), 118–25. Perceived strengths. State-of-the-art reviews are
of considerable value for those new to an area or
for those seeking to identify potential opportunities
Scoping review
for contemporary research. Instead of having to
Description. This type of review provides a read multiple articles describing specific
preliminary assessment of the potential size and developments, the reader can derive a feel for the
scope of available research literature. It aims to quantity and main characteristics of a topic from
identify the nature and extent of research evidence a single review article. An entire body of
(usually including ongoing research). publishing output has developed around these
acknowledged advantages both within Medicine
Perceived strengths. Scoping reviews are able to (e.g. the Annual Review of ... Neurology, Cardiology
inform policymakers as to whether a full systematic etc.) and within Library and Information Science
review is needed. They share several characteristics (e.g. the Annual Review of Information Science and
of the systematic review in attempting to be Technology and the various New Review titles,
systematic, transparent and replicable. including New Review of Information and
Library Research and New Review of Academic
Perceived weaknesses. Scoping reviews cannot Librarianship).
usually be regarded as a final output in their own
right, primarily because limitations in their rigour Perceived weaknesses. Limitations of the state-
and limitations in their duration mean that they of-the-art review are common to any ‘cross-
hold the potential for bias. In particular, they sectional’ method of surveying a field. Such
typically do not include a process of quality methods are time bound and may distort the
assessment. There is thus a danger that the overall picture of development of a field. For
existence of studies rather than their intrinsic example, if a topic has been extensively covered
Perceived strengths. Systematic reviews seek to Perceived weaknesses. While the initial search
draw together all known knowledge on a topic process may meet the exacting requirements of a
area. In recent years, with the establishment of systematic review, the subsequent critical review
organizations such as Campbell Collaboration may be prone to some of the limitations of the
and the Cochrane Qualitative Methods Group, traditional review. Without explicit inclusion and
there has been a noticeable shift towards the exclusion criteria and a clearly defined process of
inclusion of a wider range of study designs synthesis, the result may be a subjective selection
incorporating quantitative, qualitative and mixed of research to support a particular line of argument.
method studies.
Example. Carroll, L. J., Cassidy, J. D., Peloso, P. M.,
Perceived weaknesses. Restricting studies for Garritty, C. & Giles-Smith, L. WHO Collaborating
inclusion to a single study design such as Centre Task Force on mild traumatic brain injury.
randomized controlled trials, as practised in the Systematic search and review procedures: results
early years of the Cochrane Collaboration, can of the WHO Collaborating Centre Task Force on
limit the application of this methodology to mild traumatic brain injury. Journal of Rehabilitation
providing insights about effectiveness rather Medicine 2004, 43(Suppl.), 11–4.
than seeking answers to more complex search
questions; for example, why a particular intervention
Systematized review
is effective.
Description. Systematized reviews attempt to
Example. Weightman, A. L. & Williamson, J. The include one or more elements of the systematic
value and impact of information provided through review process while stopping short of claiming
library services for patient care: a systematic that the resultant output is a systematic review.
review. Health Information and Libraries Journal They may identify themselves parenthetically as a
2005, 22(1), 4–25. ‘systematic’ review. Systematized reviews are
reviews. The most obvious, and least satisfactory, overall awareness of the general technique and
is to rely on the labels and terminology used by the its potential benefits in providing a synthesis of
authors themselves. However, the considerable literature in a particular area, possibly with a view
variation in use of terminology, with both overlap to informing local practice. Secondly, calls are
and the difficulty in distinguishing between review increasingly being made at a national and
types, make such an approach unfeasible. A international level to develop the LIS evidence
second approach, advanced by the Government base, both in terms of primary/original research
Social Research’s Rapid Evidence Assessment and secondary/synthesized research in the form of
(REA) toolkit,44 focuses on inputs, specifically the reviews. Finally, there is mounting recognition
time taken to complete each type of review. Again, that the systematic consideration of previously
this was not considered appropriate in the published research may be a more effective way of
development of the typology presented in this targeting funds than simply commissioning
paper, as the time taken for a review depends on further primary research.
numerous factors, beyond the type of review. The typology presented in this paper provides
These include, but are not exclusive to, the an explicit basis for those involved in commission-
resources available, the quantity and quality of ing reviews, and those involved in delivering to
the literature, and the expertise or experience of such commissions, to gain a clear understanding
the reviewers. The final approach, in contrast to a of what is being requested and the resources that
methodology based on inputs, is to focus on the will be required to meet the specification.
tangible processes required in completing a review,
namely search, appraisal, synthesis and analysis,
Using reviews
embodied in the SALSA framework. This
approach relies neither on terminology nor on For the LIS evidence base to be transformed from
inputs but, in a manner congruent with the its current embryonic state to provide a firm
principles of evidence-based practice, considers foundation on which to base professional practice
the nature and extent of the review processes as requires expansion of both the breadth and
embodied in the description of the methodology. quality of the evidence being created. Whilst LIS
Consideration of the 14 review types and practitioners may be more predisposed than other
associated methodologies reveals that, whilst labels professions to the inherent advantage of using
may supply a pragmatic ‘shorthand’ for authors, pre-existing evidence, the potential panacea of
there are frequent inconsistencies or overlaps reviews and their associated methodologies
between the descriptions of nominally different remains a distant prospect. However, whether the
review types. Currently, there is no internationally evidence takes the form of primary or secondary
agreed set of discrete, coherent and mutually studies, it is equally important to undertake an
exclusive review types. The authors contend that appraisal of quality. This should consider both
the only pragmatic way to identify to which of its robustness (validity and reliability) and its
these various types a particular review belongs is relevance to the local context (applicability). Work
to inspect closely the four main processes associated on developing critical appraisal skills to inform
with that review’s development. In this way, a the practice of LIS workers was first reported in
clearer understanding of the distinguishing 1999.49 At that time, it was noted that information
features of each review type can be built up within workers face significant barriers with regard to
the systematic review community through both lack of knowledge of research methodology and
direct comparison and emerging precedent. statistics. From such small beginnings has grown a
more widely spread awareness of critical appraisal
and ongoing development of these skills. One
The rising value of reviews
study undertaken in the north-west of England
Several drivers are leading to a growth in the reports that, of 55 librarians who responded to
number of reviews published within, and beyond, their survey, a significant 86% had attended
the health LIS sector. Firstly, there is a greater appraisal skills training.50
Tools for appraising review methodologies have evidence base remain to be addressed by now accepted
been published by the Critical Appraisal Skills ‘orthodox’ systematic review methods. In such
Programme (CASP).51 However, the diversity of instances, a LIS worker may choose to develop
review methodologies characterized in the a systematic review, either singularly or within
typology above suggests that it is inappropriate to a team.
try to assess reviews using a ‘one size fits all’
approach. Mapping existing and future types of
Developing reviews
health information review against the SALSA
framework can assist LIS workers in identifying A growing literature attests to the roles LIS workers
the inputs, processes, strengths and deficiencies can undertake within the review development
that accompany each review type. process. This is particularly apparent in relation to
For the scale of work to be manageable, systematic reviews, perhaps because their clear
particularly within the often uncharted territories and structured methodology makes them prime
of LIS research, reviews generally seek to address candidates for input from information specialists.
tightly defined research questions. This may not The US Medical Library Association (MLA) has
always be appropriate to the type of reviews most acknowledged that librarians are increasingly
needed by the LIS profession. Thus, it is important invited to join research teams to provide an expert
that practitioners recognize and develop an searching component to the team.52 However,
awareness of the broader context and evidence although LIS professionals have adopted diverse
base within which a particular review exists. roles within the systematic review context, as the
Judgement could then be made on the basis of MLA statement suggests, these are primarily of a
‘fitness for purpose’ and not against a single ‘gold supportive nature. Typically, LIS professionals
standard’ of what a review should or should not contribute to the data collection and data
be. A case in point is the state-of-the-art review, management phases of a review, withdrawing from
that is, a review that offers new perspectives on an the team once the literature is identified and
issue or highlights areas in need of further obtained in readiness for its appraisal, extraction
research. If an expert in a given field has written a and synthesis. One such collaborative venture is
state-of-the-art review, it is likely to provide a fairly described by Swinkels et al.,53 who outline how the
accurate representation of current knowledge and relationship between academic, clinician and
future priorities for research. Depending on the librarian can lead to iterative searching.
extent of communication within the chosen field, it Elsewhere, Harris reports on extending the role of
may also capture an informal consensus among expert searcher to involve the LIS worker in
fellow experts. Nevertheless, by virtue of its preparing the outputs from reviews.54 Grant
currency, a state-of-the-art review may in actuality et al.55 extol the benefits of LIS worker involvement
be reporting a supposed gap in the evidence that is in a review, citing the opportunities for capability
already being, or has already been, addressed, but building to be gained by all involved through peer
has yet to be published. Clearly, a reliance exclusively teaching and learning at all stages of a review
on published literature in this specific context cycle. They further identify the potential to
provides a significant limitation to the useful- transfer these skills within an evidence-based
ness of an otherwise valuable type of review. library and information practice context. Beverley
‘Fitness for purpose’ requires that, if such a et al. report a systematic review where information
review is being used in the context of planning professionals participated effectively in all aspects
and commissioning research, its use be supple- of preparing a review56—a model repeated in a
mented by interrogation of research registers more recent review by Booth et al.21 Brettle
and contact with experts. exemplifies a natural progression of these latter
Notwithstanding the potential of many of the approaches in describing how a group of health
other types of review identified in the typology pre- librarians in the north-west of England are using
sented, the relative infancy of review activities within the process of undertaking a systematic review on
the LIS sector means that significant gaps in the the most effective methods of evaluating clinical
7 Alonso, P. G. Conservation and circulation in map 25 Childs, S., Blenkinsop, E., Hall, A. & Walton, G. Effective
libraries: a brief review. Geography and Map Division e-learning for health professionals and students—barriers
Bulletin 1968: 74, 15 – 8. and their solutions. A systematic review of the literature—
8 Skinner, R. E. & Marcotte, J. M. Bibliographic instruction findings from the HeXL project. Health Information and
in the health sciences: an historical review. Medical Libraries Journal 2005, 22(Suppl. 2), 20–32.
Reference Services Quarterly 1982, 1, 53 – 76. 26 Fanner, D. & Urquhart, C. Bibliotherapy for mental health
9 Booth, A. Systematic reviews of health information service users. Part 1: a systematic review. Health
services and systems. Health Information and Libraries Information and Libraries Journal 2008, 25, 237–52.
Journal 2001, 18, 60 – 3. 27 Grant, M. J. The role of reflection in the library and
10 Koufogiannakis, D. & Crumley, E. Research in librarianship: information sector: a systematic review. Health Information
issues to consider. Library Hi Tech 2006, 24, 324 – 40. and Libraries Journal 2007, 24, 155–66.
11 Oxford English Dictionary. Oxford English Dictionary. 28 Koufogiannakis, D. & Wiebe, N. Effective methods for
2008. Available from: http://dictionary.oed.com/ teaching information literacy skills to undergraduate
(accessed 4 March 2009). students: a systematic review and meta-analysis.
12 Ankem, K. Evaluation of method in systematic reviews and Evidence Based Library and Information Practice 2006,
meta-analyses published in LIS. Library and Information 1, 3–43.
Research 2008, 32, 91–104. 29 Wagner, K. C. & Byrd, G. D. Evaluating the effectiveness of
13 Walton, G. Health Information and Libraries Journal: clinical medical librarian programs: a systematic review of
Strategy for 2007– 2010. Oxford: Wiley-Blackwell, 2006. the literature. Journal of the Medical Library Association
14 Davies, K. The information-seeking behaviour of doctors: 2004, 92, 14–33.
a review of the evidence. Health Information and Libraries 30 Weightman, A. L. & Williamson, J. The value and impact
Journal 2007, 24, 78 – 94. of information provided through library services for patient
15 Rossall, H., Boyes, C., Montacute, K. & Doherty, P. care: a systematic review. Health Information and Libraries
Developing research capacity in health librarians: a review Journal 2005, 22, 4–25.
of the evidence. Health Information and Libraries Journal 31 Research and Innovative Technology Administration
2008, 25, 159–74. (RITA). Principles for the Organization of the
16 Ward, R., Stevens, C., Brentnall, P. & Briddon, J. The Transportation Research Thesaurus. Available from:
attitudes of health care staff to information technology: http://ntl.bts.gov/tools/trt/trt_principles.html
a comprehensive review of the research literature. Health (accessed 9 March 2009).
Information and Libraries Journal 2008, 25, 81– 97. 32 Miles, M. B. & Huberman, A. M. Qualitative data analysis:
17 Hall, A. & Walton, G. Information overload within the a sourcebook of new methods. London: Sage, 1984.
health care system: a literature review. Health Information 33 National Center for Biotechnology Information. Review
and Libraries Journal 2004, 21, 102 – 8. Literature as Topic. 2005. Available from:
18 Boulos, M., Kamel, N., Hetherington, L. & Wheeler, S. http://www.ncbi.nlm.nih.gov/sites/
Second Life: an overview of the potential of 3-D virtual entrez?cmd = Retrieve&db=mesh&dopt=Full&list_uids=
worlds in medical and health education. Health 68012196 (accessed 4 March 2009).
Information and Libraries Journal 2007, 24, 233 – 45. 34 EPPI-Centre. Descriptive Mapping. 2006. Available from:
19 Ankem, K. Use of information sources by cancer patients: http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=
results of a systematic review of the research literature. 175&language=en-US (accessed 4 March 2009).
Information Research 2006, 11(3), Available from: 35 Subject Centre for Information and Computer Sciences,
http://InformationR.net/ir/11-3/paper254.html. Outputs—meta-analysis, meta-synthesis and guidelines.
20 Beverley, C. A. & Winning, M. A. Clinical librarianship: 2007. Available from: http://www.ics.heacademy.ac.uk/
a systematic review of the literature. Health Information resources/rlos/systematic_review/Unit4_systematic_review/
and Libraries Journal 2003, 20(Suppl. 1), 10 – 21. page_01.htm (accessed 4 March 2009).
21 Booth, A., Carroll, C., Papaioannou, D., Sutton, A. & 36 Oliver, S., Harden, A., Rees, R., Shepherd, J., Brunton, G.,
Wong, R. Applying the findings of a systematic review of Garcia, J. & Oakley, A. An emerging framework for
workplace-based e-learning. Health Information and integrating different types of evidence in systematic
Libraries Journal 2009, 26, 4–21. reviews for public policy. Evaluation & the Health
22 Brettle, A. Information skills training: a systematic review Professions 2005, 11, 428–66.
of the literature. Health Information and Libraries Journal 37 Oxman, A. D., Cook, D. J. & Guyatt, G. H. How to use an
2003, 20(Suppl. 1), 3 – 9. Overview. 1994. Available from: http://www.cche.net/
23 Brettle, A. Evaluating information skills training in health usersguides/overview.asp (accessed 4 March 2009).
libraries: a systematic review. Health Information and 38 Booth, A. Brimful of STARLITE: toward standards for
Libraries Journal 2007, 24, 18 – 37. reporting literature searches. Journal of the Medical Library
24 Brown, C. The information trail of the ‘Freshman 15’—a Association 2006, 94, 421–9.
systematic review of a health myth within the research and 39 Mulrow, C. D. & Cook, D. J. Systematic reviews: synthesis
popular literature. Health Information and Libraries of best evidence for health care decisions. Philadelphia, PA:
Journal 2008, 25, 1–12. American College of Physicians, 1998.
40 Booth, A. The neglected voice: is there a role for qualitative 49 Booth, A. & Brice, A. Clear-cut?: facilitating health
systematic reviews in EBLIP? Transforming the profession librarians to use information research in practice.
through evidence-based library and information fractice. Health Information and Libraries Journal 2003, 20(1s),
Evidence-Based Library and Information Practice, 4th 45–52.
International Conference, 6 –11 May 2007. 2007. 50 Maden, M. Attitudes and barriers to critical appraisal
Available from: http://eblip4.unc.edu/program.html facilitation among UK healthcare librarians. Impact and
(accessed 4 March 2009). Influence: Evolving to Succeed. Cardiff, 2008. Available
41 Booth, A. Counting what counts: performance from: http://www.cilip.org.uk/specialinterestgroups/
measurement and evidence-based practice. Performance bysubject/health/events/conference/HLG%
Measurement and Metrics 2006, 7, 63 –74. 20Conference%202008%20Presentations.htm.
42 Noyes, J. et al. Chapter 20: Qualitative research and 51 Public Health Resource Unit. Ten Questions to Help you
Cochrane reviews. Cochrane Handbook for Systematic Make Sense of Reviews. 2006. Available from:
Reviews of Interventions, version 5.0.1. September 2008. http://www.phru.nhs.uk/Doc_Links/S.Reviews
Available from: www.cochrane-handbook.org %20Appraisal%20Tool.pdf (accessed 4 March 2009).
(accessed 4 March 2009). 52 Medical Library Association. Medical Library Association
43 Centre for Reviews and Dissemination. Systematic Policy Statement: Role of Expert Searching in Health
Reviews: CRD’s Guidance for Undertaking Reviews in Sciences Libraries. 2007. Available from: http://
Health Care. 2009. Available from: 332005, http:// www.mlanet.org/resources/expert_search/
www.york.ac.uk/inst/crd/SysRev/!SSL!/WebHelp/ policy_expert_search.html (accessed 4 March 2009).
SysRev3.htm (accessed 4 March 2009). 53 Swinkels, A., Briddon, J. & Hall, J. Two physiotherapists.
44 Government Social Research. REA Toolkit: Rapid one librarian and a systematic literature review:
Evidence Assessment Toolkit Index. 2008. Available from: collaboration in action. Health Information and Libraries
http://www.gsr.gov.uk/professional_guidance/rea_toolkit/ Journal 2006, 23, 248–56.
(accessed 11 March 2009). 54 Harris, M. R. The librarian’s roles in the systematic review
45 Butler, G., Deaton, S., Hodgkinson, J., Holmes, E. & process: a case study. Journal of the Medical Library
Marshall, S. Quick but not Dirty: Rapid Evidence Association 1993, 93, 81–7.
Assessments as a Decision Support Tool in Social Policy. 55 Grant, M. J., Howarth, M., Leigh, J. & Murray, C.
2005. Available from: http://www.gsr.gov.uk/downloads/ Capability building within a systematic review: the inimita-
new_research/archive//quick_not_dirty05.pdf ble role of the librarian. Impact and Influence: Evolving to
(accessed 4 March 2009). Succeed. HLG Conference. Cardiff, 2008. Available from:
46 National Library of Medicine. Medical Subject Headings. http://www.cilip.org.uk/specialinterestgroups/bysubject/
Publication Characteristics (Publication Types): Scope health/events/conference/HLG%20Conference%202008%
Notes. 2005. Available from: http://www.nlm.nih.gov/ 20Presentations.htm.
mesh/pubtypes2006.html (accessed 4 March 2009). 56 Beverley, C. A., Booth, A. & Bath, P. A. The role of the
47 Higgins, J. P. T. & Green, S. Cochrane Handbook for information specialist in the systematic review process: a
Systematic Reviews of Interventions, version 5.0.1 health information case study. Health Information and
(updated September 2008). 2008. Available from: Libraries Journal 2003, 20, 65–74.
www.cochrane-handbook.org (accessed 4 March 2009). 57 Brettle, A. Systematic reviews and evidence based library
48 Bond, K., Horvath, T., Harvey, K., Wiysonge, C. S. & and information practice. Evidence Based Library and
Read, J. S. The Cochrane Library and mother-to-child Information Practice 2009, 4, 43–50.
transmission of HIV: an umbrella review. Evidence-Based
Child Health: A Cochrane Review Journal 2007, 2, 4–24. Received 18 February 2009; Accepted 4 March 2009