USING GOOGLE SCHOLAR FOR SYSTEMATIC REVIEW

From review of this issue I would conclude that:

(1) neither database (Google Scholar (GS), PUBMED) is sufficient for optimal discovery of all
highly relevant content in a topical medical search;
(2) that there are several myths and misunderstanding concerning their true differences and
divergent focus and specialization;
(3) that there is data to support the contention that Google Scholar can indeed be used effectively
for systematic review provided one understands its unique modes of operation and execution;
(4) that one is better served seeing GS and PUBMED more as complimentary than as exclusive
of one another; and that
(5) an optimal search process would involve multiple general search databases coupled with
specialized collections, with searches being executed best in a highly articulated form (in GS for
instance, using scope qualifiers in Boolean expressions). Below I provide the basis for these
conclusions:

To begin with, the two search databases, Google Scholar and PUBMED, reflect different
relevancy algorithms: PubMed uses algorithms based on MeSH terms, with the most recent
articles reported at the top of the list (which likely have not had adequate time to be
appropriately cited), while in stark contrast, Google Scholar's proprietary algorithms (first
released in 2004 in beta) have been found to favor the number of citations as an important
criterion in the initial list of articles, with date of publication not an important criterion [1,2].

GOING HEAD-TO-HEAD: GOOGLE SCHOLAR versus PUBMED

Several studies [3,4] suggest that Google Scholar searches compare favorably with PubMed
searches but have both advantages and disadvantages. Another recent study [5], building on
these foundations, focused on content relevance and article quality and suggests that the Google
Scholar search engine retrieves more relevant, higher quality articles. And in a comparative
study as to locating primary literature to answer drug-related questions, no significant differences
were identified in the number of target primary literature articles located between the Google
Scholar versus PUBMED databases [6]. In addition, as to a single focused query (risk factors for
sarcoma) Google Scholar resulted in a higher sensitivity (proportion of relevant articles, meeting
the search criteria), compared to PubMed which resulted in a higher specificity (proportion of
lower quality articles not meeting the criteria, that are not retrieved) [7].

Similarly, one of the most recent and comprehensive robust studies, this one from the University
of Rouen [8], examined explicitly the core question of "Is the coverage of Google Scholar
enough to be used alone for systematic reviews", performing a study to assess the coverage of
GS specifically for the studies included in systematic reviews, and to evaluate if GS was
sensitive enough to be used alone for systematic reviews, in order ultimately to assess the
percentage of studies which could have been identified by searching only GS; there were 738
original studies included in a specially constructed gold standard database. The results: GS
retrieved all 738 studies (100% hits) mined from 29 systematic reviews, allowing the authors to
conclude that "The coverage of GS for the studies included in the systematic reviews is 100%. If
the authors of the 29 systematic reviews had used only GS, no reference would have been
missed. With some improvement in the research options, to increase its precision, GS could
become the leading bibliographic database in medicine and could be used alone for systematic
reviews".

What this entails is that despite GS not covering all the medical literature, nonetheless its
coverage of the studies of sufficient quality or relevance to be included in a systematic review
was complete, so that if the authors of these 29 systematic reviews had relied only on GS, they
would have obtained the very same results. In contrast, it's been shown that the recall ratios of
Medline RCTs only ranges between 35% and 56% [9,10]. This is in essential agreement with
still another recent study [5] where PubMed and Google Scholar searches were compared by
evaluating the first 20 articles recovered for four clinical questions for relevance and quality, GS
provided more relevant results that PubMed (although the difference was not significant),
serving as another reminder that we should not overestimate the precision of PubMed in real life
[8]).

LOOKING FORWARD
We await further enhancements to GS to provide reliable advanced search functions, a controlled
vocabulary, and improved scope of coverage and currency, but even in the latest instantiation GS
performs at a respectable level of recall and precision, and can be enhanced with judicious
Boolean expressions and some undocumented qualifiers. All told, as the Rouen study concludes:
"the coverage of GS is much higher than previously thought for high quality studies ". (And note
that other comparisons have found GS more than credible; in a comparison with Web of
Science/WoS [11], the study authors concluded that: "since its inception, GS has shown
substantial expansion, and that the majority of recent works indexed in WoS are now also
retrievable via GS"). And I would add one further caution: despite the correct claim of many
advanced search features being absent from GS but present in PUBMED, nonetheless this has
less relevancy that one might believe: only 7% of respondents used these features in their
searches for the Canadian study [12], only 37% used controlled vocabularies, and only 20% used
filters such as the Clinical Queries feature in PubMed [13,14]. Therefore, in the real-world rather
than the theoretical domain, the two search technologies are less far apart than the advanced
features suggested, when we look at actual usage patterns.

But the debate will certainly continue for some while, with divergent opinions [15,16]. But it is
increasingly clear from a critical review of the data to date that the two databases should be
considered complimentary and not mutually exclusive, each with unique advantages and
tradeoffs: thus, noting that recent evidence suggests Google Scholar may have closed the gap
between itself and PUBMED, and that it is now often leading in searches (with one family of
journals reporting that 60% of their traffic is coming from Google Scholar, ahead of PUBMED
and other traditional medical databases), University of Utah researchers [17] assessed efficiency
and completeness of searching for known moderate and high quality RCTs in PubMed versus
Google Scholar, finding that each database consistently identified one of the two highest quality
studies, but neither database identified both, yet the difference search time was nearly three-fold
(to accomplish the search by experienced researchers, it search time was 63 minutes for GS but
194 minutes for PUBMED, without the later providing any superior results). This again reflects
what I have called the INCOMPLETENESS THEOREM OF MEDICAL SEARCH: namely that
no single search is sufficient to identify all relevant quality studies, cross-confirmed in still
another recent study where Canadian researchers [12] evaluated the recall (proportion of relevant
articles found) and precision (ratio of relevant to nonrelevant articles) of searches performed in
PubMed and Google Scholar, with primary studies included in the systematic reviews serving as
the reference standard for relevant articles, finding that for For quick clinical searches, compared
with PubMed, the average search in Google Scholar retrieved twice as many relevant articles
(PubMed: 11%; Google Scholar: 22%), with precision being similar in both databases (PubMed:
6%; Google Scholar: 8%). And note that it would be tempting but erroneous to attribute the two-
fold greater retrieval manifold as due to differences in content coverage, since 78% of the tested
articles were available in BOTH databases.

These and other studies assessing different medical databases have demonstrated that no single
search engine provides all the related articles, full capturing of the complete body of available
literature on a subject requiring searches over multiple databases, depending on the topic. Thus, a
much more comprehensive search would include cross-spectrum searching[18], and as an
example I note that I myself use an extensive resource collectivity of approximately 18 databases
and tools including ones for specialized content: see "METHODOLOGY FOR THIS REVIEW"
below).

As to peer-review, often claimed a major factor that distinguishes Google Scholar (unrestricted)
from PUBMED, in fact despite the widely held but erroneous belief that PubMed will only
consider Peer Reviewed literature, it is explicitly stated on their website that this is not the case:
Most journals in PubMed are peer-reviewed or refereed. Non-editorial journal-staff review
original articles before the articles are accepted for publication. Criteria for peer review and the
qualifications of peers or referees vary among publishers. We have no list of peer-
reviewed/refereed journals in PubMed; and you cannot limit your search to peer-reviewed
journals using PubMed [http://www.nlm.nih.gov/services/peerrev.html] [19].

USING ARTICULATED/SMART SEARCHES

Finally, it pays to learn how to execute articulated ("smart") searches in GS: thus, in answer to a
question in another topic concerning searching for all systematic reviews and meta-analyses
concerning HRV (heart rate variability), I advised the Google Scholar smart search (besides a
MeSH-enriched PUBMED search):

insubject:"heart rate variability" intitle:("systematic review" | meta-analysis)

or the somewhat more permissive relaxed smart search:

insubject:"heart rate variability" intext:("review" | meta-analysis)

leveraging the power of the scope qualifiers "insubject", "intitle" and "intext" when coupled with
appropriate Boolean operators. It also pays to remember that GS is an "opportunistic" search
engine, as it will try to data-mine any resources that could be of relevance rather than honing to
the more narrow constraints of a formal PUBMED search (often providing riches not otherwise
easily uncovered, so that its claimed lesser precision is not at all necessarily a disadvantage, as
some of the discovered resources (like dissertations, commissioned monographs, peer-reviewed
CMEs, etc.) could themselves - as I have often found - contain bibliographical references to
invaluable materials not located through PUBMED, that could greatly enrich the quality of any
paper using its technology.

METHODOLOGY OF THE REVIEW

A search of the PUBMED, Cochrane Library / Cochrane Register of Controlled Trials,
MEDLINE/MedlinePlus, EMBASE, AMED (Allied and Complimentary Medicine Database),
CINAHL (Cumulative Index to Nursing and Allied Health Literature), PsycINFO, ISI Web of
Science (WoS), BIOSIS, LILACS (Latin American and Caribbean Health Sciences Literature),
ASSIA (Applied Social Sciences Index and Abstracts), SCEH (NHS Evidence Specialist
Collection for Ethnicity and Health), and scope-qualified Boolean searches submitted to Google
Scholar and SLIM, was conducted without language or date restrictions, and updated again
current as of date of publication, with systematic reviews and meta-analyses extracted separately.
Search was expanded in parallel to include just-in-time (JIT) medical feed sources as returned
from Terkko (provided by the National Library of Health Sciences - Terkko at the University of
Helsinki). Unpublished studies were located via contextual search, and relevant dissertations
were located via NTLTD (Networked Digital Library of Theses and Dissertations), OpenThesis
or Proquest. Sources in languages foreign to this reviewer were translated by language
translation software.

REFERENCES
1. Beel, J. & Gipp, B. Google Scholar's ranking algorithm: the impact of citation counts (an
empirical study). Proceedings of the 3rd International Conference on Research Challenges in
Information Science 2009a, 439446.
2. Beel, J. & Gipp, B. Google Scholar's Ranking Algorithm: The impact of articles' age (an
empirical study). Proceedings of the 6th International Conference on Information Technology:
New Generations 2009b, 160164.
3. Shultz, M. Comparing test searches in PubMed and Google Scholar. JMIA 2007, 95, 442445.
4. Anders, M. E. & Evans, D. P. Comparison of PubMed and Google Scholar literature searches.
Respiratory Care 2010, 55, 578583.
5. Nourbakhsh E, Nugent R, Wang H, Cevik C, Nugent K. Medical literature searches: a
comparison of PubMed and Google Scholar. Health Info Libr J 2012; 29(3):214-22.
6. Freeman MK, Lauderdale SA, Kendrach MG, Woolley TW. Google Scholar versus PubMed
in locating primary literature to answer drug-related questions. Ann Pharmacother 2009;
43(3):478-84.
7. Mastrangelo, G. , Fadda, E. , Rossi, C. , Zamprogno, E. , Buja, A. & Cegolon, L. Literature
search on risk factors for sarcoma: PubMed and Google Scholar may be complementary sources.
BMC Research 2010, 3, 131134.
8. Gehanno JF, Rollin L, Darmoni S. Is the coverage of Google Scholar enough to be used alone
for systematic reviews. BMC Med Inform Decis Mak 2013; 13:7.
9. Trp JC, Schulte J, Antes G: Nearly half of dental randomized controlled trials published in
German are not included in Medline. Eur J Oral Sci 2002, 110:405-411.
10. Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M: A comparison of hand searching
versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med 2002,
21:1625-1634.
11. de Winter JCF, Zadpoor AA, Dodou D. The expansion of Google Scholar versus Web of
Science: a longitudinal study. Scientometrics 2014; 98(2): 1547-1565.
12. Shariff SZ, Bejaimal SA, Sontrop JM, et al. Retrieving clinical evidence: a comparison of
PubMed and Google Scholar for quick clinical searches. J Med Internet Res 2013; 15(8):e164.
13. Shariff SZ, Bejaimal SA, Sontrop JM, Iansavichus AV, Weir MA, Haynes RB, et al.
Searching for medical information online: a survey of Canadian nephrologists. J Nephrol
2011;24(6):723-732.
14. Shariff SZ, Sontrop JM, Haynes RB, Iansavichus AV, McKibbon KA, Wilczynski NL, et al.
Impact of PubMed search filters on the retrieval of evidence by physicians. CMAJ 2012 Feb
21;184(3):E184-E190.
15. Bramer WM, Giustini D, Kramer BM, Anderson P. The comparative recall of Google
Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of
searches used in systematic reviews. Syst Rev 2013; 2:115.
16. Boeker M, Vach W, Motschall E. Google Scholar as replacement for systematic literature
searches: good relative recall and precision are not enough. BMC Med Res Methodol 2013;
13:131.
17. Thiese M, Effiong A, Passey D, Ott U, Hegmann K. Pubmed vs. Google Scholar: A Database
Arms Race? BMJ Qual Saf 2013;22:A33.
18. Zheng B, Zheng W, Zhu Y, Guo C, Wu W, Chen C. Are PubMed alone and English
literature only enough for a meta-analysis? Ann Oncol 2013; 24(4):1130.
19. Kejariwal D, Mahawar KK. Is Your Journal Indexed in PubMed? Relevance of PubMed in
Biomedical Scientific Literature Today. WebmedCentral MISCELLANEOUS
2012;3(3):WMC003159.

Google scholar for systematic reviews: what limit on search returns?. Available from:
https://www.researchgate.net/post/Google_scholar_for_systematic_reviews_what_limit_on_sear
ch_returns [accessed Jul 30, 2017].

I personally adopted another way to search google scholar efficiently as a crude search of it can
be very time consuming and fruitless.
1- First I extract the included articles I found in Medline and Scopus.
2- I search the extracted articles in Google Scholar and the citing articles (can be retrieved by
"cited by" link of the articles) can provide you with many more relevant articles.

This way you can find many more relevant evidence especially non-English ones and as
Christopher mentioned theses, etc.

I think this way is a compromise between the high sensitivity (providing much much more
articles than other databases) and the poor specificity (providing a huge number of irrelevant
articles) of Google Scholar.

Best wishes,

Ramin
Google scholar for systematic reviews: what limit on search returns?. Available from:
https://www.researchgate.net/post/Google_scholar_for_systematic_reviews_what_limit_on_sear
ch_returns [accessed Jul 30, 2017].

I have a methods paper on this exact subject in review at the moment and a practical application
manuscript also in review. Send me a message if you would like to see drafts. In summary (for
environmental sciences systematic reviews):

1. GS only displays the first 1,000 search results and the ordering of these results is not clear. GS
is therefore not suitable as a standalone resource and it's use in a SR is not in keeping with the
need for transparency and repeatability.

2. Title-only searches obviously yield fewer results, but a higher proportion is grey literature than
full-text results. So the utility of and ideal search settings in GS depend on what you are using
GS for: grey or academic. Grey literature occurs mostly around the 200-300th results (pages 20-
30) so searching the first 50-100 results (as often seen in env management SRs) is a poor practice
if grey literature is the target.

3. We have developed a method to download and extract as citations these first 1,000 results,
making the search process transparent and updatable. (request our methods paper for this
process)

4. Similar searches in GS and Web of Science yield poorly overlapping results (especially where
results are far larger than the first 1,000 displayed). GS is therefore a useful addition to
traditional searching.

We conclude that GS is a useful addition to traditional SR searches, and that new methods allow
transparency and updatability using GS.

Thanks,

Neal

Google scholar for systematic reviews: what limit on search returns?. Available from:
https://www.researchgate.net/post/Google_scholar_for_systematic_reviews_what_limit_on_sear
ch_returns [accessed Jul 30, 2017].