PHD Thesis - Benefits and Harms of The HPV Vaccines - Lars Jørgensen MD (Jcl649) - 12 November 2018

GRADUATE SCHOOL OF HEALTH AND MEDICAL SCIENCES
UNIVERSITY OF COPENHAGEN
PhD Thesis
Benefits and Harms of the Human Papillomavirus

(HPV) Vaccines
Lars Jørgensen MD
THIS THESIS WAS SUBMITTED TO

THE GRADUATE SCHOOL OF HEALTH AND MEDICAL SCIENCES
UNIVERSITY OF COPENHAGEN ON 12 NOVEMBER 2018
Benefits and harms of the HPV vaccines • Lars Jørgensen MD • 12 November 2018
Name of department Nordic Cochrane Centre
Institutnavn Nordisk Cochrane Center
Author Dr Lars Jørgensen MD
Title Benefits and harms of the human papillomavirus (HPV) vaccines
Titel Gavnlige og skadelige virkninger af HPV-vaccinerne
Subject description This thesis investigates the benefits and harms of the HPV vaccines: we in-
dexed the vaccines’ study programmes; obtained, assessed and synthesised
data from their clinical study reports; compared the available reports with
corresponding study documents; and applied and evaluated Cochrane’s risk
of bias tool for our critical appraisal of the documents.
Principal supervisor Prof Peter C Gøtzsche DrMedSci
Co-supervisor Dr Tom Jefferson MD MSc MRCGP FFPHM
Submitted 12 November 2018
Word count 5,971
Page 2 of 637
Table of contents
Preface and acknowledgements 4

Summary in English 6
Resumé på dansk [summary in Danish] 7
Introduction 8
Objectives 12
Description of research projects 13
Summary of results 15
Conclusions and perspectives for further research 21
Abbreviations 22
References 23
Paper 1: Evaluation of Cochrane’s risk of bias tool 31
Paper 2: Index of the HPV vaccine study programmes 95
Paper 3: Challenges of an independent HPV vaccine assessment 139
Paper 4: Systematic review of the HPV vaccines’ clinical study reports 264
Paper 5: Comparison of HPV vaccine study documents 545
Page 3 of 637
Preface and acknowledgements
This thesis is a synopsis based on the following five research papers:
1. Jørgensen L, Paludan-Müller AS, Laursen DRT, Savović J, Boutron I, Sterne JAC, Higgins JPT and
Hróbjartsson A. Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical
trials: overview of published comments and analysis of user practice in Cochrane and non-
Cochrane reviews. Syst Rev. 2016 May 10;5:80. https://doi.org/10.1186/s13643-016-0259-8.
2. Jørgensen L, Gøtzsche PC and Jefferson T. Index of the human papillomavirus (HPV) vaccine
industry clinical study programmes and non-industry funded studies: a necessary basis to ad-
dress reporting bias in a systematic review. Syst Rev. 2018 Jan 18;7(1):8.
https://doi.org/10.1186/s13643-018-0675-z.
3. Jørgensen L, Doshi P, Gøtzsche PC and Jefferson T. Challenges of independent assessment of

potential harms of HPV vaccines. BMJ. 2018 Sep 24;362:k3694.
https://doi.org/10.1136/bmj.k3694.
4. Jørgensen L, Gøtzsche PC and Jefferson T. Benefits and harms of the human papillomavirus
(HPV) vaccines: systematic review with meta-analyses of trial data from clinical study reports.
Submitted for publication. 2018.
Protocol: https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20170030.pdf.
Amendment: https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20171116.pdf.
5. Jørgensen L, Gøtzsche PC and Jefferson T. Benefits and harms of the human papillomavirus
(HPV) vaccines: comparison of clinical study reports with trial registry entries and journal pub-
lications. Submitted for publication. 2018.
Protocol: https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20180320.pdf.
Page 4 of 637
The following three papers support the thesis:
1. Jefferson T and Jørgensen L. Human papillomavirus vaccines, complex regional pain syndrome,
postural orthostatic tachycardia syndrome, and autonomic dysfunction - a review of the regu-
latory evidence from the European Medicines Agency. Indian J Med Ethics. 2017 Jan-
Mar;2(1):30-37. https://doi.org/10.20529/IJME.2017.006.
2. Jefferson T and Jørgensen L. Redefining the ‘E’ in EBM. BMJ Evid Based Med. 2018
Apr;23(2):46-47. https://doi.org/10.1136/bmjebm-2018-110918.
3. Jørgensen L, Gøtzsche PC and Jefferson T. The Cochrane HPV vaccine review was incomplete
and ignored important evidence of bias. BMJ Evid Based Med. 2018 Oct;23(5):165-168.
https://doi.org/10.1136/bmjebm-2018-111012.
Acknowledgements
I would like to thank my principal supervisor, Prof Peter C Gøtzsche, and my co-supervisor, Dr Tom
Jefferson, who gave me the opportunity to do this thesis. Both were great mentors and were excep-
tionally helpful, supportive and taught me a great deal about doing research and being a researcher. I
would also like to thank my other co-authors for their great collaboration—in particular, Prof Asbjørn
Hróbjartsson from whom I learned a great deal about research rigour and medical writing. Also, thanks
to Dr Erik von Elm from Cochrane Switzerland, who offered me an office space for my research stay.
Thanks to all my colleagues at Nordic Cochrane Centre for making my PhD an enjoyable experience.
Special thanks to Jannie Hedegaard and Frihild Askham for their excellent secretarial assistance. Finally,
thanks to my partner Allison Crank for reading various drafts and giving me helpful suggestions.
Financial support
The Nordic Cochrane Centre, which is funded by the Danish government, funded this PhD.
Page 5 of 637
Summary in English
The human papillomavirus (HPV) vaccines are global interventions given to healthy individuals to pre-
vent HPV-related diseases, such as cervical cancer. The vaccines are considered safe and effective by
health care authorities, but safety signals raised concerns whether reporting bias had influenced the
authorities’ considerations of benefits and harms. To address reporting bias, we investigated the ben-
efits and harms of the HPV vaccines with the Cochrane risk of bias tool, as well as an index of the study
programmes and the clinical study reports from the HPV vaccine study programmes and compared the
reports to the corresponding study documents (trial register entries and journal publications).
Paper 1: Our evaluation of the Cochrane risk of bias tool showed that many risk of bias domains
were judged at an unclear risk—possibly due to the fact that journal publications do not provide enough
information for assessments—and that the tool might be improved for clinical study reports (1).
Paper 2: The HPV vaccine study programmes demonstrated deficiencies and variability in the
availability of study data in trial register entries and journal publications. Only half of the completed
studies listed on ClinicalTrials.gov posted their results and a third of the studies were not published (2).
Paper 3: Our analysis of accessing the HPV vaccine trial data from clinical study reports showed
that regulatory policies are in need of change to increase transparency. For example, it was not possible
to get a complete set of unredacted clinical study reports of the HPV vaccines (3).
Paper 4: Our systematic review of 24 clinical study reports with nearly 100,000 participants
showed that at four years follow-up the HPV vaccines decreased HPV-related precursors to cancer and
treatment procedures but increased serious nervous system disorders and general harms. The trials
used biased designs and underreported harms, which prevented adequate harms assessment.
Paper 5: Our comparison of corresponding study documents showed no effect differences of
pooled estimates but demonstrated that the clinical study reports were quantitatively and qualitatively
superior with more outcome data and information that improved risk of bias judgments.
In conclusion, it is not clear to what extent the HPV vaccines benefits outweigh their harms, as
the study programmes and clinical study reports were influenced by reporting bias and biased trial de-
signs. The clinical study reports were superior study documents that contained better information for
risk of bias judgements. Clinical study reports should, therefore, be used for systematic reviews.
Page 6 of 637
Resumé på dansk [summary in Danish]

HPV-vaccinerne er globale interventioner, der gives til raske individer for at forhindre HPV-relaterede
sygdomme (f.eks. livmoderhalskræft). Sundhedsmyndighederne betragter vaccinerne som sikre og ef-
fektive, men ”reporting bias” påvirkede vaccinernes godkendelser. For at adressere denne bias, under-
søgte vi gavnlige og skadelige virkninger af HPV-vaccinerne ved brug af Cochrane’s bias-værktøj, et in-
deks af studieprogrammerne, de kliniske studierapporter fra HPV-vaccineprogrammerne og sammen-
lignede rapporterne med tilsvarende studieregistre og tidsskriftspublikationer:
Artikel 1: Vores evaluering af bias-værktøjet viste, at mange domæner blev bedømt havende
uklar bias-risiko - sandsynligvis fordi tidsskriftspublikationer ikke indeholder tilstrækkelig information
til bias-bedømmelser - og at værktøjet kunne forbedres til vurdering af kliniske studierapporter (1).
Artikel 2: Studieprogrammerne var mangelfulde og varierede ift. tilgængeligheden af data i
forsøgsregistre og tidsskriftpublikationer. Kun halvdelen af de fuldførte studier på ClinicalTrials.gov
havde tilgængelige resultater, og en tredjedel af studierne var ikke publiceret i tidsskrifter (2).
Artikel 3: Vores analyse af tilgængeligheden af de kliniske studierapporter viste, at processen
for at få adgang til rapporterne kan forbedres for at øge gennemsigtigheden, da det ikke var muligt at
få et komplet og uredigeret sæt kliniske studierapporter for HPV-vaccinerne (3).
Artikel 4: Vores systematiske oversigt af 24 kliniske studierapporter med næsten 100.000 del-
tagere viste, at HPV-vaccinerne efter fire års opfølgning nedsatte HPV-relaterede forstadier til kræft og
behandlingsprocedurer, men øgede alvorlige nervesystemsskadevirkninger og generelle skadevirknin-
ger. Studierapporterne brugte utilstrækkelige forsøgsdesign og underrapporterede alvorlige skadevirk-
ninger, hvilket forhindrede en adækvat vurdering af skadevirkningerne.
Artikel 5: Vores sammenligning af forskellige studiedokumenter viste ingen effektforskelle af
puljede estimater, men viste, at de kliniske studierapporter var kvantitativt og kvalitativt overlegne in-
deholdende mere data og information, der forbedrede vores vurderinger af risikoen for bias.
Afslutningsvis er det ikke klart i hvor høj grad HPV-vaccinernes gavnlige virkninger er større
end deres skadevirkninger, da studieprogrammerne og -rapporterne var påvirkede af bias og utilstræk-
kelige design. De kliniske studierapporter var bedre studiedokumenter, der indeholdte mere informa-
tion til bias-vurderinger. Kliniske studierapporter bør derfor bruges til systematiske oversigter.
Page 7 of 637
Introduction
Human papillomavirus (HPV) infection

The HPV family consists of more than 150 DNA viruses. HPV infections are associated with nearly 5% of
all cancers: 70-90% of anal, 70-100% of cervical, 50-70% of oropharyngeal, 40-60% of penile, 30-50% of
vaginal and 30-50% of vulvar cancer (4). HPV can also cause genital lesions, genital warts and respiratory
papillomatosis (4). While most HPV infections disrupt cell cycle regulator mechanisms, 25 HPV types
have been shown to have oncogenic potential—13 ‘high-risk’ types: HPV 16, 18, 31, 33, 35, 39, 45, 51,
52, 56, 58, 59 and 68; and 12 ‘low-risk’ types: HPV 6, 11, 40, 42, 43, 44, 53, 54, 61, 72, 73 and 81 (5).
Most people will get HPV infections; with the majority being cleared by the immune system
without sequelae (6). Persistent HPV infections increase the risk of abnormal histology and
precancerous lesions, which mostly regresses; for example, cervical intraepithelial neoplasia (CIN)
regresses in approximately 90% of low (CIN1), 50-70% of moderate (CIN2) and 20-30% of
high/carcinoma-in-situ (CIN3) degree cases (6). When infected with more than one HPV type (co-
infection), infections persist longer and increase the risk of precancerous lesions (7). Four in five HPV-
related cancers occur in the cervix with more than 85% of cervical cancers containing HPV (7).
Globally, cervical cancer is the third most occuring female-specific cancer (after breast and
colorectal cancer) (4). Approximately 90% of cervical cancer deaths occur in developing regions—
especially Africa, where cervical cancer is the leading female cancer (8). In developed regions, such as
the United States, cervical cancer annually occurs in approximately 10 in 100,000 women; in
comparison, breast cancer occurs in more than 100 in 100,000 women (9,10). Most developed countries
have invested in cervical screening using cytological Papanicolaou—in short, Pap—tests to indicate
abnormal histology. As a result, from the mid 1990s to 2010s, the global annual death rate of cervical
cancer declined by 200,000 deaths: from 470,000 to 270,000 (43%) (9).
Cumulative cervical screening may offer more than 80% protection against cancer, but to avoid
one cancer, an estimated 2,700 women need to be screened (11). The majority of positive Pap tests are
due to low-grade or no disease (12), which cause unwarranted referral procedures (13). Some now
Page 8 of 637
recommend an active surveillance approach for this group (6). In developed regions, most patients are
diagnosed in early cervical cancer stages with an 80% five-year survival rate (14).
HPV vaccination
Since 2006, the HPV vaccines—GlaxoSmithKline’s Cervarix™ and Merck Sharp and Dohme’s Gardasil™
and Gardasil 9™—have been recommended to females in more than 130 countries against HPV-related
diseases (15). The targeted HPV types are associated with the majority of HPV-related cancers—espe-
cially HPV 16 and 18 (4). Cervarix targets HPV 16 and 18; Gardasil targets HPV 6, 11, 16 and 18; and
Gardasil 9 targets HPV 6, 11, 16, 18, 31, 33, 45, 52 and 58. The approved indication is cervical cancer as
well as anal, vaginal and vulvar cancer depending on the regulatory agency (16–18). The recognized
serious harms include anaphylaxis and syncope (16–18).
In 2006, the vaccines got approved as three-dose regimens; in 2015, the World Health
Organization (WHO) approved a two-dose regimen for females aged 9 to 15 and noted that one dose
may suffice (4). Currently, about 2% of the global female population is HPV vaccinated (15). Developed
regions account for more than 70% of vaccinated women and less than 15% of cervical cancers (15).
According to the HPV vaccine manufacturers data published in journal publications, the protection
against abnormal cervical histology or worse (CIN2+) irrespective of HPV type in women aged 15 to 26
years is approximately 60% for Cervarix and 20% for Gardasil (19). The protection has been reported to
last up to twelve years for per-protocol populations against HPV 16/18-related CIN2+ (20), and the
protection likely better when vaccinated at a young age (21). Some researchers estimate that the
vaccines will be able to greatly decrease the incidence of cervical cancer (22).
Herd immunity has been reported in some regions for HPV infections and genital warts (23)—
also with low vaccination rates (29 to 55%) (24). Lower CIN2+ rates were reported in the United States,
but this decline was confounded by a decrease in vaccinated women that attended cervical screening
(25).
Page 9 of 637
Evidence gaps
Most drug regulators and public health bodies state that the HPV vaccines are safe and reduce HPV-
related cancers (26–29), but the regulators’ vaccine approvals were primarily based on surrogate
outcomes in per-protocol analyses of vaccine-specific HPV-types, which introduce bias.
Most researchers have not found serious harms associated with the HPV vaccines (30–33), but
other researchers have found rare serious nervous system disorders—such as, postural orthostatic
tachycardia syndrome (POTS) and complex regional pain syndrome (CRPS) (34–36)—associated with
the HPV vaccines. Some cases of POTS and CRPS were not recognised or not reported by the HPV
vaccine manufacturers. For example, when the Danish regulator asked the HPV vaccine manufacturer
Sanofi-Pasteur-MSD to assess its database, the company only identified three of 26 cases of POTS that
had been linked to the HPV vaccines by the Danish authorities (37). In addition, an EMA joint response
assessment report identified six cases of POTS and CRPS that were associated with Gardasil 9, but the
manufacturer—in this case, Merck Sharp & Dohme—had not identified these (38). EMA had the HPV
vaccine manufacturers carry out reviews of POTS and CRPS. The manufacturers found no association
with POTS/CRPS and the HPV vaccines, but it was not appropriate for the manufacturers to conduct the
data analyses for EMA, as they that have conflicts of interest (2,39,40). In addition, the manufacturers’
HPV vaccine study programmes were influenced by reporting bias; for example, one third of the study
programmes were not published and study results were only posted for half of the completed studies
on ClinicalTrials.gov (2).
Regulatory data in systematic reviews

Approximately half of all clinical studies do not get published (41,42) and the published studies fre-
quently overestimate the measured effects compared to corresponding unpublished studies (43–46).
This is detrimental to the validity of systematic reviews (41,47–51), as they usually mainly include pub-
lished studies. However, unpublished regulatory data—such as clinical study reports and periodical
safety reports—are increasingly used as data sources in systematic reviews (52). A clinical study report
is structured according to international guidelines (53) and such reports usually include more study
information and data than the study’s published version in biomedical journals (54–57). Compared to
10
Page 10 of 637
clinical study reports, journal publications often underreport serious harms (58). Underreporting of
harms have occasionally led to the approval of drugs that were more harmful than what had been re-
ported (59–61). It can, however, be difficult to get study documents (such as clinical study reports) for
unpublished studies (56,62–65). Only 10 to 20% of systematic reviews include unpublished studies
(66,67) and only 10 to 50% of reviews include searches of trial registers, although these searches iden-
tify additional studies in up to 60% of reviews (68–71).
Cochrane’s risk of bias tool

Systematic reviews of randomised trials have had a large impact on clinical practice, but the included
trials in systematic reviews are often affected by biased designs and reporting bias (72). A meta-analysis
of biased effect estimates will often produce a biased pooled estimate with a narrow confidence inter-
val that gives greater credibility compared to a wide confidence interval. As systematic reviews are
often limited by biased trials, it is important for authors of systematic reviews to adequately address
the risk of bias in included trials (73). The Cochrane tool was made to assess risk of bias in trials and was
first introduced in 2008 and updated in both 2011 and 2018. The tool is based on six empirically inves-
tigated bias domains (and a supplementary domain called ‘other bias’) that were selected to cover the
most important bias mechanisms that occur in trials (74). For each of the bias domains, researchers are
requested to judge the risk of bias on an overall level and on an individual outcome level and to docu-
ment their judgements with verbatim quotes. This enables researchers to conduct sensitivity and sub-
group analyses based on risk of bias, for example, ‘high’ vs. ‘low’ risk of bias (1).
11
Page 11 of 637
Objectives
Paper 1: Evaluation of Cochrane’s risk of bias tool

The objectives were to evaluate the Cochrane risk of bias tool by 1) identifying and assessing user com-
ments on strengths and challenges and 2) to analyse, compare and describe how the tool was used in
100 Cochrane and 100 non-Cochrane reviews (1).
Paper 2: Index of the HPV vaccine study programmes

The objective was to address reporting bias for our systematic review of the HPV vaccines by identifying
unpublished studies with a six-step process that included correspondence with regulators and manu-
facturers and searches of regulatory reviews, trial registers and publication databases (2).
Paper 3: Challenges of independent HPV vaccine assessment

The objective was to report the challenges we faced obtaining complete unredacted clinical study re-
ports for our systematic review from the regulators and give recommendations on how the process of
obtaining regulatory data could be improved (3).
Paper 4: Systematic review of the HPV vaccines

The objective was to investigate the benefits and harms of the HPV vaccines by using the indexed study
programmes and the obtained clinical study reports. The review protocol is registered on PROSPERO:
https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20170030.pdf.
Paper 5: Comparison of HPV vaccine study documents

The objective was to compare corresponding study documents of the HPV vaccines: clinical study re-
ports, trial register entries and journal publications. The protocol is registered on PROSPERO:
https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20180320.pdf.
12
Page 12 of 637
Description of the research projects

We assessed Cochrane’s risk of bias tool by evaluating comments that were published in the medical
literature on the tool’s strengths and challenges, and described and analysed how the tool was used in
Cochrane and non-Cochrane systematic reviews (1).
We identified journal publications that commented on the tool. We focused on “major com-
ments” (with more than 100 words of text) on the strengths or challenges of the tool. We made a qual-
itative map of the themes that the commenters addressed and a categorisation according to whether
they addressed a fundamental feature of the tool or a concern related to the tool’s implementation (1).
In our observational study, we compared 100 contemporary Cochrane reviews with 100 con-
temporary non-Cochrane reviews. For example, we investigated how many reviews used the risk of bias
judgements as a basis for sensitivity and sub-group analyses (1).

To address reporting bias for our systematic review of clinical study reports, we made an index of the
HPV vaccine study programmes and indexed prospective comparative HPV vaccine studies. We cross-
verified study programme entries via inquiries to regulators and manufacturers and searched 45 trial
registers and several regulatory and publication databases including non-conventional sources such as
Drug Approval Packages (DAPs), European Public Assessment Reports (EPARs) and WikiLeaks. We as-
sessed study characteristics including types and numbers of used IDs (e.g., study ID, manufacturer ID,
trial register ID, publication ID, etc.), availability of results and publication status (2).

We described our difficulties in indexing study programmes and obtaining unredacted and complete
clinical study reports. We documented our correspondence with the regulators and the HPV vaccine
manufacturers, evaluated the processes whereby researchers may obtain clinical study reports and
identified areas that ought to be optimised (3).
13
Page 13 of 637

We made requests to the regulators and HPV vaccine manufacturers for the study programmes and
their clinical study reports. We included clinical study reports of randomised trials that compared an
HPV vaccine with a placebo or active comparator in healthy participants. The data was assessed and
extracted by two researchers. We meta-analysed data of clinically important outcomes and used data
of intention to treat analyses irrespective of the involved HPV types. As we did not obtain a complete
dataset, and since the clinical study reports lacked important parts (such as case report forms) and
often were redacted, we conducted post hoc exploratory analyses of MedDRA (Medical Dictionary for
Regulatory Activities) system organ class data and harms of special interest.

We obtained trial register entries and journal publications corresponding to the clinical study reports
that we obtained from the regulators and HPV vaccine manufacturers. We compared the correspond-
ing study documents qualitatively and quantitatively. For example, we compared the PICO (population,
intervention, control and outcomes) criteria that were used and the reporting of major design-related
biases and ratios of relative risk for 20 pre-specified outcomes that we had conducted meta-analyses
for in all three corresponding document types.
14
Page 14 of 637
Summary of results

We identified 68 comments—of which, 15 were “major” and 53 “minor”—in the databases that we
sought. The commenters considered the tool’s aims, developmental basis and transparency to be its
main strengths; its choice of bias domains, implementation, overall risk of bias judgements and some
special situations—for example, that the tool is difficult to apply to clinical study reports—were consid-
ered to be its primary challenges. The commenters suggested that the tool could be improved with
better guidelines, more empirical research on both fundamental and possible bias mechanisms and
that funding or conflicts of interest should be added as an individual bias domain. Although funding is
not a bias mechanism per se (like, for example, sequence generation), 32% of Cochrane reviews as-
sessed funding in the auxiliary ‘other bias’ domain (1).
The Cochrane tool was used in 100 of 100 assessed Cochrane reviews and 80 of 100 non-
Cochrane reviews. While the main purpose of the tool is to enable sub-group and sensitivity analyses
based on risk of bias, only 19 of 100 Cochrane reviews reported to have performed such analyses—
even though it was planned by 70 of 100 reviews. The primary reason for not performing such analyses
was that 94% of the trials included in the Cochrane reviews were judged at ‘high’ (761/1,242, 61%) or
‘unclear’ (407/1,242, 33%) risk of bias (1).
Although not recommended in the Cochrane Handbook, 88 of 100 Cochrane reviews had risk
of bias judged only on an overall level—not on individual outcome levels; 37 of 100 reviews merged
standard bias domains; and 18 of 100 reviews divided a bias domain into separate domains. As the tool
was used in such non-recommended ways, there is some uncertainty as to what the tool’s scope is and
how to use the tool most effectively. For example, we found 89% of trials with at least one domain
judged at ‘unclear’ risk, but a proportion of these judgements could likely have been avoided if the
Cochrane researchers had contacted study authors or sought additional sources such as trial registers.
Our study was used to inform version 2018 (or 2.0) of the tool (75). Version 2.0 addresses some of the
challenges that we identified in version 2011, for example, that many judgements are of unclear risk of
bias and that the tool could be optimized for clinical study reports (1).
15
Page 15 of 637

We indexed 206 comparative prospective HPV vaccine studies for our study programmes of both indus-
try and non-industry studies. Although we requested information for our index from all four HPV vac-
cine manufacturers, only GlaxoSmithKline provided such information (2).
Two-thirds (66%) of the indexed studies were trials and 43% were phase III. Compared to non-
industry studies, industry studies included more participants (mean 3,602 vs. 1,767) and were more
often multicentre (P<0.0001), but non-industry studies were longer in duration than industry studies
(42 vs. 37 months) (2).
We were able to cross-verify 78% of the indexed studies and found 90% of the studies to be
listed on ClinicalTrials.gov, but study results were only provided for 48% of the completed studies on
ClinicalTrials.gov and 38% of completed studies were not published in the journal publication databases
that we sought. Thus, if our systematic review had relied on journal publication databases, it would
have missed several studies. We also indexed more studies than major regulators had listed in their
databases (2).
Our index process’ six steps are reproducible and could be used in other systematic reviews
when there is high risk of reporting bias such as reviews of industry drugs, block-buster drugs or drugs
with safety signals reported post-approval (2).

Of our 206 indexed studies, we identified 48 industry studies that likely were eligible for our systematic
review. In 2014, we used EMA’s policy 0043 to ask EMA for its HPV vaccine clinical study reports, but
EMA only had clinical study reports for 29 industry studies. After three years (in 2017), we obtained 18
of EMA’s 29 reports; we found 12 of these to be eligible for our systematic review. In 61 batches, EMA
sent us more than 35,000 pages for the 18 reports, but the majority of reports did not include protocols,
serious harms narratives and case report forms. The reports were also often redacted and divided into
many incoherent files that made it unnecessarily complicated to put the reports back into a coherent
form. For example, over the course of twelve months, the study HPV-008 was sent to us in seven
batches that contained 17 files in random order with 4,263 pages. We were informed by EMA that this
16
Page 16 of 637
was all the pages that EMA had available of HPV-008, but HPV-008’s publicly available clinical study
report on GlaxoSmithKline’s trial register is over 7,000 pages. When we asked EMA to release the re-
ports in a more complete, coherent and rapid manner, EMA informed us that requests for regulatory
data had risen significantly—in particular, requests from the drug industry—and that the agency there-
fore was in shortage of staff, which was the main reason for the slow document release (3).
By our data lock for our systematic review in July 2017, we obtained 24 of 48 eligible industry
clinical study reports including reports from both EMA and those freely available on GlaxoSmithKline’s
trial register. Although the 24 reports included the majority of participants in the 48 studies, the reports
were limited by redactions and incompleteness (3).
We found several issues with both transparency and data sharing that ought to be improved
for independent researchers to be able to better conduct systematic reviews with regulatory data. Alt-
hough transparency and sharing of trial data may increase in the future, EMA recently reduced its data
sharing policies—mainly due to the many data requests the agency receives. Nonetheless, regulators
and industry should enhance their release of clinical study reports to impartial researchers so drugs can
be assessed in an independent manner—especially block-buster drugs with post-approval safety issues
such as the HPV vaccines. It would also be helpful if regulators could provide detailed lists of the clinical
study reports they hold and release complete reports in a reasonable timeframe (3).

Of 50 identified eligible studies (43 industry trials, five industry follow-up studies and two non-industry
trials), we obtained 24 clinical study reports for 24 studies that included 58,412 pages, 95,670 partici-
pants with 393,194 person-years and a mean follow-up time of 49 months. The 26 remaining studies
that we did not obtain clinical study reports for included 21% of the total eligible participants
(25,632/121,441). We used the Cochrane tool and judged all included clinical study reports to be at
‘high’ risk of bias. Serious harms were incompletely reported for 72% of the participants
(68,610/95,670), all clinical study reports contained redactions and lacked significant parts, and 99% of
control participants received an active comparator (48,289/48,595) that contained either the HPV vac-
cines aluminium-containing adjuvants or hepatitis vaccines that included the same aluminium-
17
Page 17 of 637
containing adjuvants (except for Aimmugen™). Half of the included participants (49,301/95,670) were
only allowed to enter the studies if they had never received the aluminium-containing comparators
before—even though the manufacturers state in the clinical study reports that the aluminium compar-
ators are safe.
For benefits, we found that at four years follow-up the HPV vaccines decreased HPV-related
carcinoma in situ (367 vs. 490, RR 0.73 [95% CI 0.53 to 1.00], number needed to vaccinate [NNV] 387,
P=0.05, I2=67%), moderate intraepithelial neoplasia or worse (952 vs. 1,239, RR 0.78 [95% CI 0.66 to
0.91], NNV 190, P=0.002, I2=53%) and treatment procedures (1,081 vs. 1,416, RR 0.71 [95% CI 0.63 to
0.80], NNV 75, P<0.00001, I2=45%).
For harms, we found that the HPV vaccines increased serious nervous system disorders
grouped in the MedDRA (Medical Dictionary for Regulatory Activities) system organ class (72 vs. 46, RR
1.49 [95% CI 1.02 to 2.16], number needed to harm [NNH] 1,325, P=0.04, I2=0%) but the vaccines did
not increase fatal harms (45 vs. 38, RR 1.19 [95% CI 0.65 to 2.19], P=0.58, I2=30%) or serious harms
(1,404 vs. 1,357, RR 1.01 [95% CI 0.94 to 1.08], P=0.79, I2=0%)—nor did the vaccines increase (or de-
crease) any of the nearly 1,000 different individual fatal and serious harms in the clinical study reports
that were classified with MedDRA preferred terms. The HPV vaccines increased general harms (13,248
vs. 12,394, RR 1.07 [95% CI 1.03 to 1.11], NNH 51, P<0.00001, I2=77%)—in particular, fatigue (4,933 vs.
4,489, RR 1.13 [95% CI 1.08 to 1.18], NNH 67, P<0.00001, I2=22%), headache (5,561 vs. 5,246, RR 1.06
[95% CI 1.02 to 1.11], NNH 83, P=0.009, I2=40%) and myalgia (3,989 vs. 3,047, RR 1.41 [95% CI 1.24 to
1.60], NNH 26, P<0.00001, I2=80%) were increased.
For harms of special interest, there were no reports of chronic fatigue syndrome (CFS), com-
plex regional pain syndrome (CRPS), Guillain-Barré syndrome (GBS) or postural orthostatic tachycardia
syndrome (POTS) in the clinical study reports. However, since some cases of POTS and CRPS had not
been recognised by the manufacturers in their studies, we performed post hoc exploratory harm anal-
yses of POTS and CRPS. For these analyses we found that the serious harms that were judged ‘definitely
associated’ with POTS or CRPS by a blinded physician, who had clinical expertise with POTS and CRPS,
were increased by the HPV vaccines, both for POTS (56 vs. 26, RR 1.92 [95% CI 1.21 to 3.07], NNH 1,073,
P=0.006, I2=0%) and CRPS (95 vs. 57, RR 1.54 [95% CI 1.11 to 2.14], NNH 906, P=0.010, I2=0%). POTS and
18
Page 18 of 637
CRPS are syndromes that have a certain overlap with immunological and nervous system disorders, but
most participants were only included in the included trials if they had no history of such disorders
(63,468/95,670). Although the exploratory POTS and CRPS analyses do not prove that the HPV vaccines
cause POTS and CRPS—since the analyses could not take symptom duration, symptom clustering or the
diagnostic criteria into account—the analyses provide a safety signal justifying that independent anal-
yses of POTS and CRPS should be undertaken based on the clinical study reports’ case report forms and
individual participant data.
We conducted sub-group analyses and found that compared to older HPV vaccinated partici-
pants, younger participants—who are those mainly intended for vaccination—had fewer cases of mod-
erate HPV-related intraepithelial neoplasia or worse (age 15 to 29: 784 vs. 1,079, RR 0.71 [95% CI 0.61
to 0.83]; age 21 to 72: 168 vs. 160, RR 1.04 [95% CI 0.84 to 1.29]; ratio of relative risk [RRR] 1.46 [1.12
to 1.91]) and fatal harms (age 15 to 27: 24 vs. 32, RR 0.77 [95% CI 0.45, 1.33]; age 21 to 72: 21 vs. 6, RR
3.13 [95% CI 1.29 to 7.61]; RRR 0.25 [95% CI 0.09 to 0.70]). We did not find any differences between
younger and older age groups for serious nervous system disorders (age 10 to 35: 53 vs. 35, RR 1.46
[95% CI 0.95 to 2.25]; age 21 to 72: 19 vs. 11, RR 1.56 [95% CI 0.75 to 3.25]; RRR 0.93 [95% CI 0.40 to
2.19]), serious harms judged ‘definitely associated’ with CRPS (age 9 to 35: 76 vs. 48, RR 1.48 [95% CI
1.03 to 2.12]; age 21 to 72: 19 vs. 9, RR 2.11 [95% CI 0.67 to 6.69]; RRR 0.70 [95% CI 0.21 to 2.34]) or
serious harms judged ‘definitely associated’ with POTS (age 12 to 35: 43 vs. 21, RR 1.86 [95% CI 1.10,
3.15]; age 21 to 72: 13 vs. 5, RR 2.22 [95 CI 0.76 to 6.47]; RRR 0.84 [95% CI 0.25 to 2.76]).
Our review had several strengths—it was based on study programmes, randomised trials, clin-
ical study reports, pre-specified outcomes, intention to treat analyses and absolute risk estimates. Our
review also had several limitations—especially that the clinical study reports and sample were incom-
plete. For example, the 26 studies with no available clinical study reports could have influenced the
results—in particular, the results with p-values around our cut-off of 0.05 and with wide confidence
intervals. A fourth (8/31) of the significant results we found were likely to have occurred by chance,
since a fifth of the meta-analyses we performed were statistically significant (31/166) with a p-value of
0.05 (166*0.05=8). As our sample only included one small Gardasil 9 trial (V503-006, that did not inves-
tigate HPV-related cancers or precursors to cancers) and a two-dose Gardasil 9 regimen is currently
19
Page 19 of 637
being implemented in many countries’ childhood vaccination programmes, our review is less relevant
for those who have received or will receive Gardasil 9. We did, however, obtain a clinical study report—
of a phase 3 multicentre trial of 7,106 and 7,109 healthy females aged 16 to 26 that were randomised
to receive three doses of Gardasil 9 and Gardasil—where Gardasil 9 increased both serious harms (233
vs. 183, RR 1.27 [95% CI 1.05 to 1.54], P=0.010; reported from day 0 to 390) and general harms (‘sys-
temic adverse events’: 2,086 vs. 1,929, RR 1.08 [95% CI 1.03 to 1.14], P=0.003; reported 0 to 14 days
post-vaccination) compared to Gardasil.

For our comparison of the 24 clinical study reports with other trial document types, we obtained 24
corresponding trial register entries from ClinicalTrials.gov and 23 corresponding journal publications via
our index (2). There were large dissimilarities between the quantity of data and information in the three
study document types. For example, compared to the trial register entries and journal publications, the
clinical study reports contained more pages (median 1,351 vs. 32 and 11), inclusion criteria (mean 7.0
vs. 5.8 and 4.0), exclusion criteria (mean 17.8 vs. 11.7 and 5.0), benefit data points (6,879 vs. 230 and
3,015) and harm data points (167,550 vs. 64,092 and 51,870) for the 20 prespecified outcomes. Conse-
quently, if our systematic review of clinical study reports had relied on trial register entries or journal
publications, it would have missed 99% or 56% of the assessed benefit data points and 62% or 69% of
the harm data points. However, the ratio of relative risk differences that could be calculated (41 of 60)
were not statistically significant.
As there are no space restriction on ClinicalTrials.gov and most biomedical journals allow very
large electronic appendices, the lower amount of data points in the trial register entries and journal
publications were likely due to reporting bias.
20
Page 20 of 637
Conclusions and perspectives for further research

Our evaluation of the Cochrane tool informed the tool’s recent update: version 2.0. Although some
commenters disagreed, we found the tool to be useful in assessing risk of bias of the clinical study
reports we included in our systematic review.
Researchers that conduct systematic reviews—especially ones of industry interventions—may
adopt our index process and therefore possibly address risk of reporting bias to a better degree com-
pared to only using conventional journal publication database searches. While only one of four HPV
vaccine manufacturers offered data for our index and we could not cross-verify a fifth of the study
index, we think we came close to constructing near-complete study programme of the HPV vaccines.
Regulators and industry should register and publish all studies and make sure that the regula-
tory data they release are coherent and complete since clinical study reports are the quantitively and
qualitatively superior study documents.
On the basis of the obtained clinical study reports, we showed that the HPV vaccines decrease
HPV-related precursors to cancers and treatment procedures, but the HPV vaccines also increased se-
rious nervous system disorders and general harms—despite that the included studies were inade-
quately designed to assess harms and subject to reporting bias.
If our systematic review had relied on trial register entries and journal publications instead of
clinical study reports, it would have had no data for a quarter of our pre-specified outcomes. While the
inclusion of clinical study reports led to significantly more eligible and available data, no changes in the
direction of available results occurred when comparing the risk ratios of corresponding study docu-
ments as ratios of relative risks.
To better identify bias and increase validity and transparency of systematic reviews, we rec-
ommend that researchers who conduct systematic reviews use study programmes, regulatory data—
especially clinical study reports with case report forms and individual participant data—and the
Cochrane risk of bias tool.
21
Page 21 of 637
List of abbreviations
CI Confidence interval
CIN Cervical intraepithelial neoplasia
CFS Chronic fatigue syndrome
CRPS Chronic regional pain syndrome
EMA European Medicines Agency
FDA Food and Drug Administration
GBS Guillain-Barré syndrome
GSK GlaxoSmithKline
HPV Human papillomavirus
I2 Statistical heterogeneity
MedDRA Medical Dictionary for Regulatory Activities
Merck Merck & Co., Inc. or Merck Sharp & Dohme outside the United States and Canada
NNH Number needed to harm
NNV Number needed to vaccinate
PICO Patient, intervention, comparator and outcome
POF Premature ovarian failure
POTS Postural orthostatic tachycardia syndrome
PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses
PROSPERO International Prospective Register of Systematic Reviews
RR Risk ratio
RRR Ratio of relative risk
22
Page 22 of 637
References
1. Jørgensen L, Paludan-Müller AS, Laursen DRT, Savović J, Boutron I, Sterne JAC, et al. Evaluation of
the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published com-
ments and analysis of user practice in Cochrane and non-Cochrane reviews. Syst Rev. 2016 May
10;5:80. doi: 10.1186/s13643-016-0259-8.
2. Jørgensen L, Gøtzsche PC, Jefferson T. Index of the human papillomavirus (HPV) vaccine industry
clinical study programmes and non-industry funded studies: a necessary basis to address reporting
bias in a systematic review. Syst Rev. 2018 Jan 18;7(1):8. doi: 10.1186/s13643-018-0675-z.
3. Jørgensen L, Doshi P, Gøtzsche PC, Jefferson T. Challenges of independent assessment of potential
harms of HPV vaccines. BMJ. 2018 Sep 24;362:k3694. doi: 10.1136/bmj.k3694.
4. World Health Organization. Human papillomavirus vaccines: WHO position paper. Wkly Epidemiol
Rec. 2017 May 12;92(19):241-68.
5. IARC HPV Working Group. Primary End-points for Prophylactic HPV Vaccine Trials. International
Agency for Research on Cancer. 2014.
6. Tainio K, Athanasiou A, Tikkinen KAO, Aaltonen R, Cárdenas J, Hernándes, et al. Clinical course of
untreated cervical intraepithelial neoplasia grade 2 under active surveillance: systematic review and
meta-analysis. BMJ. 2018 Feb 27;360:k499. doi: 10.1136/bmj.k499.
7. de Sanjose S, Quint WG, Alemany L, Geraets DT, Klaustermeier JE, Lloveras B, et al. Human papillo-
mavirus genotype attribution in invasive cervical cancer: a retrospective cross-sectional worldwide
study. Lancet Oncol. 2010 Nov;11(11):1048-56. doi: 10.1016/S1470-2045(10)70230-8.
8. Bruni L, Barrionuevo-Rosas L, Albero G, Serrano B, Mena M, Gómez D, Muñoz J, Bosch FX, de
Sanjosé S. Human Papillomavirus and Related Diseases in Africa. 2017 Jul 27. Available from:
http://www.hpvcentre.net/statistics/reports/XFX.pdf.
9. Arbyn M, Castellsagué X, de Sanjosé S, Bruni L, Saraiya M, Bray F, et al. Worldwide burden of cervi-
cal cancer in 2008. Ann Oncol. 2011 Dec;22(12):2675-86. doi: 10.1093/annonc/mdr015.
10. Bray F, Ferlay J, Laversanne M, Brewster DH, Gombe Mbalawa C, Kohler B, et al. Cancer Incidence
in Five Continents: Inclusion criteria, highlights from Volume X and the global status of cancer regis-
tration. Int J Cancer. 2015 Nov 1;137(9):2060-71. doi: 10.1002/ijc.29670.
23
Page 23 of 637
11. Barut MU, Kale A, Kuyumcuoğlu U, Bozkurt M, Ağaçayak E, Özekinci S, et al. Analysis of Sensitivity,
Specificity, and Positive and Negative Predictive Values of Smear and Colposcopy in Diagnosis of
Premalignant and Malignant Cervical Lesions. Med Sci Monit. 2015 Dec 10;21:3860-7. doi:
10.12659/MSM.895227.
12. Landy R, Castanon A, Hamilton W, Lim AWW, Dudding N, Hollingworth A, et al. Evaluating cytology
for the detection of invasive cervical cancer. Cytopathology. 2016 Jun; 27(3): 201–209. doi:
10.1111/cyt.12259.
13. O’Connor M, Gallagher P, Waller J, Martin CM, O’Leary JJ, Sharp L, et al. Adverse psychological
outcomes following colposcopy and related procedures: a systematic review. BJOG. 2016
Jan;123(1):24-38. doi: 10.1111/1471-0528.13462.
14. Östensson E, Fröberg M, Leval A, Hellström A-C, Bäcklund M, Zethraeus N, et al. Cost of Prevent-
ing, Managing, and Treating Human Papillomavirus (HPV)-Related Diseases in Sweden before the In-
troduction of Quadrivalent HPV Vaccination. PLoS One. 2015 Sep 23;10(9):e0139062. doi:
10.1371/journal.pone.0139062.
15. Clendinen C, Zhang Y, Warburton RN, Light DW. Manufacturing costs of HPV vaccines for develop-
ing countries. Vaccine. 2016 Nov 21;34(48):5984-5989. doi: 10.1016/j.vaccine.2016.09.042.
16. Cervarix. Available from: https://www.fda.gov/BiologicsBloodVaccines/Vaccines/ApprovedProd-
ucts/ucm186957.htm.
17. Gardasil. Available from: https://www.fda.gov/BiologicsBloodVaccines/Vaccines/ApprovedProd-
ucts/UCM094042.
18. Gardasil 9. Available from: https://www.fda.gov/BiologicsBloodVaccines/Vaccines/ApprovedProd-
ucts/ucm426445.htm.
19. D.M. Harper, L.R. DeMars. HPV vaccines – A review of the first decade. Gynecol Oncol. 2017
Jul;146(1):196-204. doi: 10.1016/j.ygyno.2017.04.004.
20. Kjaer SK, Nygård M, Dillner J, Brooke Marshall J, Radley D, Li M, et al. A 12-Year Follow-up on the
Long-Term Effectiveness of the Quadrivalent Human Papillomavirus Vaccine in 4 Nordic Countries.
Clin Infect Dis. 2018 Jan 18;66(3):339-345. doi: 10.1093/cid/cix797.
24
Page 24 of 637
21. Dehlendorff C, Sparén P, Baldur-Felskov B, Herweijer E, Arnheim-Dahlström L, Ploner A, et al. Ef-

fectiveness of varying number of doses and timing between doses of quadrivalent HPV vaccine
against severe cervical lesions. Vaccine. 2018 Oct 15;36(43):6373-6378. doi: 10.1016/j.vac-
cine.2018.09.011.
22. Hall MT, Simms KT, Lew J-B, Smith MA, Brotherton JM, Saville M, et al. The projected timeframe
until cervical cancer elimination in Australia: a modelling study. Lancet Public Health. 2018 Oct 1. doi:
10.1016/S2468-2667(18)30183-X.
23. Drolet M, Bénard É, Boily M-C, Ali H, Baandrup L, Bauer H, et al. Population-level impact and herd
effects following human papillomavirus vaccination programmes: a systematic review and meta-anal-
ysis. Lancet Infect Dis. 2015 May;15(5):565-80. doi: 10.1016/S1473-3099(14)71073-4.
24. Guo F, Cofie LE, Berenson AB. Cervical Cancer Incidence in Young U.S. Females After Human Papil-
lomavirus Vaccine Introduction. Am J Prev Med. 2018 Aug;55(2):197-204. doi: 10.1016/j.ame-
pre.2018.03.013.
25. Hariri S, Johnson ML, Bennett NM, Bauer HM, Park IU, Schafer S, et al. Population-based trends in
high-grade cervical lesions in the early human papillomavirus vaccine era in the United States. Cancer.
2015 Aug 15;121(16):2775-81. doi: 10.1002/cncr.29266.
26. European Medicines Agency. Assessment report Review under Article 20 of Regulation (EC) No
726/2004 Human papillomavirus (HPV) vaccines. Available from: http://www.ema.eu-
ropa.eu/docs/en_GB/document_library/Referrals_document/HPV_vaccines_20/Opinion_pro-
vided_by_Committee_for_Medicinal_Products_for_Human_Use/WC500197129.pdf
27. Garland SM, Kjaer SK, Muñoz N, Block SL, Brown DR, DiNubile MJ, et al. Impact and Effectiveness
of the Quadrivalent Human Papillomavirus Vaccine: A Systematic Review of 10 Years of Real-world
Experience. Clin Infect Dis. 2016 Aug 15;63(4):519-27. doi: 10.1093/cid/ciw354.
28. Angelo M-G, David M-P, Zima J, Baril L, Dubin G, Arellano F, et al. Pooled analysis of large and
long-term safety data from the human papillomavirus-16/18-AS04-adjuvanted vaccine clinical trial
programme. Pharmacoepidemiol Drug Saf. 2014 May;23(5):466-79. doi: 10.1002/pds.3554.
25
Page 25 of 637
29. Rey-Ares L, Ciapponi A, Pichon-Riviere A. Efficacy and safety of human papilloma virus vaccine in
cervical cancer prevention: systematic review and meta-analysis. Arch Argent Pediatr. 2012
Dec;110(6):483-9. doi: 10.1590/S0325-00752012000600005.
30. Feiring B, Laake I, Bakken IJ, Greve-Isdahl M, Wyller VB, Håberg SE, et al. HPV vaccination and risk
of chronic fatigue syndrome/myalgic encephalomyelitis: A nationwide register-based study from Nor-
way. Vaccine. 2017 Jul 24;35(33):4203-4212. doi: 10.1016/j.vaccine.2017.06.031.
31. Miranda S, Chaignot C, Collin C, Dray-Spira R, Weill A, Zureik M. Human papillomavirus vaccination
and risk of autoimmune diseases: A large cohort study of over 2 million young girls in France. Vaccine.
2017 Aug 24;35(36):4761-4768. doi: 10.1016/j.vaccine.2017.06.030.
32. Scheller NM, Pasternak B, Mølgaard-Nielsen D, Svanström H, Hviid A. Quadrivalent HPV Vaccina-
tion and the Risk of Adverse Pregnancy Outcomes. N Engl J Med. 2017 Mar 30;376(13):1223-1233.
doi: 10.1056/NEJMoa1612296.
33. Phillips A, Patel C, Pillsbury A, Brotherton J, Macartney K. Safety of Human Papillomavirus Vac-
cines: An Updated Review. Drug Saf. 2018 Apr;41(4):329-346. doi: 10.1007/s40264-017-0625-z.
34. Blitshteyn S, Brook J. Postural tachycardia syndrome (POTS) with anti-NMDA receptor antibodies
after human papillomavirus vaccination. Immunol Res. 2017 Feb;65(1):282-284. doi: 10.1007/s12026-
016-8855-1.
35. Chandler RE, Juhlin K, Fransson J, Caster O, Edwards IR, Norén GN. Current Safety Concerns with
Human Papillomavirus Vaccine: A Cluster Analysis of Reports in VigiBase®. Drug Saf. 2017
Jan;40(1):81-90. doi: 10.1007/s40264-016-0456-3.
36. Brinth LS, Pors K, Theibel AC, Mehlsen J. Orthostatic intolerance and postural tachycardia syn-
drome as suspected adverse effects of vaccination against human papilloma virus. Vaccine. 2015 May
21;33(22):2602-5. doi: 10.1016/j.vaccine.2015.03.098.
37. Weber C, Andersen S. The company behind the HPV vaccine underestimated the extent of serious
side effects [Firma bag HPV-vaccinen underdrev omfanget af alvorlige bivirkninger]. 2015 Oct 26.
Available from: https://www.b.dk/content/item/105701.
38. Rapporteurs’ Day 150 Joint Response Assessment Report of Gardasil 9. 2014 Nov 25.
EMEA/H/C/3852.
26
Page 26 of 637
39. Gøtzsche PC, Jefferson T, Brinth LS, Jørgensen KJ, Margrete A. Complaint to the European ombuds-
man over maladministration at the European Medicines Agency (EMA) in relation to the safety of the
HPV vaccines. Available from: http://nordic.cochrane.org/sites/nordic.cochrane.org/files/public/up-
loads/ResearchHighlights/Complaint-to-ombudsman-over-EMA.pdf.
40. Jefferson T, Jørgensen L. Human papillomavirus vaccines, complex regional pain syndrome, pos-
tural orthostatic tachycardia syndrome, and autonomic dysfunction - a review of the regulatory evi-
dence from the European Medicines Agency. Indian J Med Ethics. 2017 Jan;2(1):30-37. doi:
10.20529/IJME.2017.006.
41. McGauran N, Wieseler B, Kreis J, Schüler Y-B, Kölsch H, Kaiser T. Reporting bias in medical re-
search - a narrative review. Trials. 2010 Apr 13;11:37. doi: 10.1186/1745-6215-11-37.
42. Ross JS, Tse T, Zarin DA, Xu H, Zhou L, Krumholz HM. Publication of NIH funded trials registered in
ClinicalTrials.gov: cross sectional analysis. BMJ. 2012 Jan 3;344:d7292. doi: 10.1136/bmj.d7292.
43. Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in random-
ized drug trials: a reflection of treatment effect or adverse events? JAMA. 2003 Aug 20;290(7):921-8.
doi: 10.1001/jama.290.7.921.
44. Eyding D, Lelgemann M, Grouven U, Härter M, Kromp M, Kaiser T, et al. Reboxetine for acute
treatment of major depression: systematic review and meta-analysis of published and unpublished
placebo and selective serotonin reuptake inhibitor controlled trials. BMJ. 2010 Oct 12;341:c4737. doi:
10.1136/bmj.c4737.
45. Jørgensen AW, Hilden J, Gøtzsche PC. Cochrane reviews compared with industry supported meta-
analyses and other meta-analyses of the same drugs: systematic review. BMJ. 2006 Oct
14;333(7572):782. doi: 10.1136/bmj.38973.444699.0B.
46. Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. Industry sponsorship and research outcome.
Cochrane Database Syst Rev. 2017 Feb 16;2:MR000033. doi: 10.1002/14651858.MR000033.pub3.
47. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, Cronin E, et al. Systematic Review of the Empir-
ical Evidence of Study Publication Bias and Outcome Reporting Bias. PLoS One. 2008 Aug
28;3(8):e3081. doi: 10.1371/journal.pone.0003081.
27
Page 27 of 637
48. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to
statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009 Jan
21;(1):MR000006. doi: 10.1002/14651858.MR000006.pub3.
49. Huić M, Marušić M, Marušić A. Completeness and Changes in Registered Data and Reporting Bias
of Randomized Controlled Trials in ICMJE Journals after Trial Registration Policy. PLoS One.
50. Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. Selective reporting bias of harm
outcomes within studies: findings from a cohort of systematic reviews. BMJ. 2014 Nov 21;349:g6501.
doi: 10.1136/bmj.g6501.
51. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of re-
search findings: an updated review of related biases. Health Technol Assess. 2010 Feb;14(8):iii, ix-xi, 1-
193. doi: 10.3310/hta14080.
52. Jefferson T, Jørgensen L. Redefining the ‘E’ in EBM. BMJ Evid Based Med. 2018 Apr;23(2):46-47.
doi: 10.1136/bmjebm-2018-110918.
53. Guidelines: ICH. Available from: http://www.ich.org/products/guidelines.html
54. Maund E, Guski LS, Gøtzsche PC. Considering benefits and harms of duloxetine for treatment of
stress urinary incontinence: a meta-analysis of clinical study reports. CMAJ. 2017 Feb 6;189(5):E194-
E203. doi: 10.1503/cmaj.151104.
55. Hodkinson A, Gamble C, Smith CT. Reporting of harms outcomes: a comparison of journal publica-
tions with unpublished clinical study reports of orlistat trials. Trials. 2016 Apr 22;17(1):207. doi:
10.1186/s13063-016-1327-z.
56. Schroll JB, Penninga EI, Gøtzsche PC. Assessment of Adverse Events in Protocols, Clinical Study Re-
ports, and Published Papers of Trials of Orlistat: A Document Analysis. PLoS Med. 2016 Aug
16;13(8):e1002101. doi: 10.1371/journal.pmed.1002101.
57. Sharma T, Guski LS, Freund N, Gøtzsche PC. Suicidality and aggression during antidepressant treat-
ment: systematic review and meta-analyses based on clinical study reports. BMJ. 2016 Jan 27;352:i65.
doi: 10.1136/bmj.i65.
28
Page 28 of 637
58. Golder S, Loke YK, Wright K, Norman G. Reporting of Adverse Events in Published and Unpublished
Studies of Health Care Interventions: A Systematic Review. PLoS Med. 2016 Sep 20;13(9):e1002127.
doi: 10.1371/journal.pmed.1002127.
59. Lenzer J. FDA is incapable of protecting US “against another Vioxx.” BMJ. 2004 Nov
27;329(7477):1253. doi: 10.1136/bmj.329.7477.1253.
60. Gøtzsche PC. Deadly medicines and organised crime: How big pharma has corrupted health care.
Radcliffe Publishing. 2013.
61. Cohen D. Rosiglitazone: what went wrong? BMJ. 2010 Sep 6;341:c4848. doi: 10.1136/bmj.c4848.
62. Gøtzsche PC, Jørgensen AW. Opening up data at the European Medicines Agency. BMJ. 2011 May
10;342:d2686. doi: 10.1136/bmj.d2686.
63. Doshi P, Jefferson T. Open data 5 years on: a case series of 12 freedom of information requests for
regulatory data to the European Medicines Agency. Trials. 2016 Feb 11;17:78. doi: 10.1186/s13063-
016-1194-7.
64. Jefferson T, Doshi P, Thompson M, Heneghan C, Group CARI. Ensuring safe and effective drugs:
who can do what it takes? BMJ. 2011 Jan 11;342:c7258. doi: 10.1136/bmj.c7258.
65. Gøtzsche PC. Why we need easy access to all data from all clinical trials and how to accomplish it.
Trials. 2011 Nov 23;12:249. doi: 10.1186/1745-6215-12-249.
66. van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in
Cochrane reviews may not be worth the effort. J Clin Epidemiol. 2009 Aug;62(8):838-844.e3. doi:
10.1016/j.jclinepi.2008.09.010.
67. Golder S, Loke YK, Wright K, Sterrantino C. Most systematic reviews of adverse effects did not in-
clude unpublished data. J Clin Epidemiol. 2016 Sep;77:125-133. doi: 10.1016/j.jclinepi.2016.05.003.
68. Baudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of searching clinical trial regis-
tries in systematic reviews of pharmaceutical treatments: methodological systematic review and rea-
nalysis of meta-analyses. BMJ. 2017 Feb 17;356:j448. doi: 10.1136/bmj.j448.
69. Jones CW, Keil LG, Weaver MA, Platts-Mills TF. Clinical trials registries are under-utilized in the
conduct of systematic reviews: a cross-sectional analysis. Syst Rev. 2014 Oct 27;3:126. doi:
10.1186/2046-4053-3-126.
29
Page 29 of 637
70. Potthast R, Vervölgyi V, McGauran N, Kerekes MF, Wieseler B, Kaiser T. Impact of inclusion of in-
dustry trial results registries as an information source for systematic reviews. PLoS One. 2014 Apr
71. Enst WA van, Scholten RJPM, Hooft L. Identification of additional trials in prospective trial regis-
ters for Cochrane systematic reviews. PLoS One. 2012;7(8):e42812. doi: 10.1371/jour-
nal.pone.0042812.
72. Cochrane Handbook. Available from: http://training.cochrane.org/handbook
73. Hróbjartsson A, Boutron I, Turner L, Altman DG, Moher D. Assessing risk of bias in randomised
clinical trials included in Cochrane Reviews: the why is easy, the how is a challenge. Cochrane Data-
base Syst Rev. 2013 Apr 30;(4):ED000058. doi: 10.1002/14651858.ED000058.
74. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collabora-
tion’s tool for assessing risk of bias in randomised trials. BMJ. 2011 Oct 18;343:d5928. doi:
10.1136/bmj.d5928.
75. Cochrane's risk of bias tool version 2.0. Available from: https://sites.google.com/site/riskofbi-
astool/welcome/rob-2-0-tool/current-version-of-rob-2.
76. Interim Clinical Study Report for Study 580299/008 (HPV-008). Available from: https://www.gsk-
clinicalstudyregister.com/files2/gsk-580299-008-clinical-study-report-redact.pdf.
77. Paavonen J, Naud P, Salmerón J, Wheeler CM, Chow S-N, Apter D, et al. Efficacy of human papillo-
mavirus (HPV)-16/18 AS04-adjuvanted vaccine against cervical infection and precancer caused by on-
cogenic HPV types (PATRICIA): final analysis of a double-blind, randomised study in young women.
Lancet. 2009 Jul 25;374(9686):301-14. doi: 10.1016/S0140-6736(09)61248-4.
30
Page 30 of 637
Paper 1
Evaluation of Cochrane’s risk of bias tool
Page 31 of 637
Page 32 of 637
Page 33 of 637
Jørgensen et al. Systematic Reviews (2016) 5:80
DOI 10.1186/s13643-016-0259-8
RESEARCH Open Access
Evaluation of the Cochrane tool for

assessing risk of bias in randomized clinical
trials: overview of published comments and
analysis of user practice in Cochrane and
non-Cochrane reviews
Lars Jørgensen1*, Asger S. Paludan-Müller1, David R. T. Laursen1, Jelena Savović2,3, Isabelle Boutron4,
Jonathan A. C. Sterne2,3, Julian P. T. Higgins2,3 and Asbjørn Hróbjartsson1,5
Abstract
Background: The Cochrane risk of bias tool for randomized clinical trials was introduced in 2008 and has frequently
been commented on and used in systematic reviews. We wanted to evaluate the tool by reviewing published
comments on its strengths and challenges and by describing and analysing how the tool is applied to both Cochrane
and non-Cochrane systematic reviews.
Methods: A review of published comments (searches in PubMed, The Cochrane Methodology Register and Google
Scholar) and an observational study (100 Cochrane and 100 non-Cochrane reviews from 2014).
Results: Our review included 68 comments, 15 of which were categorised as major. The main strengths of the tool
were considered to be its aim (to assess trial conduct and not reporting), its developmental basis (wide consultation,
empirical and theoretical evidence) and its transparent procedures. The challenges of the tool were mainly considered
to be its choice of core bias domains (e.g. not involving funding/conflicts of interest) and issues to do with
implementation (i.e. modest inter-rater agreement) and terminology. Our observational study found that the tool was
used in all Cochrane reviews (100/100) and was the preferred tool in non-Cochrane reviews (31/100). Both types of
reviews frequently implemented the tool in non-recommended ways. Most Cochrane reviews planned to use risk of
bias assessments as basis for sensitivity analyses (70 %), but only a minority conducted such analyses (19 %) because, in
many cases, few trials were assessed as having “low” risk of bias for all standard domains (6 %). The judgement of at
least one risk of bias domain as “unclear” was found in 89 % of included randomized clinical trials (1103/1242).
Conclusions: The Cochrane tool has become the standard approach to assess risk of bias in randomized clinical trials
but is frequently implemented in a non-recommended way. Based on published comments and how it is applied in
practice in systematic reviews, the tool may be further improved by a revised structure and more focused guidance.
Keywords: Cochrane, Systematic review, Bias, Tool, Comment, User practice, Randomized clinical trial
* Correspondence: lj@cochrane.dk; larsjorgensens@gmail.com

1
The Nordic Cochrane Centre, Rigshospitalet 7811, Blegdamsvej 9, 2100
Copenhagen, Denmark
Full list of author information is available at the end of the article
© 2016 Jørgensen et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to
the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Page 34 of 637
Jørgensen et al. Systematic Reviews (2016) 5:80 Page 2 of 13
Background clinical trials and an observational study of how the tool is

Since the early 1990s, the number of published syste- used in systematic reviews (please refer to Additional file 1
matic reviews of randomized trials, both Cochrane and for the study’s PRISMA checklist).
non-Cochrane reviews, has steadily increased. The ideal
of taking a systematic approach to identify, summarise Review of published comments
and analyse comparable clinical trials as a basis for We sought publications that explicitly commented on
therapeutic decisions has become more widespread, and the tool. We defined “major comments” as longer com-
systematic reviews have had a huge impact on clinical ments with a substantial reflection (typically ≥100 words
research and practice. of text) on the strengths or weaknesses of the tool, for
However, one obstacle to the usefulness of a syste- example, in the form of an editorial. We also included
matic review is the possibility that some of the included “minor comments,” which we defined as shorter com-
trials are biased due to flaws in their design, conduct, ments without a substantial reflection (typically <100
analysis or reporting. A meta-analysis of biased effect es- words of text) on the strengths or weaknesses of the
timates will likely produce a biased pooled analysis with tool, for example, in the form of minor elements of a
increased precision and greater credibility. Thus, for au- discussion in a publication. We excluded “peripheral re-
thors of a systematic review, it is paramount to adequately marks” on the tool, which we defined as remarks that
address the risk of bias in the included trials [1]. were implicit or short and tangential. If an author had
For this purpose, the Cochrane tool for assessing risk several publications included with similar comment
of bias in randomized clinical trials (i.e. the tool) was contents, only the publication with the most detailed
released in 2008 and updated in 2011. The tool is based comment was considered major.
on seven bias domains: sequence generation and alloca- We searched PubMed, The Cochrane Methodology
tion concealment (both within the domain of selection Register and Google Scholar for publications from the
bias or allocation bias), blinding of participants and start of 2008 to the end of 2014. No language restriction
personnel (performance bias), blinding of outcome as- was applied, and Google Translate was used for non-
sessors (detection bias), incomplete outcome data (attri- familiar languages. The search strategy was developed
tion bias), selective reporting (reporting bias) and an iteratively (see Additional file 2).
auxiliary domain: “other bias.” For each bias domain, the One author (LJ) decided on inclusion of publications
tool urges users to assign a judgement of “high,” “low” and categorised them as “major comments” and “minor
or “unclear” risk of bias and to document the basis for comments” (and “peripheral remarks”). A second author
their judgements (e.g. with verbatim quotes). The bias (AS) checked the categorisation. Two authors (LJ and
domains of the tool were selected with the intention to AS) extracted data independently. Any disagreements
cover all fundamental bias mechanisms in randomized were solved by discussion and by consulting a third
trials [2]. author (DL or AH).
Several years have passed since the release of the first The following information was extracted: publication
version of the tool. Over this period, the tool has been year, publication type, tool version considered (i.e. 2008
used in numerous systematic reviews, the scientific or 2011) and the exact wording of the comment.
debate on risk of bias has proceeded (for example, Comments from the included publications were cate-
reflecting on the role of source of funding [3–6] or gorised according to whether they expressed “strengths,”
other “meta-biases” [7]) and research publications have “challenges” or “suggestions” and summarised into
analysed user experience [8] and inter-agreement rates broader themes (each addressing a similar type of topic).
[9–11]. Additionally, a complementary tool for assessing We noted the numerical distribution of comparable
non-randomized trials has been developed [12]. comments, but our main intention was a qualitative
Researchers from the original development team and mapping of the themes addressed and a categorisation
members of the Cochrane Bias Methods Group are according to whether they addressed a core design
planning a revision of the tool. To evaluate the tool and feature of the tool or an issue related to implementation.
to provide a better basis for the revision, we intended (1)
to identify, summarise and analyse published comments Observational study of how the tool is used in systematic
on the strengths and challenges of the tool and (2) to reviews
describe and analyse how the tool is used in both One author (DL) identified 100 Cochrane reviews (or
Cochrane and non-Cochrane reviews. Cochrane review updates) from PubMed in reverse
chronological order from 31.12.2014 until 20.11.2014
Methods (see Additional file 2). The same author manually identi-
This study involved a review of published comments on fied 100 non-Cochrane reviews from PubMed in reverse
the Cochrane tool for assessing risk of bias in randomized chronological order from 31.12.2014 until 22.12.2014.
Page 35 of 637
A second author (AS) checked the inclusion. We de- trials and non-randomized clinical trials, we disregarded
fined a non-Cochrane review as a self-declared sys- the non-randomized trials.
tematic review with at least one included randomized
clinical trial. We excluded any non-Cochrane review Results
that was also published as a Cochrane review. Review of published comments
Three authors (AS, DL and LJ) extracted data inde- We read 976 full text publications of which we excluded
pendently: intervention type (pharmacological or non- 908 (Fig. 1). Thus, we included 68 publications, of which
pharmacological); inclusion of meta-analyses; number of we categorised 15 as “major comments” and 53 as
trials and how many trials were categorised as “high,” “minor comments” (Tables 1 and 2).
“unclear” and “low” risk of bias; the method used for The strengths of the tool were addressed in five “major
judging risk of bias (or quality) and how it was imple- comments” relating to three themes: aims, developmen-
mented; the type and frequency of both standard and tal basis and transparency. The comments praised the
non-standard domain use; the use of merging or split- tool for aiming to assess conduct (and not reporting),
ting of standard domains (e.g. merging blinding domains being based on theoretical and empirical evidence and
or splitting for different outcomes); the use of the “other on broad consultation and facilitating transparent assess-
bias” domain; how risk of bias assessments were incor- ment of bias.
porated into statistical analysis using sensitivity analyses; The challenges of the tool were addressed in 15 “major
whether risk of bias judgements were explicitly men- comments” relating to four themes: choice of the core
tioned in the abstract, discussion or conclusion; and bias domains, implementation, overall risk of bias and
whether The Grading of Recommendations Assessment, special situations. The comments on choice of core bias
Development and Evaluation (short GRADE) had been domains expressed concern whether the chosen domains
incorporated. We compared differences in proportions comprehensively address all threats to validity (for ex-
between Cochrane and non-Cochrane reviews using ample, five comments reflected on including funding as
Fisher’s exact test. In cases where Cochrane or non- an independent bias domain). Comments on implemen-
Cochrane reviews included both randomized clinical tation pointed to difficulties in the subjective
Fig. 1 Flowchart of the inclusion of comments on the Cochrane risk of bias tool for randomized clinical trials— evaluation of the Cochrane tool
for assessing risk of bias in randomized clinical trials. 1N= the number of records/comments screened for inclusion. 2Of the 976 full-texts assessed,
793 full-texts did not comment on the Cochrane risk of bias tool for randomized clinical trials (i.e. the tool). 3Seven records (ordered through The
Royal Danish Library) were not retrievable and therefore not assessed. 4183 publications were independently assessed by two authors to check
type, categorisation and commentary. 5Major comments were defined as longer comments with a substantial reflection (typically ≥100 words of
text) on the strengths or challenges of the tool. 6Minor comments were defined as shorter comments without a substantial reflection (typically
<100 words of text) on the strengths or challenges of the tool. 7Peripheral remarks (defined as implicit or short and tangential) were excluded
Page 36 of 637
Table 1 Characteristics of published comments on the Cochrane All themes addressed in the “major comments” were
risk of bias tool for randomized clinical trials— evaluation of the represented in the “minor comments” (see Additional
Cochrane tool for assessing risk of bias in randomized clinical trials file 2). Additional themes addressed only in the “minor
Publication characteristics Number of comments: 68 (100 %) comments” included graphical representation, external
Publication category validity and non-randomized designs. Specifically, (i) one
Majora 15 (22 %) comment praised the tool for its graphical representa-
Minorb 53 (78 %)
tion of risk of bias assessments, (ii) one comment criti-
cised that the tool does not address external validity
Publication type
(and only focuses on internal validity) and (iii) one com-
Comment/editorial/letterc 6 (9 %) ment noted that non-randomized trials should be in-
Survey/qualitative case study 33 (49 %) cluded in Cochrane reviews and should be addressed in
Experimental/observational study 23 (33 %) risk of bias assessments. The latter two suggestions are
Other 6 (9 %) inconsistent with the aim of the tool, which is to assess
Tool version considered/applied
only bias (i.e. internal validity) in randomized clinical tri-
als. Such comments help to unveil the assumptions and
2011 54 (79 %)
basic structure of the tool but would be difficult to
2008 6 (9 %) implement without significantly changing the tool.
Not specified 8 (12 %) Other comments reflected concerns about the imple-
Year of publication mentation of the tool. An example is the suggestion for
2008–2010 9 (13 %) improved guidelines for how to assess selective outcome
2011 10 (15 %)
reporting. Also, improved training options and more
detailed guidelines aimed to improve agreement rates
2012 8 (12 %)
address the implementation of the tool. Such suggestions
2013 14 (20 %) are easier to implement while keeping the fundamental
2014 27 (40 %) structure of the tool intact.
a
Major comments were defined as longer comments with a substantial
reflection (typically ≥100 words of text) on the strengths or challenges of the
Cochrane risk of bias tool for randomized clinical trials (i.e. the tool)
Analysis of user patterns in systematic reviews
b
Minor comments were defined as shorter comments without a substantial All Cochrane reviews assessed risk of bias using the
reflection (typically ≤100 words of text) on the strengths or challenges of the tool. Cochrane risk of bias tool (100/100, 100 %) (Tables 3
c
Comments, editorials and letters (to the editor) were defined as such
if self-declared and 4). Most of the non-Cochrane reviews assessed risk
of bias (80/100, 80 %), with the Cochrane tool being the
interpretation of the tool and expressed concerns about most frequently used (31/80, 39 %). Other tools and
modest inter-observer agreement, difficulty in assessing scales used to assess risk of bias included the Jadad
selective reporting of outcomes, terminological ambigu- Quality Assessment Scale (19/80, 24 %) [13] and the
ity (i.e. of the terms subjective/objective) and the low Physiotherapy Evidence Database (short PEDro) scale
proportion of reviews using risk of bias assessments as a (5/80, 6 %) [14] (Table 4).
basis for sensitivity analyses. The comments on overall The majority of Cochrane reviews included one or
risk of bias expressed concern about the challenges in more meta-analyses (85/100, 85 %). According to the in-
assigning an overall risk of bias to a trial based on risk formation reported in their methods section, most of the
of bias of single domains to the trial as such. A single Cochrane reviews had planned to perform sensitivity
comment regarded the special situation where the tool analyses based on risk of bias (70/100, 70 %). One fifth
was used to assess risk of bias based on clinical study re- of the Cochrane reviews reported to have performed
ports (and not clinical trial publications). sensitivity analyses (19/100, 19 %). Few reviews based
Specific suggestions to improve the tool were included sensitivity analyses on an overall risk of bias (2/19,
in nine “major comments” relating to three themes: im- 11 %). Most reviews based sensitivity analyses on indi-
proved guidelines, further research and the inclusion of vidual bias domains (9/19, 47 %) or did not state what
funding as a bias domain. The comments on guidelines sensitivity analyses were based on (8/19, 42 %). The ma-
suggested that updated and improved guidance and jority of the Cochrane reviews who did not conduct the
more training options for users were needed. The com- planned analyses reported that the lack thereof was due
ments on research suggested further methodological re- to insufficient data (41/50, 82 %), either because there
search (for example, blind versus non-blind risk of bias were few trials included in the review or few trials with
assessments). The comments on funding suggested that “low” risk of bias. The remaining reviews did not explain
funding/conflicts of interest should be incorporated into why they did not perform the planned analyses (9/50,
the tool as a specific bias domain. 18 %) (Tables 3 and 4).
Page 37 of 637
Table 2 Selected key points of major comments on the Cochrane risk of bias tool for randomized clinical trials: strengths,
challenges and suggestions
First authora Category Theme Key point
Armijo-Olivo Strengths None mentioned
Challenges • Implementation (1.) 1. “…the large number of trials classified as high or unclear RoB
• Overall risk of bias (2.) [risk of bias] casts doubts about the discrimination power of the RoB
• Bias domains (3.) [risk of bias] tool to […] explain variability of treatments effects
across studies…”
2. “…the overall assessment of the RoB [risk of bias] may not be useful
to determine quality of individual trials.”
3. “…other methodological factors could be important for evaluating
RoB and could be considered for inclusion in the RoB [risk of bias]
tool after careful empirical evidence testing.”
Suggestions • Guidelines (1.) 1. “Improved guidelines to apply the RoB [risk of bias] tool and revisions
to the tool for different health areas are needed.”
Bero Strengths None mentioned
Challenges • Bias domains (1.) 1. “The current Cochrane risk of bias tool is insufficient to assess bias
related to study funding sources.”
Suggestions • Funding (1.) 1. “…the Cochrane risk of bias tool should include funding source as a
standard item because: 1. Funding source fits the definition of bias, 2. There
is empirically-based evidence of bias related to funding source, 3. The observed
bias related to funding source cannot be captured by the risk of bias criteria
currently assessed with the risk of bias tool, 4. Risks of bias are not mutually
exclusive, 5. Bias may be related to funding source even when all studies
are industry-funded.”
Boutron Strengths • Aims (1. 2.) 1. “…the tool aims at being completely transparent, with a separation of
• Improvement (3.) the facts and reviewers’ judgments. This aim is particularly important
• Transparency (1.) because reviewers, editors, and readers can challenge the author on
the judgment.”
2. “…the tool is intended to assess the risk of bias related to the design,
conduct, and analyses of the trial and not the quality of reporting.”
3. “This tool has been an important step forward in the assessment of
the risk of bias in systematic reviews and meta-analyses.”
Challenges None mentioned
Suggestions None mentioned
De Bruin Strengths None mentioned
Challenges • Implementation (1.) 1. “…many do assess methodological quality, but very few incorporate them
[/risk of bias assessments] in their analyses.”
Suggestions • Guidelines (1.) 1. “…systematic reviewers could consider adapting the risk-of-bias tool to
the literature…”
Hartling Strengths None mentioned
Challenges • Implementation (1.) 1. “Low agreement between reviewers suggests the need for more specific
• Overall risk of bias (2.) guidance regarding interpretation and application of the Risk of Bias (ROB) tool…”
• Special situations (3.) 2. “The majority of trials in the sample were assessed as high or unclear risk
of bias…This raises concerns about the ability of the ROB [risk of bias] tool
to detect differences across trials that may relate to biases in estimates of
treatment effects.”
3. “…trials with different design features (e.g., crossover) or hypotheses
(e.g., equivalence, non-inferiority), and those examining non-pharmacological
interventions appear to create more ambiguity for risk of bias assessments.”
Suggestions • Guidelines (1.) 1. “There is a need for more detailed guidelines to apply […] the ROB
[risk of bias] tool and […] further testing with the modified tool is warranted.”
Hróbjartsson Strengths • Aims (1.) 1. “The risk of bias tool provides a standardised approach, based on items
• Background (1.) selected on both theoretical and empirical grounds, and following broad
consultations with clinical research methodologists.”
Challenges • Bias domains (2.) 1. “The risk of bias tool is a comparatively recent development that still
• Implementation (1.) likely needs refinement.”
2. “It is not clear that the risk of bias tool in its present version addresses
this problem [of funding] adequately.”
Page 38 of 637
challenges and suggestions (Continued)
Ivers Strengths None mentioned
Challenges • Bias domains (1.) 1. “The risk of bias tool does not capture all sources of methodological bias
• Implementation (2.) and poor reporting interferes with the assessment of many domains.”
• Overall risk of bias (3.) 2. “While the overall risk of bias assessment using the Cochrane Risk of Bias
Tool has been shown to differentiate effect sizes (i.e. higher risk of bias studies
usually have larger effect sizes), 10 studies at high risk of bias may still offer
valuable knowledge…”
3. “…assigning trials with high risk of bias in a single domain a status of high
risk of bias overall may be arguable.”
Jefferson Strengths • Aims (1.) 1. “The real strength of the risk of bias tool appears not to be in the final
judgements it enables, but rather in the process it helps facilitate: critical
assessment of a clinical trial.”
Challenges • Bias domains (1. 3.) 1. “The current Cochrane risk of bias tool is not adequate for the task as it
• Implementation (2.) does not reliably identify all types of important biases, and nor does it organise
and check the coherence
of large amounts of information.”
2. “We found the Cochrane risk of bias tool to be difficult to apply to clinical
study reports…[since]…its use lends itself to a checklist approach (in which
each design item is sought and, if found, eliminated from the bias equation
rather than with thought and consideration).”
3. “Many of the variables we found to be important when assessing the trial
(e.g. date of trial protocol, date of un-blinding, date of participant enrolment)
are simply not captured in the risk of bias tool…”
Katikireddi Strengths None mentioned
Challenges • Implementation (1.) 1. “…reviewers are struggling to understand and/or operationalize current
guidance on how to conduct and incorporate critical appraisal [/risk of bias]
within synthesis.”
Suggestions • Guidelines (1.) 1. “Further research is required to establish the relative importance of different
• Research (1.) forms of bias and their likely impact […] and also to clarify how critical
appraisals should be incorporated into SR [systematic review] findings.”
Morissette Strengths • Aims (1.) 1. “The Cochrane 'Risk of bias' tool differs from other quality appraisal tools
because it questions the degree to which a study’s results should be believed…”
Challenges • Implementation (1.) 1. “The results of our review provide no clear guidance as to whether risk of
bias assessments should be completed in a blind or un-blind manner.”
Suggestions • Research (1.) 1. “…we encourage further research in this area [of blind vs. un-blind risk of
bias assessment] and recommend using all of the important components of
the Cochrane 'Risk of bias' tool.”
Moustgaard Strengths None mentioned
Challenges • Implementation (1.) 1. “No characterization of subjective vs. objective outcomes relevant to risk of
performance bias is given explicitly in the Cochrane Handbook nor did we find
it in the methodological articles or the clinical trial reports we reviewed.”
Roseman Strengths None mentioned
Challenges • Bias domains (1.) 1. “…inclusion of risk of bias from conflicts of interest could reflect mechanisms
through which industry involvement can influence study outcomes that are
not fully captured by the current domains of the risk of bias tool.”
Suggestions • Funding (1.) 1. “…we recommend that the Cochrane Collaboration reconsider its position
that trial funding and trial author-industry financial ties not be included in the
risk of bias assessment.”
Savović Strengths • Aims (3.) 1. “…[the tool has] a standardized approach to bias assessments…”
• Background (1.) 2. “…[the tool has] transparency provided by requesting quotes…”
• Transparency (2.) 3. “…[the tool provides] a platform to encourage critical thinking.”
Page 39 of 637
challenges and suggestions (Continued)
Challenges • Bias domains (1.) 1. “Some of the items that authors have included (such as sample size
• Implementation (2.) calculations and funding source) are explicitly discouraged in the
Cochrane Handbook guidance. While there is evidence that some factors
are empirically associated with effect estimates, such as single versus
multicentre design, early stopping of trials and funding source [14-16],
the extent to which these should be considered alongside the main bias
domains is still a topic of debate.”
2. “The main purpose of this evaluation was to identify potential
problems with the RoB [risk of bias] that can be rectified, and we suspect
that users who encountered problems are more likely to have responded.
This speculation is based on the high proportion of respondents who
reported having problems with some aspects of the RoB tool, especially
with individual RoB domains.”
Suggestions • Guidelines (1.) 1. “It is important that guidance and training materials continue to be
developed for all aspects of the tool…”
Sterne Strengths None mentioned
Challenges • Bias domains (1.) 1. “The current RoB [risk of bias] tool does not work well for assessment of
selective reporting.”
Suggestions • Funding (1.) 1. “…the Cochrane risk of bias tool should not include funding source as a
standard item.”
Vale Strengths None mentioned
Challenges • Implementation (1. 2.) 1. “The Cochrane Handbook states that because the ability to measure the
• Bias domains (2.) true bias (or even the true risk of bias) is limited, then the possibility to
validate a tool to assess that risk is also limited. Nevertheless, authors of
Cochrane systematic reviews are required to use the Cochrane risk of bias tool.”
2. “Assessing risk of bias was particularly difficult for the more subjective
domains [i.e. ‘selective outcome reporting’ and ‘other bias’].”
Major comments were defined as longer comments with a substantial reflection (typically ≥100 words of text) on the strengths or challenges of the Cochrane risk
of bias tool for randomized clinical trials (i.e. the tool)
a
See Additional file 2 for references
One tenth of the non-Cochrane reviews that had any differences were found between the non-Cochrane re-
risk of bias assessment reported plans for sensitivity views that used the Cochrane tool versus the non-
analyses based on risk of bias assessments (8/80, 10 %). Cochrane reviews that used other risk of bias tools
One in seven of all the non-Cochrane reviews reported when comparing the use of risk of bias results in the
to have performed sensitivity analyses based on risk of abstract and discussion/conclusion.
bias or quality assessments (11/80, 14 %). In nine reviews, The majority of Cochrane reviews (64/100, 64 %) and
the sensitivity analyses were based on an overall risk of few non-Cochrane reviews (4/80, 5 %) incorporated
bias (9/11, 82 %) (Table 4). GRADE in their overall assessment of confidence in the
Two Cochrane reviews performed subgroup analyses results (Table 4).
(both with “low” versus “high” risk of bias) (2/100, 2 %). The majority of Cochrane reviews applied all standard
None of the non-Cochrane reviews performed subgroup domains (59/100, 59 %). Only few Cochrane reviews
analyses based on risk of bias. explicitly assessed risk of bias on an outcome level (i.e.
Most Cochrane reviews explicitly commented on risk differentiating between subjective versus objective out-
of bias assessments in the discussion and/or conclusion comes) (12/100, 12 %). Most Cochrane reviews (88/100,
(89/100, 89 %), although fewer incorporated this infor- 88 %) performed one risk of bias assessment without
mation into the abstract (80/100, 80 %). Most of the making it clear whether this assessment concerned a sin-
non-Cochrane reviews that applied the Cochrane tool gle outcome, a group of outcomes or the trial as a whole.
and some of the non-Cochrane reviews that applied A similar pattern was seen for non-Cochrane reviews
non-Cochrane tools explicitly commented on risk of (Table 5).
bias assessments in the discussion and/or conclusion One third of the Cochrane reviews merged standard
(Cochrane tool: 25/31, 81 %; non-Cochrane tools: 12/ bias domains (37/100, 37 %), most often merging “per-
49, 24 %) and more than half incorporated this infor- formance bias” and “detection bias” into a single blind-
mation into the abstract (Cochrane tool: 18/31, 58 %; ing bias domain (31/37, 84 %) (predominantly done in
non-Cochrane tools: 30/49, 61 %). No significant updates of reviews that had originally used the 2008
Page 40 of 637
Table 3 Characteristics of included Cochrane and non-Cochrane reviews— evaluation of the Cochrane tool for assessing risk of bias
in randomized clinical trials
Publication characteristics 100 Cochrane reviews (100 %) 100 non-Cochrane reviews (100 %) P value*
Intervention
Pharmacologic 55 (55 %) 29 (29 %) 0.020
Non-pharmacologic 45 (45 %) 71 (71 %) 0.061
Review has ≥1 meta-analysis
Yes 85 (85 %) 45 (45 %) 0.0065
Included trials
Number of randomized clinical trials in total 1242 1249
-Lowa risk of bias 74 (6 %) 25 of 424e (6 %) 1.00
b e
-Unclear risk of bias 407 (33 %) 226 of 424 (53 %) 0.0001
-Highc risk of bias 761 (61 %) 173 of 424e (41 %) 0.0001
f
Reviews with ≥1 low risk of bias trial and ≥1 high risk of bias trial 26 (26 %) 6 of 18 (33 %) 0.60
Reviews with ≥1 low risk of bias trial and ≥1 high or unclear 32 (32 %) 8 of 18f (44 %) 0.47
risk of bias trial
Number of randomized clinical trials includedd in a review
-One to five 39 (39 %) 38 (38 %) 1.00
-Six to ten 23 (23 %) 26 (26 %) 0.75
->Ten 38 (38 %) 36 (36 %) 0.89
*P values were calculated with Fisher’s two-tailed exact test
a
If a trial had all standard domains (not including the “other bias” domain) judged as “low” risk of bias, we defined the trial as “low risk of bias”
b
If a trial had at least one of the standard domains (not including the “other bias” domain) judged as “unclear” risk of bias and no domains judged as “high” risk
of bias, we defined the trial as “unclear risk of bias.” The judgement of at least one standard risk of bias domain (not including the “other bias” domain) as
“unclear” was found in 1103 of 1242 included randomized clinical trials (89 %)
c
If a trial had at least one of the six standard domains (not including the “other bias” domain) judged as “high” risk of bias, we defined the trial as “high risk of bias”
d
We only included systematic reviews with one or more randomized clinical trials included in their analyses
e
It was only possible to assess whether a trial was judged as “low,” “unclear” or “high” risk of bias in 18 non-Cochrane reviews (which provided information on risk
of bias judgements for all six standard domains (not including the “other bias” domain) for individual trials via a “risk of bias graph/summary” or “characteristics of
studies” section)
f
The 18 non-Cochrane reviews included 424 randomized clinical trials in total
version of the tool in which the domains were merged of the non-Cochrane reviews reported what specific
(21/31, 68 %)). Approximately one fifth of the Cochrane items were considered as “other biases” (Table 6).
reviews split a standard bias domain into separate sub- Very few of the randomized clinical trials included in
entities (18/100, 18 %), for example, blinding (within the the Cochrane reviews had all standard domains judged
performance bias domain) was split into blinding of as “low” risk of bias (74 of 1242 trials, 6 %). Most had at
personnel and blinding of patients or incomplete out- least one standard domain judged as “unclear” risk of
come data (i.e. attrition bias) was split into assessment bias (407 of 1242 trials, 33 %) or as “high” risk of bias
of intention-to-treat and assessment of dropouts. Again, (761 of 1242 trials, 61 %). A similar pattern was found
a similar pattern was seen for non-Cochrane reviews for the non-Cochrane reviews (Table 3).
(Table 5). Thus, only a few reviews could conduct sensitivity
A minority of Cochrane reviews added non-standard analyses based on overall risk of bias, e.g. the Cochrane
bias domains to the tool (11/100, 11 %). “Baseline imbal- reviews with at least one trial with all standard domains
ance” (6/11, 55 %) and “funding/conflicts of interest” (5/ judged as “low” risk of bias and at least one trial with
11, 45 %) were the most used. A similar pattern was one bias domain judged as “high” risk of bias (26/100,
found for non-Cochrane reviews (Table 6). The majority 26 %) (or as “high”/“unclear” risk of bias (32/100, 32 %)).
of Cochrane reviews used the “other bias” domain op- A similar pattern was found for the non-Cochrane
tion for the same purpose (73/100, 73 %). “Baseline im- reviews (Table 3).
balance” (33/73, 45 %) and “funding/conflicts of interest”
(23/73, 32 %) were also the most used “other biases.” Discussion
Most non-Cochrane reviews that used the Cochrane tool Published comments about the Cochrane risk of bias
included the “other bias” domain (17/31, 55 %), but none tool considered it to be an important step forward but
Page 41 of 637
Table 4 User patterns of risk of bias implementations in Cochrane and non-Cochrane reviews— evaluation of the Cochrane tool for
assessing risk of bias in randomized clinical trials
Risk of bias implementation 100 Cochrane reviews (100 %) 100 non-Cochrane reviews (100 %) P value*
Risk of bias assessment in reviews
Any risk of bias (or quality) assessment 100 (100 %) 80 (80 %) 0.30
Cochrane risk of bias tool 100 (100 %) 31 of 80 (39 %) 0.0002
Jadad scale 0 (0 %) 19 of 80 (24 %) 0.0001
PEDro scale 0 (0 %) 5 of 80 (6 %) 0.019
Own construct or other scale 0 (0 %) 25 of 80 (31 %)c 0.0001
Descriptive use of risk of bias assessment
Explicit mentions risk of bias in abstract 80 (80 %) 18 of 31d (58 %) 0.42
d
Explicit mentions risk of bias in discussion/conclusion 89 (89 %) 25 of 31 (81 %) 0.76
Explicit mentions risk of bias in both abstract and discussion/conclusion 73 (73 %) 15 of 31d (48 %) 0.31
Sensitivity and subgroup analyses based on risk of bias
Review planned (in methods) to do sensitivity analyses 70 (70 %) 8 of 80 (10 %) 0.0001
Review performed sensitivity analyses 19 (19 %) 11 of 80 (14 %) 0.55
Based on overall risk of bias 2 of 19 (11 %) 9 of 11 (82 %) 0.015
Based on individual risk of bias domains 9 of 19 (47 %) 2 of 11 (18 %) 0.45
Unclear what analyses were based on 8 of 19 (42 %) 0 of 11 (0 %) 0.077
Review performed, but did not plan sensitivity analyses 1 of 19 (5 %) 8 of 11 (72 %) 0.0084
Review performed subgroup analysesa 2 (2 %) 0 of 80 (0 %) 0.50
Review planned, but did not perform analyses 50 of 70 (71 %) 5 of 8 0.52
Due to insufficient datab 41 of 50 (82 %) 3 of 5 0.73
No explanation provided 9 of 50 (18 %) 2 of 5 0.33
GRADE
Review incorporated GRADE 64 (64 %) 4 of 80 (5 %) 0.0001
a
All subgroup analyses were based on “low” versus “high” risk of bias
b
“Insufficient data” was due to few trials included in the review or few trials judged as “low risk of bias”
c
15 non-Cochrane reviews made their own risk of bias construct/tool, eight incorporated two constructs/tools and the following constructs/tools (/methods) were
used 18 times in total: CASP (×2), CEBM, Chalmers, CONSORT (×2), CTAM, Downs and Black criteria (×2), Evidence-based medicine toolkit, GRADE (×2), Methods
Guide for Effectiveness and Comparative Effectiveness Reviews, MOOSE (×2), Newcastle Ottawa, QUOROM and STROBE
d
31 of 100 non-Cochrane reviews used the Cochrane risk of bias tool for randomized clinical trials (i.e. the tool) and were compared to the 100 Cochrane reviews
that used the tool for randomized clinical trials
highlighted some challenges including its omission of complements previous studies of user experience [8]
funding/conflicts of interest and its modest inter- and inter-observer variance [9–11].
agreement rates. Suggestions for improvement included It is challenging to search for published comments as
more explicit guidelines and training options. The tool not all are indexed in standard databases. However, we
was used in 100 % of Cochrane reviews and in 31 % of focused on “major comments,” which are more reliably
non-Cochrane reviews in a sample published towards identified. It is reasonable to assume that the threshold
the end of 2014. Often the tool was implemented in a for publishing a comment pointing out a problem with
non-recommended way. Also, 70 % of Cochrane reviews the tool (and maybe suggesting an improvement) is
planned to use the risk of bias assessment as basis for lower than for publishing a comment praising the tool.
sensitivity analyses, but only 19 % of Cochrane reviews Thus, we consider the qualitative summary of the
conducted such analyses, in many cases, because there expressed themes as more interesting than the quantita-
were few trials with “low” risk of bias. tive distribution of the themes. The analyses of how the
tool was used were based on samples of representative
Strengths and weaknesses and contemporary Cochrane and non-Cochrane reviews,
We are not aware of other reviews of published com- enabling both a description and comparison between the
ments on the Cochrane risk of bias tool. Our study two types of reviews.
Page 42 of 637
Table 5 Use of risk of bias and risk of bias domains in the Cochrane and non-Cochrane reviews that applied the Cochrane risk of
bias tool for randomized clinical trials
Use of risk of bias and risk of bias domains 100 Cochrane reviews (100 %) 31 non-Cochrane reviews (100 %)f P value*
Use of risk of bias
Summarises risk of bias on an outcome levela 12 (12 %) 2 (6 %) 0.73
b
Unclear what level risk of bias was summarised on 88 (88 %) 29 (94 %) 0.88
Use of risk of bias standardc domains
Review uses the 2011 tool version 100 (100 %) 26 (84 %) 0.65
Review uses all standardc domains 59 (59 %) 16 (52 %) 0.73
-Sequence generation 100 (100 %) 30 (97 %) 1.00
-Allocation concealment 100 (100 %) 30 (97 %) 1.00
-Blinding of patients and care providers 62 (62 %) 21 (68 %) 0.87
-Blinding of outcome assessors 65 (65 %) 20 (65 %) 1.00
-Incomplete outcome data 99 (99 %) 29 (94 %) 0.88
-Selective reporting 87 (87 %) 25 (81 %) 0.88
c
Merging and splitting of standard domains
Review merges two standardc domains 37 (37 %) 8 (26 %) 0.53
d
-Merges risk of bias domains on an outcome level 6 of 37 (16 %) 0 of 8 (0 %) 0.57
-Does not merge risk of bias domains on an outcome level 31 of 37 (84 %) 8 of 8 (100 %) 0.79
c e
Review splits a standard domain into two or more domains 18 (18 %) 7 (23 %) 0.62
a
One or more domains were separately assessed for more than one outcome or groups of outcomes (i.e. subjective versus objective outcomes)
b
Review has a singular risk of bias assessment despite more than one outcome included in the review. No review based its risk of bias assessment on a singular
or primary outcome
c
The six standard domains (not including the “other bias” domain) included in the Cochrane risk of bias tool for randomized clinical trials (i.e. the tool)
d
i.e. merges blinding of patients and care providers with blinding of outcome assessors into one blinding domain and evaluates blinding for subjective/objective
or explicit (≥2) outcomes.
e
i.e. splits blinding of patients and care providers into blinding of personnel and blinding of patients or splits incomplete outcome data into assessment of
intention-to-treat and assessment of dropouts.
f
Other similar studies scale was used less often). A low proportion of reviews
Based on feedback from focus groups and an online incorporated sensitivity analyses based on risk of bias in
survey, Savović and colleagues concluded that users of their conclusion.
the Cochrane tool identified positive experiences and Our study confirms and expands on the findings of
perceptions of the tool and that revisions and associated Hopewell and colleagues. We found that all 100
guidance as well as improved provision of training may Cochrane reviews in our sample used the Cochrane risk
improve implementation [8]. Several studies have ana- of bias tool, but that only one in five Cochrane reviews
lysed the assessment of risk of bias in systematic reviews conducted sensitivity analyses based on risk of bias as-
[10–15]. Hartling and colleagues and Armijo-Olivo and sessments, despite the fact that seven in ten had planned
colleagues concluded unsatisfactory agreement rates by to do so.
users of the tool and suggested the need for more de-
tailed guidance in assessing the risk of bias [9, 15]. Com- Mechanisms and implications
ments made by the authors of all three studies are Based on the degree of implementation, the tool has
included in our study. proven successful. All Cochrane reviews and a fair
Hopewell and colleagues [16] studied assessment of proportion of non-Cochrane reviews used the tool in
risk of bias in Cochrane and non-Cochrane reviews 2014. However, the tool is often used in ways not
indexed in The Database of Abstracts of Reviews of Ef- recommended.
fects (DARE) [17] and published in 2012. They reported Firstly, both Cochrane and non-Cochrane reviews im-
that all reviews incorporated some kind of assessment of plemented non-standard domains, either as fully new
risk of bias, even though Cochrane reviews more often domains or incorporated into the “other bias” function.
specified which tool was used. Also, the Cochrane tool Approximately one in six Cochrane reviews added
was used more often in Cochrane reviews (and the Jadad “intervention differed between groups” under “other
Page 43 of 637
Table 6 Use of additional non-standard domains and the “other bias” domain in the Cochrane and non-Cochrane reviews that
applied the Cochrane risk of bias tool for randomized clinical trials
Use of additional domains and “other bias” 100 Cochrane reviews (100 %) 31 non-Cochrane reviews (100 %)c P value*
Additional domains
Any additional domain(s) 11 (11 %) 6 (19 %) 0.37
-Adds “baseline imbalance” 6 of 11 (55 %) 2 of 6 (33 %) 1.00
-Adds “funding” or “conflicts of interest” 5 of 11 (45 %) 1 of 6 (17 %) 0.62
-Adds “intention to treat” 2 of 11 (18 %) 2 of 6 (33 %) 0.62
-Adds “compliance” 2 of 11 (18 %) 1 of 6 (17 %) 1.00
-Adds “follow up” 3 of 11 (27 %) 2 of 6 (33 %) 1.00
-Adds “timing of outcome assessment” 2 of 11 (18 %) 1 of 6 (17 %) 1.00
-Adds “overall risk of bias” 1 of 11 (9 %) 4 of 6 (67 %) 0.14
-Adds other additional domaina 6 of 11 (55 %) 2 of 6 (33 %) 1.00
b
Other bias
Includes the “other bias” domain 73 (73 %) 17 (55 %) 0.41
-Used for “baseline imbalance” 33 of 73 (45 %) 0 of 17 (0 %) 0.0059
-Used for “funding” or “conflicts of interest” 23 of 73 (32 %) 2 of 17 (12 %) 0.24
-Used for “intervention differed between groups” 16 of 73 (22 %) 0 of 17 (0 %) 0.069
-Used for “unclear reporting by trial publication author” 15 of 73 (21 %) 0 of 17 (0 %) 0.12
-Used for “trial design” 11 of 73 (15 %) 0 of 17 (0 %) 0.20
a
All of the following other additional domains appeared ones in review samples: Cochrane reviews: “co-interventions avoided or similar,” “confounding variables,”
“definition of incomplete response,” “definition of local recurrence,” “method of follow up” and “size”; non-Cochrane reviews: “co-intervention” and “double blinding”
b
“Other bias”—comments were interpreted and categorised (e.g. the “other bias” comment “There were baseline differences between groups.” was categorised as
“baseline imbalance”). The five most used “other bias”—categories are listed
c
bias,” though this problem is intended to be addressed (or authors simply do not use all sources of information
under “performance bias.” Furthermore, a similar pro- as recommended and possibly opt for “unclear” based
portion of Cochrane reviews added “unclear reporting” on the published report). A better guideline on how to
under “other bias,” although the tool specifically ad- move from the level of individual bias domains to an
dresses conduct and not reporting (unclear reporting overall risk of bias is warranted.
would normally result in contacting trial authors for Thirdly, most reviews based their risk of bias assess-
clarification). Thus, there seems to be a widespread un- ment on a singular risk of bias assessment despite in-
certainty as to the scope of what the tool seeks to evalu- cluding more than one outcome and several reviews
ate. Adding bias domains and using the “other bias” (mostly updates) merged “blinding of participants and
option are primarily intended for special situations, for personnel” and “blinding of outcome assessor” into a
example, when assessing crossover trials. Thus, better single blinding bias domain. The latter was recom-
guidance as to what is meant by “bias,” “bias domain” mended in the 2008 version of the tool, but not in the
and the basic purpose of the tool is warranted. updated 2011 version [18]. Hopefully, the merging of
Secondly, only a minority of reviews used the risk of blinding associated bias domains will be addressed when
bias assessments as a basis for sensitivity analyses. This the reviews in question are updated (again).
problem seems to be a result of few trials having a “low” Fourthly, risk of bias is very often assessed based on
risk of bias, although sensitivity analyses may be based incomplete or missing information. The judgement of at
on “unclear” versus “high” risk of bias. Only 6 % of the least one risk of bias domain as “unclear” was found in
trials included in our review sample had been classified 1103 of 1242 included randomized clinical trials (89 %).
as “low” risk of bias for all domains. It is unclear Though “unclear” may be a reasonable option in some
whether such a low proportion (also found by e.g. Har- trials, this large proportion is a considerable problem. In
tling and colleagues [9] and Hopewell and colleagues many cases, the uncertainty can be resolved by contact-
[16]) is a fair reflection of the “true” risk of bias in trials ing trial authors (who are often able to provide the infor-
or whether the tool as currently applied is too sensitive mation) or by searching publicly available trial registers.
Page 44 of 637
Occasionally, one may access trial protocols, internal trials have been reported in meta-epidemiological studies
company study reports or reports by drug regulation to be associated with exaggerated treatment effects, such
agencies (such as the United States’ Food & Drug as sample size [25], development country status [26], sin-
Administration) to facilitate better risk of bias judge- gle centre status [27] and stopping a trial early [28]. The
ments [19]. Improved guidelines on how to access list of potential bias domains selected purely on empirical
and acquire the relevant information for assessing grounds will quickly become quite large and involve a risk
risk of bias are warranted. of spurious inclusion of bias domains that are secondary
Furthermore, low inter-rater agreement rates for risk in nature (and thus, in principle, explainable by the core
of bias assessors are a potential problem for users of sys- bias domains). However, an open question is whether a
tematic reviews. Readers may consider whether a re- pragmatic and careful selection of a few empirically
view’s conclusion would have been different if other defined bias domains that are simple to assess (such as
reviewers had assessed the risk of bias in the included sample size or single centre status) may act as proxy mea-
trials. It is prudent to check the risk of bias assessments sures and supplement a risk of bias tool based on mechan-
in a review. Fortunately, the tool has a configuration that istically defined core bias domains.
facilitates such checking. Studies assessing between-rater
agreement for complex assessment procedures often Conclusions
have modest agreement rates [20], which in some cases Based on published comments, the Cochrane tool for
may be improved with training [21]. The Cochrane tool assessing risk of bias in randomized clinical trials is
is no exception. Disagreement seems to occur when ter- regarded as an important step forward but challenged by
minology is used inconsistently (e.g. for blinding [22]), how to deal with the risk of bias associated with fund-
when judgements are based on insufficient information ing/conflicts of interest and modest inter-rater agree-
or when the intervention is more complex (e.g. in non- ment. The tool is used in a very high proportion of
pharmacological trials [9]). In addition, reviewers often Cochrane reviews and in many non-Cochrane reviews,
encounter problems when assessing the domains “in- but often in a non-recommended way, for example, by
complete outcome data” and “selective outcome report- incorporating additional bias domains. The tool has
ing” [8]. Clarified terminology, revised structure, better become the standard approach to assess risk of bias in
training options and guidance will hopefully improve randomized clinical trials. Its implementation may be
agreement rates. It will be interesting to read the result further improved by a revised structure, further research
of a forthcoming study on the impact of training [23]. and more focused guidance.
Funding/conflicts of interest is also a challenge for the
tool. It is widely believed that industry funding and other Additional file
conflicts of interest are associated with higher estimates
of treatment effects in randomized trials [24]. It is more Additional file 1: PRISMA 2009 Checklist. (DOCX 179 kb)
controversial whether this association is appropriately Additional file 2: Appendices - Evaluation of the Cochrane tool for
accounted for by adding “funding/conflicts of interest” assessing risk of bias in randomized clinical trials. (DOCX 229 kb)
as an independent bias domain. Adding a domain would

Abbreviations
go against the logic structure of the tool, which is based
DARE: The Database of Abstracts of Reviews of Effects; GRADE: The Grading
on core bias domains that reflect fundamental, inde- of Recommendations Assessment, Development and Evaluation;
pendent bias mechanisms. An alternative option would PEDro: Physiotherapy Evidence Database.
be to address the issue within the existing bias domains
Competing interests
(for example, under risk of selective outcome reporting), All authors are affiliated with the Cochrane Collaboration. JH, JS, IB, JACS
while paying careful attention to any clinical or meth- and AH have comments included in our review of published comments.
odological differences between industry funded and non- We have no further conflicts of interest to declare.
funded trials, such as selection of control groups. The Authors’ contributions

problem with the latter option is that detailed informa- LJ contributed to the design of the study, the collection and assembly of data,
tion on trial conduct is often missing. It is notable that the analysis and interpretation of the data, the drafting of the article, the critical
revision of the article for important intellectual content and the final approval
5 % of Cochrane reviews added funding as a separate of the article. AS contributed to the design of the study, the collection and
domain and that 32 % incorporated funding into the assembly of data, the analysis and interpretation of the data, the critical revision
“other bias” function. Clearly, more work is needed on of the article for important intellectual content and the final approval of the
article. DL contributed to the design of the study, the collection and assembly
this issue. of data, the critical revision of the article for important intellectual content and
A general tension exists between bias in randomized tri- the final approval of the article. JS contributed to the conception of the study,
als as defined mechanistically in the tool, and as defined the critical revision of the article for important intellectual content and the final
approval of the article. IB contributed to the conception of the study, the critical
empirically based on results from meta-epidemiological revision of the article for important intellectual content and the final approval
studies. Several design features of randomized clinical of the article. JACS contributed to the conception of the study, the critical
Page 45 of 637
revision of the article for important intellectual content and the final approval September 2014. Available from http://www.riskofbias.info. Accessed 20 Jan
of the article. JH contributed to the conception of the study, the design of the 2015.
study, the critical revision of the article for important intellectual content and 13. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ,
the final approval of the article. AH contributed to the conception of the study, et al. Assessing the quality of reports of randomized clinical trials: is blinding
the design of the study, the analysis and interpretation of the data, the drafting necessary? Control Clin Trials. 1996;17(1):1–12.
of the article, the critical revision of the article for important intellectual content 14. The Physiotherapy Evidence Database (PEDro) Scale. Available from:
and the final approval of the article. http://www.pedro.org.au/english/downloads/pedro-scale/. Accessed 20
Jan 2015.
15. Armijo-Olivo S, Ospina M, da Costa BR, Egger M, Saltaji H, Fuentes J, et al.
Acknowledgements Poor reliability between Cochrane reviewers and blinded external reviewers
LJ would like to thank Allison E. Crank for her assistance in editing the when applying the Cochrane risk of bias tool in physical therapy trials. PLoS
manuscript. One. 2014;9(5):e96920.
The study received no funding or grant other than standard salary to the 16. Hopewell S, Boutron I, Altman DG, Ravaud P. Incorporation of assessments
data collectors (LJ, AS and DL) provided by The Nordic Cochrane Centre of risk of bias of primary studies in systematic reviews of randomised trials:
(Rigshospitalet, Copenhagen). The National Institute supports JS for Health a cross-sectional study. BMJ Open. 2013;3(8):e003342.
Research Collaboration for Leadership in Applied Health Research and Care 17. The Database of Abstracts of Reviews of Effects (DARE). Available from:
West (NIHR CLAHRC West). The views expressed are those of the authors and http://www.crd.york.ac.uk/CRDWeb/. Accessed 20 Jan 2015.
not necessarily those of the NHS, the NIHR or the Department of Health. 18. The Cochrane Handbook. Available from: http://handbook.cochrane.org/.
Accessed 20 Jan 2015.
Author details 19. Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJ, et al.
1
The Nordic Cochrane Centre, Rigshospitalet 7811, Blegdamsvej 9, 2100 Risk of bias in industry-funded oseltamivir trials: comparison of core reports
Copenhagen, Denmark. 2School of Social and Community Medicine, versus full clinical study reports. BMJ Open. 2014;4(9):e005253.
University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK. 3The 20. Brorson S, Hróbjartsson A. Training improves agreement among doctors
National Institute for Health Research Collaboration for Leadership in Applied using the Neer system for proximal humeral fractures in a systematic
Health Research and Care West (NIHR CLAHRC West) at University Hospitals review. J Clin Epidemiol. 2008;61(1):7–16.
Bristol NHS Foundation Trust, Bristol, UK. 4Methods of Therapeutic Evaluation 21. Brorson S, Bagger J, Sylvest A, Hróbjartsson A. Improved interobserver
of Chronic Diseases Team, Epidemiology and Biostatistics, Sorbonne Paris variation after training of doctors in the Neer system. A randomised trial.
Cité Research Centre, L’Institut National de la Santé et de la Recherche J Bone Joint Surg (Br). 2002;84(7):950–4.
Médicale, Unite Mixte de Recherche 1153, Paris, France. 5Research Unit for 22. Haahr MT, Hróbjartsson A. Who is blinded in randomized clinical trials? A
Evidence-Based Medicine, University of Southern Denmark, Odense, study of 200 trials and a survey of authors. Clin Trials Lond Engl. 2006;3(4):
Denmark. 360–5.
23. da Costa BR, Resta NM, Beckett B, Israel-Stahre N, Diaz A, Johnston BC, et al.
Received: 22 December 2015 Accepted: 27 April 2016 Effect of standardized training on the reliability of the Cochrane risk of bias
assessment tool: a study protocol. Syst Rev. 2014;3(1):144.
24. Bero L. Industry sponsorship and research outcome: a Cochrane review.
JAMA Intern Med. 2013;173(7):580–1.
References 25. Dechartres A, Trinquart L, Boutron I, Ravaud P. Influence of trial sample size
1. Hróbjartsson A, Boutron I, Turner L, Altman DG, Moher D. Assessing risk of on treatment effect estimates: meta-epidemiological study. BMJ. 2013;346:
bias in randomised clinical trials included in Cochrane Reviews: the why is f2304.
easy, the how is a challenge. Cochrane Database Syst Rev. 2013;4:ED000058. 26. Panagiotou OA, Contopoulos-Ioannidis DG, Ioannidis JPA. Comparative
2. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. effect sizes in randomised trials from less developed and more developed
The Cochrane Collaboration’s tool for assessing risk of bias in randomised countries: meta-epidemiological assessment. BMJ. 2013;346:f707.
trials. BMJ. 2011;343:d5928. 27. Dechartres A, Boutron I, Trinquart L, Charles P, Ravaud P. Single-center trials
3. Bero LA. Why the Cochrane risk of bias tool should include funding source show larger treatment effects than multicenter trials: evidence from a meta-
as a standard item. Cochrane Database Syst Rev. 2013;12:ED000075. epidemiologic study. Ann Intern Med. 2011;155(1):39–51.
4. Sterne JAC. Why the Cochrane risk of bias tool should not include funding 28. Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, et al. Stopping
source as a standard item. Cochrane Database Syst Rev. 2013;12:ED000076. randomized trials early for benefit and estimation of treatment effects:
5. Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. Industry sponsorship systematic review and meta-regression analysis. JAMA. 2010;303(12):1180–7.
and research outcome. Cochrane Database Syst Rev. 2012;12:MR000033.
6. Roseman M, Turner EH, Lexchin J, Coyne JC, Bero LA, Thombs BD. Reporting
of conflicts of interest from drug trials in Cochrane reviews: cross sectional
study. BMJ. 2012;345:e5155.
7. Goodman S, Dickersin K. Metabias: a challenge for comparative
effectiveness research. Ann Intern Med. 2011;155(1):61–2.
8. Savović J, Weeks L, Sterne JAC, Turner L, Altman DG, Moher D, et al.
Evaluation of the Cochrane Collaboration’s tool for assessing the risk of bias
in randomized trials: focus groups, online survey, proposed
recommendations and their implementation. Syst Rev. 2014;3:37.
9. Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, Submit your next manuscript to BioMed Central
et al. Testing the risk of bias tool showed low reliability between individual and we will help you at every step:
reviewers and across consensus assessments of reviewer pairs. J Clin
Epidemiol. 2013;66(9):973–81. • We accept pre-submission inquiries
10. Hartling L, Bond K, Vandermeer B, Seida J, Dryden DM, Rowe BH. Applying • Our selector tool helps you to find the most relevant journal
the risk of bias tool in a systematic review of combination long-acting beta-
• We provide round the clock customer support
agonists and inhaled corticosteroids for persistent asthma. PLoS One. 2011;
6(2):e17242. • Convenient online submission
11. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs Seida J, et al. • Thorough peer review
Risk of bias versus quality assessment of randomised controlled trials: cross
• Inclusion in PubMed and all major indexing services
sectional study. BMJ. 2009;339:b4012.
12. Sterne JAC, Higgins JPT, Reeves BC. On behalf of the development group • Maximum visibility for your research
for ACROBAT-NRSI. A Cochrane risk of bias assessment tool: for non-
randomized studies of interventions (ACROBAT-NRSI), Version 1.0.0, 24 Submit your manuscript at
www.biomedcentral.com/submit
Page 46 of 637
PRISMA 2009 Checklist
Reported on
Section/topic # Checklist item
page #
TITLE
Title 1 Identify the report as a systematic review, meta-analysis, or both. Not applicable
ABSTRACT
Structured summary 2 Provide a structured summary including, as applicable: background; objectives; 2
data sources; study eligibility criteria, participants, and interventions; study
appraisal and synthesis methods; results; limitations; conclusions and
implications of key findings; systematic review registration number.
INTRODUCTION
Rationale 3 Describe the rationale for the review in the context of what is already known. 3
Objectives 4 Provide an explicit statement of questions being addressed with reference to Not applicable
participants, interventions, comparisons, outcomes, and study design (PICOS).
METHODS
Protocol and 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web Protocol exists
registration address), and, if available, provide registration information including
registration number.
Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report 3 and 4
characteristics (e.g., years considered, language, publication status) used as
criteria for eligibility, giving rationale.
Information sources 7 Describe all information sources (e.g., databases with dates of coverage, Appendices
contact with study authors to identify additional studies) in the search and
date last searched.
Search 8 Present full electronic search strategy for at least one database, including any Appendices
limits used, such that it could be repeated.
Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in 3 and 4
systematic review, and, if applicable, included in the meta-analysis).
Data collection 10 Describe method of data extraction from reports (e.g., piloted forms, 3 and 4
process independently, in duplicate) and any processes for obtaining and confirming
data from investigators.
Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding 3 and 4
sources) and any assumptions and simplifications made.
Risk of bias in 12 Describe methods used for assessing risk of bias of individual studies (including Not applicable
individual studies specification of whether this was done at the study or outcome level), and how
this information is to be used in any data synthesis.
Summary measures 13 State the principal summary measures (e.g., risk ratio, difference in means). 4
Synthesis of results 14 Describe the methods of handling data and combining results of studies, if 4
done, including measures of consistency (e.g., I2) for each meta-analysis.
Page 1 of 2
Reported on
page #
Risk of bias across 15 Specify any assessment of risk of bias that may affect the cumulative evidence Not applicable
studies (e.g., publication bias, selective reporting within studies).
Page 47 of 637
Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, Not applicable
meta-regression), if done, indicating which were pre-specified.
RESULTS
Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the 4, 5 and 6
review, with reasons for exclusions at each stage, ideally with a flow diagram.
Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., Not applicable
study size, PICOS, follow-up period) and provide the citations.
Risk of bias within 19 Present data on risk of bias of each study and, if available, any outcome level Not applicable
studies assessment (see item 12).
Results of individual 20 For all outcomes considered (benefits or harms), present, for each study: (a) Not applicable
studies simple summary data for each intervention group (b) effect estimates and
confidence intervals, ideally with a forest plot.
Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals and Not applicable
measures of consistency.
Risk of bias across 22 Present results of any assessment of risk of bias across studies (see Item 15). Not applicable
studies
Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup Not applicable
analyses, meta-regression [see Item 16]).
DISCUSSION
Summary of 24 Summarize the main findings including the strength of evidence for each main Not applicable
evidence outcome; consider their relevance to key groups (e.g., healthcare providers,
users, and policy makers).
Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at Not applicable
review-level (e.g., incomplete retrieval of identified research, reporting bias).
Conclusions 26 Provide a general interpretation of the results in the context of other 10
evidence, and implications for future research.
FUNDING
Funding 27 Describe sources of funding for the systematic review and other support (e.g., 10
supply of data); role of funders for the systematic review.
From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS
Med 6(7): e1000097. doi:10.1371/journal.pmed1000097
For more information, visit: www.prisma-statement.org.
Page 2 of 2
Page 48 of 637
Appendices
Appendix 1: Review of published comments: Search strategy for identifying the comments on the Cochrane risk of
bias tool for randomized clinical trials
The search strategy was developed iteratively with keywords (and synonyms): risk (chance, likely*, probab*), bias
(qualit*, valid*, (systematic) error, risk of bias, RoB), tool (instrument, assess*, examin*, score, apprais*), Cochrane
(CDSR, CCRBT, ROBT, systematic, review, (meta-)analys*), editor* (comment*, letter* (to the editor) and
correspond*).
It was challenging to strike an appropriate balance between search sensitivity and specificity. Test search strings often
included what we considered too many or too few publications. The main search string was:
“(bias) AND (tool OR assess*) AND (Cochrane)”
To increase the sensitivity, we performed additional searches (with the identified keywords and synonyms in the most
beneficial combinations). We used the ”sorted by relevance”-function in PubMed and Google Scholar, and stopped
screening, evaluating and assessing publications when no “major/minor comments” (or “peripheral remarks”) had
been identified in a substantial amount of the sorted publications (>100 references). Furthermore, we retrospectively
searched additional comments from the most active commenters. Many were members of The Cochrane Bias
Methods Group so we searched additional comments from the remainder of the Group assuming they would be
interested in commenting on the Cochrane risk of bias tool for randomized clinical trials. Most of the assessed full-
texts did not comment on the tool (but were systematic reviews that used the tool or publications that
described/explained i.e. the basis of the tool). We decided that a full text assessment of approximately 1000
publications would constitute a fairly thorough search (we assessed 976 full-texts).
Page 49 of 637
Appendix 2: Analysis of use of the Cochrane risk of bias tool for randomized clinical trials: selection of Cochrane and
non-Cochrane reviews
1: We included 100 Cochrane reviews via PubMed with the following search string:
”The Cochrane database of systematic reviews”[Journal] AND (systematic[sb] AND (”0001/01/01”[PDAT]: ”2014/12/31”[PDAT]))
15 Cochrane reviews were excluded to reach 100 Cochrane reviews: 12 Cochrane reviews did not include any
randomized clinical trials in their analysis, two had been withdrawn and one did not involve a clinical intervention.
About half of the included Cochrane reviews were 1st edition Cochrane reviews (45/100, 45%) and about half were
updates of previous published Cochrane reviews (55/100, 55%). Most updates were 2nd edition Cochrane reviews
(33/55, 60%), some were 3rd edition (16/55, 29%) and few were 4th (5/55, 9%) and 5th (1/55, 2%) editions.
2: We included 100 non-Cochrane reviews from PubMed with the following search string:
(((systematic[sb]) AND (”0001/01/01”[PDAT]: ”2014/12/31”[PDAT])) AND (systematic[Title/Abstract] OR systematical[Title/Abstract] OR

systematically[Title/Abstract])) NOT ”The Cochrane database of systematic reviews”[Journal]
65 non-Cochrane reviews were excluded to reach 100 non-Cochrane reviews: 57 non-Cochrane reviews did not
include any randomized clinical trials and eight non-Cochrane reviews were not retrievable through The Royal Danish
Library. All of the included non-Cochrane reviews were 1st edition reviews (100%) and none were updates of previous
published reviews or published as Cochrane reviews.
Page 50 of 637
Appendix 3: Comments on the Cochrane risk of bias tool for randomized clinical trials
Table I: Major comments on the Cochrane risk of bias tool for randomized clinical trials
Reference First author Key findings from comments Selected commentaries from references Wordcount
1 Armijo-Olivo Strengths: “Can we trust risk of bias results reported in Cochrane reviews? Can we trust 259
… assessments using the risk of bias tool?”...”The large number of trials classified as
high or unclear risk of bias casts doubts about the discrimination power of the
Challenges: risk of bias tool.”...”In addition, the items included in the risk of bias may be
Many “high/unclear risk” trials insufficient to represent the construct of interest: “Risk of bias”. Other items not
Insufficient tool domains considered in this tool may need to be added to provide a more comprehensive
Poor risk of bias agreement evaluation.”...”Empirical evidence supports the evaluation of randomization,
allocation concealment and blinding of clinical trials, all of which are included in
Suggestions: the risk of bias tool. While there is insufficient evidence to support other domains
Need for improved guidelines and being included, other methodological factors could be important for evaluating
empirical evidence supporting risk of bias and could be considered for inclusion in the risk of bias tool after
additional domains careful empirical evidence testing.”...”Poor agreement [using the risk of bias tool]
Perform risk of bias for (main) was not only demonstrated at the trial level but also at the meta-analysis
outcomes level.”...”Improved guidelines to apply the risk of bias tool and revisions to the
tool for different health areas are needed. In addition, empirical evidence
supporting additional items for the risk of bias tool needs to be
developed.”…“The majority of Cochrane reviews analyzed did not clearly specify
the outcome used for risk of bias assessments. This directly reduces
reproducibility of risk of bias assessment for outcome-dependent domains of the
tool. Cochrane reviewers should report risk of bias assessments separately for
each outcome analyzed, or at least for the main outcomes of the review.”
2 Bero Strengths: “The current Cochrane risk of bias tool is insufficient to assess bias related to 128
… study funding sources.”...”Those opposed to adding funding source to the
Cochrane risk of bias tool argue that all mechanisms of bias can be identified,
Challenges: perfectly measured, and incorporated quantitatively in the results of meta-
The tool does not include funding analysis.”...”The Cochrane risk of bias tool should include funding source as a
standard item because: 1. Funding source fits the definition of bias, 2. There is
Suggestions: empirically-based evidence of bias related to funding source, 3. The observed
Include funding in the tool bias related to funding source cannot be captured by the risk of bias criteria
currently assessed with the risk of bias tool, 4. Risks of bias are not mutually
exclusive, 5. Bias may be related to funding source even when all studies are
industry-funded.”
3 Boutron Strengths: “[The tool] aims at being completely transparent, with a separation of the facts 234
Aim of transparency and reviewers’ judgments. This aim is particularly important because reviewers,
Based on empirical evidence editors, and readers can challenge the author on the judgment.”...”[The tool]
An important step forward in risk of domains were chosen on the basis of empirical evidence demonstrating potential
bias assessment for bias or exaggeration of treatment effects [5], [6], [7], [8] and [9].”...”This tool
has been an important step forward in the assessment of the risk of bias in
Challenges: systematic reviews and meta-analyses. Assessing the risk of bias with the risk of
Reliabilty: risk of bias assessment is bias tool and evaluating the quality of studies included in systematic reviews
intertwined with quality of raizes some challenges mainly related to reliability. The assessment of the risk of
reporting bias of a trial is intertwined with the quality of reporting.”...”Akl et al provide a
Interrater variability is high useful tool to improve the assessment of the blinding status. They validated their
tool by contacting authors of the published trials. Of course, this is only a
Suggestions: preliminary step in evaluating the risk of bias because it cannot help determine
Integrate Akl’s blinding assessment whether blinding was efficient and evaluate the risk of bias when blinding is not
approach in the tool implemented.”...”Furthermore, the validation relying on contacting authors can
be debated [20]. However, the authors’ approach is interesting and could be
used for other items of the risk of bias tool. This assessment could be helpful to
improve interrater agreement and transparency when assessing the risk of bias
with the risk of bias tool.”
4 De Bruin Strengths: “This paucity of research on risk of bias in HBC [health behaviour change] trial is 337
… in turn reflected in widely used instruments for assessing the risk of bias, such as
the Cochrane risk-of-bias tool (Higgins et al., 2011), which seems to be mostly
Challenges: based on evidence from non-behavioural trials.”...”...the question–behaviour
The tool seems to be mostly based effect [...] is one of these sources of bias that seems specific to the evaluation of
on non-behavioural trials educational, psychological or behavioural (i.e. non-pharmacological)
Risk of bias assessments are interventions, and may not be adequately captured by commonly used risk-of-
insufficiently included in analysis bias tools.”...”Given that a previous meta-analysis of HBC trials in which one of
even when done the authors was involved (Dr. Viechtbauer, in de Bruin et al., 2010) revealed that
attrition may not be random (i.e. patients doing better were more likely to
Suggestions: dropout), differential attrition might indeed be an important source of bias in
Page 51 of 637
Evaluate the success of non- HBC trials.”...”…systematic reviewers could consider adapting the risk-of-bias tool
conventional blinding measures to the literature under review: some common measures like blinding participants
to group assignment might not be possible, but alternative or additional
measures might be taken to reduce the risk of bias in these trials and it may be
worthwhile considering (or evaluating the success of) these.”…“Based on a
survey of 200 meta-analyses, they arrive at the conclusion that many do assess
methodological quality, but very few incorporate them in their analyses. Johnson
and colleagues were particularly interested in whether methodological quality
was used in moderator analyses (‘an interactive approach’), e.g. when examining
whether a particular behaviour change technique or modality of intervention
delivery explains heterogeneity in effect sizes. They find that this is rarely
reported by the meta-analyses reviewed, but when it is reported the moderator
results remained significant in higher quality studies, or were present among
higher but not lower quality studies. Johnson and colleagues also make the
important argument that instead of discarding trials that score lower on certain
methodological quality criteria (i.e. assuming that this affects the evidence-base),
it is much more informative to include all trials in the meta-analyses and actually
establish whether and how methodological quality influences trial results.”
5 Hartling Strengths: “Low agreement between reviewers suggests the need for more specific 637
The tool is based on methods – not guidance regarding interpretation and application of the Risk of Bias tool (RoB) or
reporting possibly re-phrasing of items for clarity.”...”Examination of study-level variables
The tool is based on empirical and their association with inter-rater agreement identifies areas that require
evidence specific guidance in applying the Rob [Risk of Bias] tool. For example, nature of
the outcome (objective vs. subjective), study design (parallel vs. other), and trial
Challenges: hypothesis (efficacy/superiority vs. other).”...”Low agreement between pairs of
Low agreement between reviewers reviewers indicates the potential for inconsistent application and interpretation
Many “high/unclear risk” trials – of the RoB [Risk of Bias] tool across different groups and systematic
The tool might not be sensitive reviews.”...”Post hoc analyses showed that disagreement most often arose from
enough and the “unclear” category interpretation of the tool rather than discrepancies in the information that was
might become the default extracted from studies.”...”The majority of trials in the sample were assessed as
No differences in effect sizes across high or unclear risk of bias for many domains, likely due to inadequate reporting
risk of bias categories (high, unclear at the study level. This raizes concerns about the ability of the RoB [Risk of Bias]
and low) tool to detect differences across trials that may relate to biases in estimates of
Non-parallel design and non- treatment effects.”...”No statistically significant differences were found in effect
pharmacological intervention sizes (ES) across high, unclear and low risk of bias categories.”...”the low
creates ambiguity in risk of bias agreement raizes concerns and points to the need for clear and detailed guidance
assessment in terms of applying the RoB [Risk of Bias] tool.”...”...agreement for some
Consensus-rating is better than domains may be better in classic parallel trials of pharmacological interventions,
two-reviewer rating whereas trials with different design features [...] and those examining
nonpharmacological interventions appear to create more ambiguity for risk of
Suggestions: bias assessments.”...”...the consensus rating is a more meaningful measure of
Need for renewed guidelines agreement (as opposed to reliability between two reviewers), as these ratings are
(regarding interpretation and the ones reported in systematic reviews.”...”Of particular note is that 99 percent
application), decision rules, of this sample had overall risk of bias assessments as high or unclear. This is
transparency and additional similar to three of the four other samples that had more than 90 percent
testing. assessed as high or unclear risk of bias overall.”...”If the vast majority of trials are
Make a “living database” with assessed as high or unclear risk of bias, the tool may not be sufficiently sensitive
examples and consensus from to differences in methodology that might explain variation in treatment effect
experts estimates across studies, or study methodology as a potential explanation for
heterogeneity in meta-analyses.”...”While the focus of the RoB [Risk of Bias] tool
is intended to be on methods rather than reporting, reviewers regularly indicate
that they rely on the trial reporting to make their assessments.”...”We do not
intend for our results to suggest that reviewers abandon existing tools for other
tools unless these have shown greater reliability and validity. Rather, our results
underscore the need for reviewers and review teams to be aware of the
limitations of existing tools and to be transparent in the process of risk of
bias/quality assessment. Detailed guidelines, decision rules, and transparency are
needed so that readers and end-users of systematic reviews can see how the
tools were applied. Further, pilot testing and development of review-specific
guidelines and decision rules should be mandatory and reported in
detail.”...”There is a need for more detailed guidelines to apply both the RoB
[Risk of Bias] tool and the NOS, as well as revisions to the tools to enhance clarity.
Additional testing should occur after further revisions to the tool and when
expanded guidelines are available.”...”A final caveat to note is that the RoB [Risk
of Bias] tool has undergone some revisions since we initiated the study. These
are detailed in the most recent version of the Cochrane Handbook11 but were
not incorporated into our research. The changes affected primarily the blinding
and the “other sources of bias” domains. This does not impact the general
Page 52 of 637
findings from our research; however, further testing with the modified tool is
warranted.”…”The developers of the tool aimed to distinguish between actual
methods of conducting the trials vs. reporting. Furthermore, the choice of
components for inclusion in the tool was based on the empirical evidence
demonstrating their association with effect estimates.”
6 Hróbjartsson Strengths: “The risk of bias tool provides a standardized approach, based on items selected 344
The tool has a standardized on both theoretical and empirical grounds, and following broad consultations
approach, based on theoretical and with clinical research methodologists.”...”The risk of bias tool is a comparatively
empirical grounds recent development that still likely needs refinement. The modest inter-rater
agreement rates[5] will hopefully be improved by modifications to the questions
Challenges: and enhanced training courses. However, authors of Cochrane Reviews tend to
Cochrane review authors are often be reluctant in designating an overall risk of bias for each trial or outcome and
reluctant regarding overall risk of also reluctant to incorporate the risk of bias assessment in analyses and
bias assessment and incorporation conclusions.”...”risk of bias assessment has more fundamental challenges.”...”
of risk of bias in analyses’ The risk of confounding in [meta-epidemiologic studie] comparisons is
Risk of confounded risk of bias pronounced, as compared trials may differ for other reasons.”...”The assessment
evidence base of risk of bias is to a large extent based on common sense and theoretical
Based on observational studies considerations, with an empirical basis of observational studies with a
Unclear if funding, centre-status, considerable risk of confounding.”...”the empirical evidence underlying the
early stopping and country are assessment of risk of bias in trials – an assessment necessary for ensuring that
unique biases biased trials do not lead to biased systematic reviews – is based on observational
Unclear if the tool addresses studies.”...”An increasing number of meta-epidemiological studies report
funding adequately associations between a trial characteristic and exaggerated treatment effects:
funding status,[7] number of centres participating in a trial,[8] early stopping of a
Suggestions: trial,[9] and developing country status.[10] For many of these characteristics it is
… unclear whether they represent a unique bias, confounding, publication bias,
spurious findings, or a combination of these and/or other unknown factors. It is,
nonetheless, helpful to be aware of such associations, sometimes called meta-
bias.[11]”...”Funding status is a major concern in randomized trials. The
exaggerated effects reported for industry trials [7] may to some extent be
explained as a result of publication bias or other characteristics included in the
risk of bias tool, for example selective outcome reporting. However, companies
that stand to gain financially by a positive result have substantial conflicts of
interest when they control the planning, funding, conduct, and reporting of a
trial. It is not clear that the risk of bias tool in its present version addresses this
problem adequately.”
7 Ivers Strengths: “The risk of bias tool does not capture all sources of methodological bias and 265
… poor reporting interferes with the assessment of many domains.”...”…we found
that nearly half of RCTs [randomized clinical trials] focusing on diabetes had at
Challenges: least one domain at high risk of bias…[and]…that poor follow-up, baseline
The tool does not capture all imbalances and blinding were the most common sources of high risk of
sources of methodological bias bias.”...”…sensitivity analyses may be used to explore the risk of bias related to
Poor reporting interferes with loss of follow-up, and risk of baseline imbalances in [quality improvement] trials
assessment of risk of bias domains may be reduced through restricted randomisation techniques, especially when
One domain with “high” risk leads trials are cluster-randomized with relatively few clusters.”...”...it is possible that
to overall “high” risk studies at low risk of bias have important flaws with respect to methodology
The tool only catches risk of bias and/or reporting (and vice versa), and it is possible that using other scales to
and not other methodological flaws assess study quality could have led to different results.14 While the overall risk of
bias assessment using the Cochrane Risk of Bias tool has been shown to
Suggestions: differentiate effect sizes (ie, higher risk of bias studies usually have larger effect
Use sensitivity analyses for baseline sizes),10 studies at high risk of bias may still offer valuable
and follow up knowledge...”...”Furthermore, we acknowledge that assigning trials with high risk
of bias in a single domain a status of high risk of bias overall may be
arguable.”…”Some important components of methodological quality do not
relate to bias (eg, reporting of a sample size calculation). Thus, it is possible that
studies at low risk of bias have important flaws with respect to methodology
and/or reporting (and vice versa), and it is possible that using other scales to
assess study quality could have led to different results.”
8 Jefferson Strengths: “The current Cochrane risk of bias tool is not adequate for the task as it does not 533
Facilitates the process of critical reliably identify all types of important biases, and nor does it organize and check
assessment of a clinical trial the coherence of large amounts of information.”...”...each [the tool domain] may
have more than one source of bias application...”...”In addition, the current
Challenges: Cochrane risk of bias tool was introduced after the production of our review of
The tool does not identify all published articles, making the comparison, had we had the data to undertake it,
important study design bias- more difficult to interpret and possibly unfair.”...”We found the Cochrane risk of
variables, nor organizes data or bias tool to be difficult to apply to clinical study reports. We think this is not
checks data-coherence because the tool was constructed to assess journal publications but, as with all
Each domain may have more than list-like instruments, its use lends itself to a checklist approach (in which each
Page 53 of 637
one source of bias application design item is sought and, if found, eliminated from the bias equation rather than
The tool is difficult to apply due to with thought and consideration).”...”Many of the variables we found to be
its checklist approach and relative important when assessing the trial (eg, date of trial protocol, date of unblinding,
subjectivity date of participant enrolment) are simply not captured in the risk of bias tool
Risk of bias assessment is time when used in a routine way or to review publications. We were also often unsure
consuming how to judge the risk of bias when bias itself can actually or potentially be
measured with reviewers’ access to full clinical study reports and individual
Suggestions: participant data. If, for example, the original trial protocol is available, one can
Need for further tool development judge whether reporting bias occurred. Reviewers need not guess at bias (ie,
The tool should be developed so make a judgement of ‘risk’) but can judge bias directly. However, even with
that Cochrane reviews may rely on individual participant data, some forms of bias, such as attrition bias, may still be
clinical study reports as the basic difficult to quantify, and one can only judge the risk (ie, potential) of
unit of analysis bias.”...”While the judgements of ‘low’ or ‘high’ risk of bias may imply certainty,
particularly when based on the reading of a full clinical study report, we found
ourselves often in lengthy debate and discussion over the proper level of risk of
bias before arriving at a consensus. We found the risk of bias judgements
themselves to carry a high level of subjectivity, in which different judgements can
be justified in different ways. The real strength of the risk of bias tool appears not
to be in the final judgements it enables, but rather in the process it helps
facilitate: critical assessment of a clinical trial.”…”Reviewing the complete clinical
study reports and our assessment of bias was very time consuming.”...”…the
current Cochrane risk of bias tool does not sufficiently identify possible faults
with study design, and nor does it help to organize and check the coherence of
large amounts of information that are found in clinical study reports. Our
experience suggests that more detailed extraction sheets that prompt reviewers
to consider additional aspects of study may be needed. Until a more appropriate
guide is developed, we offer our custom extraction sheets to Cochrane reviewers
and others interested in assessing risk of bias using clinical study reports and
encourage further development.”…”As evidence of reporting bias in industry trial
publication mounts, 8 16–21 we believe Cochrane reviews should increasingly
rely on clinical study reports as the basic unit of analysis.”
9 Katikireddi Strengths: “These [/the tools] judgements should be outcome-specific and ideally informed 456
… by empirical evidence of bias, the likely direction of bias and the likely magnitude
of bias. However, this may be difficult to achieve in practice, given the
Challenges: acknowledged evidence gap in the relative importance of different domains of
Risk of bias should be outcome- bias.4”...”Incorporation of risk of bias assessments into synthesis is crucial to
specific, informed by empirical ensure that SR conclusions are based on the best available evidence. Failure has
evidence and the direction and serious implications for evidence-informed policy and practice.”...”Assessing risk
magnitude of bias of bias using scoring systems continues, despite the Cochrane Collaboration's
Reviewers struggle to understand recommendations to avoid their use.”...”Approaches to incorporating risk of bias
and operationalize current assessments into the findings of SRs [systematic reviews] are (arguably
guidance on how to conduct and appropriately) varied but frequently lack transparency. Lastly, some SRs that
incorporate risk of bias assessment investigate multiple outcomes continue to ignore the potential for risk of bias to
into synthesis of data differ across outcomes.”...”...many Cochrane SRs [systematic reviews] selectively
Some review authors do not followed some but not all of the recently published guidance.”...”Half of the
consider risk of bias for individual reviews published in the sample from the Cochrane library and leading general
outcomes medical journals did not incorporate findings of critical appraisal into their
review.”...”Our findings not only show that risk of bias summary scores are still
Suggestions: frequently used but also suggest that there is confusion about how best to
Need for research on bias incorporate critical appraisals into SR [systematic reviews] findings.”...”reviewers
importance and impact, and for should clearly report findings from the most robust studies, either as a sensitivity
optimized guidelines analysis or in the primary analysis.”...”Given that the study design and type of
Risk of bias assessment should be intervention are closely related, reviewers should ideally go further and consider
incorporated in study synthesis and whether an ‘intervention selection bias’ is inadvertently introduced by focusing
conclusions only on higher quality studies.”...”Early tools for critical appraisal were appealing
because they were simple to use and resulted in a score which allowed ranking of
studies by risk of bias, facilitating incorporation into the synthesis process.17 ,19
,35 Unfortunately, this simplicity came to be regarded as a source of weakness,
as well as a strength, and these tools have been replaced by more complex
guidance intended to address some of their limitations.4 ,21 ,36 We recognize
the need for this development but take the view, supported we believe by this
study, that reviewers are struggling to understand and/or operationalize current
guidance on how to conduct and incorporate critical appraisal within synthesis.
Further research is required to establish the relative importance of different
forms of bias and their likely impact6, 37 and also to clarify how critical appraisals
should be incorporated into SR findings 38, 39. However, to ensure that SRs
[systematic reviews] really do direct decision-makers to the best available
evidence, there is an urgent need to make guidance more understandable to the
Page 54 of 637
diverse reviewers involved.”…”Lastly, some SRs [systematic reviews] that
investigate multiple outcomes continue to ignore the potential for risk of bias to
differ across outcomes.”
10 Morissette Strengths: “...if blinded assessments are associated with less bias, then this approach needs 168
The risk of bias terminology to be part of the systematic review process.”...”Potential confounding factors
“high/low” risk that we were unable to examine included the assessors’ level of experience with
The tool focuses on bias instead of risk of bias assessments.”...”The risk of bias deals with the extent to which a
study quality and is transparent in study's results are believed to be true and attempts to focus on how the study
its assessments was actually done.”...”...the risk of bias terminology overcomes rating a study as
being 'low quality' when it may not be feasible (or appropriate) to conduct a
Challenges: particular methodological component (e.g. blinding).”...”...we encourage further
Might take longer than other tools research in this area and recommend using all of the important components of
the Cochrane 'Risk of bias' tool.”…”The Cochrane 'Risk of bias' tool differs from
Suggestions: other quality appraisal tools because it questions the degree to which a study’s
Need for further research on the results should be believed, is based on other documents above and beyond the
tool i.e. if blind or non-blind risk of study report (e.g. study protocol), and the risk of bias appraisal decisions are
bias assessment is best recorded to ensure transparency (Hartling 2009).”
11 Moustgaard Strengths: “The classification of outcomes as subjective or objective is central to formal and 399
… informal assessments of the risk of bias in individual trials.”...”The terms
“subjective outcomes” and “objective outcomes” occur repeatedly in the
Challenges: Cochrane Handbook in connection with assessment of risk of bias in clinical trials,
Lack of explicit characterization and but we neither found explicit characterization of the terms (The Cochrane
classification of subjective and Handbook was included in the review because of an explicit description of the
objective outcomes for risk of bias category “patient reported outcomes”) nor is “subjective outcome” or “objective
assessment – also in relation to co- outcome” defined in the online Cochrane Glossary.”...”Many of the passages in
interventions the Cochrane Handbook that include the terms “subjective” or “objective”
Central methodological terms are outcomes concern the risk of detection bias, that is, systematic differences
understood differently between groups in how outcomes are determined.”...” No characterization of
High interrater variability subjective vs. objective outcomes relevant to risk of performance bias is given
explicitly in the Cochrane Handbook nor did we find it in the methodological
Suggestions: articles or the clinical trial reports we reviewed.”...”all outcomes may potentially
The Cochrane Handbook should be affected by cointerventions and other differences in basic care and the
explain what is meant by objective distinction between subjective and objective outcomes thus seems less relevant
and subjective outcomes when discussing risk of bias because of cointerventions. Still, these reflections are
speculative, and in the context of performance bias, the meaning of “subjective”
and “objective” outcomes remains somewhat obscure.”...”The Cochrane risk of
bias tool has been found to have a moderate interobserver agreement rate [21].
Interobserver agreement will hopefully improve once assessors become more
familiar with the tool, are offered better training, and the tool is refined. One
likely contributing factor to the modest agreement rates is the different
interpretations of what is meant by subjective and objective outcomes.”...”It
should also be noted that some outcomes that have generally been considered
“objective” may in fact leave considerable room for judgment by the assessor, for
instance, blood pressure, ultrasonic measurements, radiographic outcomes, and
so on.”...”Our study indicates that central methodological terms may be
understood differently by different readers, and we suggest that authors of
clinical trial reports explicitly explain the intended meaning of the terms
“subjective” and “objective” outcomes (as well as other terms that may be
considered ambiguous). Journal editors might contribute to greater clarity by
requiring that authors provide such explicit explanations.”…”We also suggest that
adding an explicit clarification of the intended meaning of ‘‘subjective’’ and
‘‘objective’’ outcomes in future versions of the Cochrane Handbook might further
strengthen its important role in guiding review authors.”
12 Roseman Strengths: “The 2008 edition of the [Cochrane] handbook suggested that potential bias 505
The Cochrane Handbook mentions related to the influence of trial sponsors could be considered in an optional
that funding should be collected “other sources of bias” domain of the risk of bias tool.37 In contrast, the 2011
edition specifies that this information should not be incorporated in the risk of
Challenges: bias assessment.38 Both versions of the handbook mention that review authors
Inconsistent and non-specific may consider extracting data on trial author-industry financial ties but do not
recommendations for including specify if and where this information should be reported.33, 34.”...”Given the
funding in the tool well documented influence of industry funding of drug trials on their conduct,
The Cochrane Collaboration should interpretation, and reporting,1 4 5 6 7 8 9 10 12 the Cochrane handbook and the
reconsider its position that trial PRISMA statement should be updated to require authors of systematic reviews
funding and trial author-industry and meta-analyses to report the funding sources of all included trials or to report
financial ties not be included in the that trial funding sources were not disclosed.”...”...if the funding source of
risk of bias assessment included trials is only partially reported, readers might assume that the funding
Most Cochrane reviews in 2010 did sources of other trials were available but not recorded, leaving them unsure as to
Page 55 of 637
not report funding how to interpret potential bias related to the funding sources of those
trials.”...”Beyond study funding, consumers of research consider conflicts of
Suggestions: interest from trial author-industry financial ties and employment as relevant to
Cochrane reviews should report the appraising the likelihood of bias in trials, 45 46 47.”...”...we recommend that the
funding sources of all included Cochrane Collaboration reconsider its position that trial funding and trial author-
trials industry financial ties not be included in the risk of bias assessment.”...”One
Include conflicts of interest in the concern might be that including conflicts of interest from included trials in the
risk of bias assessment risk of bias assessment could result in “double counting” of potential sources of
bias. However, ratings in the risk of bias table are not summed to a single score,
and inclusion of risk of bias from conflicts of interest could reflect mechanisms
through which industry involvement can influence study outcomes6 that are not
fully captured by the current domains of the risk of bias tool.”...”Inclusion of
conflicts of interest from included trials in the risk of bias assessment would
encourage a transparent assessment of whether industry funded trials and
independently conducted trials reach similar conclusions. It would also make it
explicit when an entire area of research has been funded by industry and would
benefit from outside scrutiny. Coding trial funding sources can be complex, and it
may not always be clear to what degree different funders played a part in a given
study.”...”A reasonably simple system would be to code trial funding as
pharmaceutical industry, non-industry (for example, public granting agency,
private not for profit granting agency), combined pharmaceutical industry and
non-industry, non-industry with study drug supplied by pharmaceutical industry,
no study funding, or not reported.22.”...”Most Cochrane reviews of drug trials
published in 2010 did not report information on trial funding sources or trial
author-industry financial ties, including employment, from included
trials...”...”...Cochrane and the PRISMA statement should require reviews to
report conflicts of interest from included trials in a way that is consistent across
reviews, and Cochrane should include this information as part of risk of bias
assessment.”
13 Savović Strengths: “The domains reported to be the most difficult to assess were [...] incomplete 563
The tool has a standardized outcome data and selective reporting of outcomes. There was wide variation in
approach, ensures transparency, is how review authors had approached the ‘other bias’ domain, with a lack of
flexible and is a platform for critical clarity over what additional items should be considered here. Some of the items
thinking that authors have included (such as sample size calculations and funding source)
are explicitly discouraged in the Cochrane Handbook guidance. While there is
Challenges: evidence that some factors are empirically associated with effect estimates, such
Difficult to assess some risk of bias as single versus multicentre design, early stopping of trials and funding source
domains - especially incomplete [14-16], the extent to which these should be considered alongside the main bias
outcome data and selective domains is still a topic of debate. The evaluation highlighted a need for more and
reporting better training and guidance materials, such as algorithms or similar structured
Wide approach to the “other bias” guidance for reaching domain-level judgements, as well as guidance on how to
domain incorporate risk of bias assessments into meta-analyses and review conclusions.
Increased workload and complexity Recommendations for changes or further developments were made based on
compared to past practice identified needs and many have already been incorporated into the new edition
of the Cochrane Handbook, while other developments are underway. As
Suggestions: suggested by evaluation participants, an online bank of worked examples for risk
Need for better training and of bias assessments will be incorporated into future versions of the Cochrane
guidance materials for judgements Handbook or made available online.”...”…the main purpose of this evaluation
and on how to incorporate risk of was to identify potential problems with the risk of bias tool that can be rectified,
bias assessments into Cochrane and we suspect that users who encountered problems are more likely to have
reviews responded.”...”…it is possible that the number of people incorrectly applying
Graphical representation should be these concepts may be higher as authors may be unaware of their
for individual outcomes misunderstandings.”...”It is important that guidance and training materials
Need for more research into continue to be developed for all aspects of the tool.”...”...44% or more of
empirical foundation of domains respondents had difficulty with assessing each of the individual risk of bias
domains.”...”…clearer guidance, ideally based on empirical evidence, is needed
on how to deal with studies at high risk of bias in meta-analyses, other syntheses
of evidence across studies and drawing conclusions.”…”Specific benefits
described included: having a standardized approach to bias assessments; the
transparency provided by requesting quotes; the flexibility of the tool; […]
providing a good framework for consideration of the risk of bias; and providing a
platform to encourage critical thinking.”…”The main drawbacks described which
were also addressed in the survey, included: the increased workload and
complexity as compared with past practice.”…”For example, several participants
raized the issue that risk of bias assessments present a particular problem when
updating systematic reviews. Adopting the new tool in an updated review
requires review authors to reassess the risk of bias of studies included in the
original review, which they were often unwilling to do, and Cochrane Review
Page 56 of 637
Groups were not resourced to do this on behalf of authors.”…” Participants also
suggested that graphical displays of risk of bias assessments across studies should
be prepared separately for individual outcomes measured in the review rather
than at the study level, as individual outcomes can be judged to be at higher or
lower risk of bias using the tool.”…“More empirical evidence is needed to further
inform considerations of what methodological aspects are most important in
assessing risk of bias. There is a particular need for assessment of the influence of
participant attrition on effect estimates, and on separate contributions to bias
from blinding of patients and caregivers versus blinding of outcome assessors.”
14 Sterne Strengths: “Little evidence that: – Trial methods are more likely to be flawed if a trial is 182
The tool does not include funding industryfunded. – Fraud is more likely if a trial is industry-funded.”...”The
problems are (mainly) with selective reporting of outcomes, non-reporting of
Challenges: whole studies, and choice of comparator.”...”The current risk of bias tool does
The tool does not work well for not work well for assessment of selective reporting.”...”The fight to access all trial
assessment of selective reporting data is of fundamental importance to the Collaboration and to evidence-based
health care, but the Cochrane risk of bias tool should not include funding source
Suggestions: as a standard item.”…”Transparent and structured assessments of conflict of
Transparent inclusion of “conflict of interest in included trials should be a routine component of Cochrane systematic
interest” reviews. Such assessments relate mainly to the context in which the review
results should be interpreted. It is vital to know whether most of the information
comes from a company with a commercial interest in the intervention of interest,
but this is not in itself a reason to dismiss the accumulated evidence. It may
nonetheless be a reason to search particularly hard for unreported or selectively
reported evidence and to carefully scrutinize the chosen comparators.”
15 Vale Strengths: “The Cochrane Handbook4 states that because the ability to measure the true 251
… bias (or even the true risk of bias) is limited, then the possibility to validate a tool
to assess that risk is also limited. Nevertheless, authors of Cochrane systematic
Challenges: reviews are required to use the Cochrane risk of bias tool.”...”...our study has
The tools ability to measure the gone some way towards validating the Cochrane risk of bias tool, since access to
true bias is limited the additional information and data enables us to get closer to the true risk of
Assessing risk of bias is difficult for bias of individual studies.”...”Assessing risk of bias was particularly difficult for
the more subjective domains the more subjective domains [...] discrepancies were more common for attrition
Many “high/unclear risk” trials bias...”...”...we applied the risk of bias assessment to overall survival—an
Lack of risk of bias-guidance for objective outcome that is commonly well reported—rather than considering all
trials with inadequate information possible outcomes as is recommended. Our results could have therefore
The tool relies on reporting represented an optimistic view of the reliability of the risk of bias assessments
using published information alone, particularly in relation to incomplete outcome
Suggestions: data and selective outcome reporting.”...”…even with additional information,
… around a third of the included studies were classified as unclear risk of bias.
Clearly, forms purposely designed to collect specific information would help
reviewers reach appropriate judgments regarding risk of bias, in particular for
those trials with inadequate information published.”…”The vast majority of
Cochrane (and other) systematic reviews is based on information extracted from
the publications of eligible studies. Therefore, most risk of bias assessments are
similarly based on trial publications. However, trial quality is not necessarily well
represented in publications.”
Page 57 of 637
Table II: Minor comments on the Cochrane risk of bias tool for randomized clinical trials
Reference First author Key findings from comments Selected commentaries from references Wordcount
16 Abou-Setta Strengths: ”The domain most often assessed as 'low' risk was 42
… 'incomplete outcome data'; 'blinding' was most often
assessed as 'high' risk.”…”…97% of trials were assessed as
Challenges: 'unclear' or 'high' risk which raizes questions regarding the
Many “high/unclear risk” trials sensitivity/specificity of the risk of bias tool.”
Sensitivity/specificity of the tool
Suggestions:
…
17 Akl Strengths: ”The Cochrane risk of bias tool requires quoting what was 88
… reported to have happened in the study and then assigning a
judgment relating to the risk of bias for each risk of bias
Challenges: item.”...”Although the Cochrane handbook suggests
The Cochrane handbook does not provide supplementing an ambiguous quote with either a “probably
specific guidance on how to make “blinding” done” or a “probably not done” statement, it does not
judgments Due to poor reporting, many provide specific guidance on how to make this
unclear risk judgment.”...”We invite the Cochrane Collaboration [...] to
consider using our structured [blinding] approach.”...”...the
Suggestions: use of “probably yes” and “probably no” can enhance the
The use of “probably yes” and “probably no” assessment of blinding...”
can enhance the assessment of blinding
18 Anette- Strengths: ”To estimate the methodological quality of studies 81
Blümle … instruments such as the Cochrane Risk of Bias tool were
developed. But it can only be appraized what is
Challenges: reported.”...”The evaluation tool that the Cochrane
Risk of bias assessment is complicated and Collaboration recommends is neither a scale nor a checklist. It
often based on incomplete reporting is an instrument by which the risk of bias of individual
The tool is very difficult to validate methodological aspects can be evaluated. Since it is
impossible to determine the actual risk of bias in a study, it is
Suggestions: very difficult to validate an assessment tool.”
…
19 Armijo-Olivo Strengths: ”Because The tool is designed only for evaluation of RCTs, the 331
Appropriate that the tool comprehensively overall quality grades are weighted heavily on the
(2nd included evaluates randomization and blinding judgements assigned to the two randomization domains
Page 58 of 637
publication (sequence generation and allocation concealment), as well as
by this Challenges: the domain relating to blinding. Given the crucial importance
author) The tool might not be able to adequately of these three elements to the risk of bias among RCTs [4], it
discriminate the comparative quality of seems appropriate that The tool comprehensively evaluates
included studies randomization and blinding, and that the scores of these
Need for futher validation of the tool domains constitute approximately 50 % of the final grade
The information needed in the tool is often assigned by this tool.”...”The EPHPP [Effective Public Health
poorly reported Practice Project] grade takes into account the validity and
reliability of data collection methods. This item is not directly
Suggestions assessed by The tool but could be considered in the context
“Validity and reliability of data collection of other bias. Similarly, the presence of confounders is
methods” and “confounders” could be specifically evaluated by the EPHPP tool, but could be
considered in the tools “other bias” considered under the ‘other sources of bias’ domain of The
Review groups should customize the tool for tool. Conversely, The tool evaluates the completeness of
specific topics follow-up and selective outcome reporting in much greater
depth than the EPHPP.”...”This raizes the question of whether
the tools [are] able to adequately discriminate the
comparative quality of the included studies.”...”It is possible
that this small sample of RCTs covering a single topic area
were all of similar quality. We encourage other teams that
work with The tool to test it under different scenarios and
disciplines, to assess a range of studies of varying quality and
topic areas. This would provide further evidence of its
validity, a requirement if recommendations for clinical
practice through Cochrane systematic reviews continue to be
based on its assessment of study quality.”...”To address the
subjective nature of judging several of The tool domains, we
suggest that research groups fine-tune The tool guidelines to
make them specific to the types of study under
analysis.”…”Poor quality of reporting should theoretically
affect our ability to complete both tools. However, we argue
that the effect was pronounced with The tool, due to the
greater depth of information required to assess each
domain.”
20 Armijo-Olivo Strengths: ”…the [tools] evidence base is restricted and incomplete. 78
… Therefore, we recommend that research evidence be
(3rd included expanded to different health areas regarding the association
publication Challenges: among methodological factors (items used in quality tools)
by this The [tools] evidence base is restricted and and their link to treatment estimates, especially those that
author) incomplete involve complex interventions such as allied health areas and
physical therapy.”...”Many of the tools used to evaluate
Suggestions: methodological quality and risk of bias of health research
… have several items linked to reporting instead of conduct.7”
21 Armijo-Olivo Strengths: ”...quality of reporting is commonly used as a proxy for trial 237
… quality, which has complicated the construct of
(4th included “quality”...”...”The number of items across quality tools is
publication Challenges: large; 130 and 48 items have been used by tools in general
by this Need for guidelines in applying and health and PT research”...”Some items are subjective,
author) interpreting the results of the tool confusing, and lack a clear definition (e.g., subjects
appropriate to study questions, discussion of bias resulting
Suggestions: from non-blinding assessment).”...”The risk of bias tool is
All items included should be based on recommended by The Cochrane Collaboration. Some groups
empirical evidence within the Collaboration have developed their own tools and
have not yet adopted the risk of bias approach (e.g. Cochrane
Bone, Joint and Muscle Trauma Group). Other Cochrane
groups have modified the risk of bias tool for their own
purposes (i.e. Cochrane Back Review Group, Cochrane Renal
Review Group). The risk of bias tool was developed more
recently than many of the other tools; current research [9,13]
recommends further testing of its psychometric properties
and validation of the tool in a wide range of research fields.
Additional guidelines will help users in applying and
interpreting the results of the risk of bias tool.”...”The Jadad
scale [37] is the most frequently cited tool in health sciences
research despite criticisms regarding its lack of
responsiveness [8] and applicability to other health research
areas such as PT and rehabilitation [5].”...”Finally, items
assessing the methodological quality (or internal validity) of
Page 59 of 637
RCTs should be based on empirical evidence of their
association with treatment effects.”
22 Armijo-Olivo Strengths: ”There is no agreement regarding which tools are optimal to 305
… accurately determine trial quality. Most tools have not been
(5th included developed using scientifically rigorous methods, lack
publication Challenges: reliability and/or have not been fully validated [15]. In
by this The tool is not the gold standard addition, the use of different tools for evaluating the quality
author) Need for guidelines in applying and of primary research can lead to different end results [17-19].
interpreting the results of the tool Thus, a clinical trial may be rated on a quality scale disparate
Many domains/components are yet to be by different measurement tools. This discrepancy in the
investigated, so the evidence base is evaluation of the quality of research may skew interpretation,
incomplete reporting, and as a result, could potentially impact
recommendations for clinical care. Finally, the tools include
Suggestions: different items, some of which relate more to the detail of
… reporting rather than methodological quality.”...”As a result
of these shortcomings with existing tools and methods for
quality assessment, there has been a shift in the traditional
scoring approach to the assessment of trial quality. Instead of
examining trial quality with tools that have not been
validated and often use composite scores, the assessment of
'risk of bias’ was proposed in 2008 by the Cochrane
Collaboration, one of the most important and influential
groups working on evidence based practice worldwide
[20].”...”The risk of bias tool includes...'other sources of bias’
(for example, early stopping for benefit, design-specific
features such as adequate wash-out period in cross-over
trials). Other methodological components within the risk of
bias tool as well as other components that have traditionally
been used to determine trial quality in health research have
not been investigated; hence the evidence base is limited and
incomplete. Furthermore, recent research [5,19]
recommends further testing of the risk of bias tool to gain a
better understanding of its psychometric properties, as well
as to validate the tool in a wider range of research fields.
Additional information will help users in applying and
interpreting the results of the risk of bias tool.” –
23 Armstrong Strengths: ”…for all included studies we will use a series of questions 88
… drawn from the Effective Public Health Practice tool for
quality assessment to assess study internal and external
Challenges: validity (Effective Public Health Practice Project n.d). This
The potential complexity of studies warrants approach is important as the potential complexity of studies
investigation into factors beyond those warrants investigation into factors beyond those explored in
explored in the Cochrane 'Risk of bias' tool the Cochrane 'Risk of bias' tool. Complexities include the
increased likelihood that these studies will be prone to bias;
Suggestions: and the importance of assessing and reporting on the
… external validity, fidelity and sustainability of interventions.”
24 Bafeta Strengths: ”The risk of bias domains were poorly reported…”...”We 62
… found that more multicentre trials were at low risk of bias for
the risk of bias domains of sequence generation and
Challenges: allocation concealment than single centre trials. After
Risk of bias domains are poorly reported adjusting for each domain of the risk of bias tool as well as
overall risk of bias, we obtained similar estimates with slightly
Suggestions: wider confidence intervals.”
…
25 Barkhordaria Strengths: ”Clearly, one among the major drawbacks of the Cochrane 96
… Risk of Bias tool is the subjective nature of its assessment
protocol. In an effort to correct for this inherent weakness of
Challenges: the instrument, the Cochrane group produced detailed
The subjective nature of the tool criteria for making judgments about the risk of bias from each
individual item [16]. Moreover, Cochrane recommended that
Suggestion: judgments be made independently by at least two people,
… with any discrepancies resolved by discussion [16]. This
approach to increase the reliability of measurement in
research synthesis protocols is akin to that described by us
[19,20] and by AHRQ [21].”
26 Berkman Strengths: ”...different studies employed different definitions of the 82
The tool continues to evolve and improve approach to protecting against a bias.”...”The Cochrane tool
continues to evolve and improve with respect to
Page 60 of 637
Challenges: operationalizing criteria to adequately measure bias
Need for criteria for risk of bias consensus categories. A number of studies have evaluated the reliability
Different studies employ different definitions and validity of the Cochrane tool and suggested
of the risk of bias approach which may be an improvements in the criteria and approach to assessing risk of
important source of heterogeneity bias in general.67, 69 These variations and ongoing
improvements suggest that the different tools used to assess
Suggestions: risk of bias are themselves a source of heterogeneity.”
…
27 Bero Strengths: ”I have argued that funding source should be considered a 64
… risk of bias [in the tool].”…”Further empirical research on the
(2nd included association of industry funding with research outcomes is
publication Challenges: needed in order to understand possible mechanisms for the
by this … observed industry bias in the context of different types of test
author) drug or study design. In the meantime, the potential for
Suggestions: industry bias should not be ignored.”
The tool should include funding
Further empirical research on the association
of industry funding with research outcomes is
needed
28 Buchberger Strengths: ”The risk of bias (RoB) tool from the Cochrane Collaboration is 99
The tool strengths include ease of use, short an instrument for the assessment of the bias potential in
turnaround time, high transparency of the controlled trials. Its strengths include ease of use, short
assessment and an easy to understand turnaround time, high transparency of the assessment and an
graphical representation of the results easy to understand graphical representation of the
results.”...”The discussion on the inclusion of this bias in the
Challenges: risk of bias tool is ongoing [17, 18].”...”…the risk of bias tool of
Other types of bias not included e.g. post hoc the Cochrane Collaboration stands out with high
changes, intervention prior to randomization, transparency, recognition of the principal types of bias, easy
insensitive instruments and fraud to use with only seven items and the most understandable
graphical presentation of results.”
Suggestions:
…
29 Cho Strengths: ”When the tool is used to assess quality, allocation 81
… concealment, sequence generation and blinding items must
all be satisfied for an article to be assessed with a lower risk
Challenges: of bias.”...”...studies that make blinding impossible are
Many “high/unclear risk” trials categorized as high-quality articles in the Jadad scale if they
The tool does not consider blinding exceptions score two or more. However, the van Tulder scale and The
tool do not consider such exceptions. We look forward to a
Suggestions: consensus for new assessment tools that correct for these
... limitations in future discussions.”
30 Chung Strengths: ”...the two most commonly used tools being Cochrane risk of 82
The tool is the most commonly used tool bias tool (44.8%) and Jadad Scale (36.3%).”...”tools for
assessing risk of bias of primary studies Cochrane risk of bias
Challenges: tool 90 (44.8) Jadad Scale 73 (36.3) Others 26(12.9) Not
… reported 9(4.5) Two or more tools used 3 (1.5) Majority of
MA reported potential sideeffects of CHM (86.6%).”...our
Suggestions: data showed that risk of bias is more likely to be taken into
... account in conclusion formulation when there are a higher
number of authors.”
31 Crocetti Strengths: ”Registered trials were 68 % less likely to have a high risk of 74
… bias for sequence generation, compared with nonregistered
trials.”...”…the domain-based, risk-assessment tool developed
Challenges: by the Cochrane Collaboration is not a validated instrument.
The tool is subjective and not validated Even with prespecified definitions, we found a large amount
High disagreement of disagreement among reviewers, in that 53 % of studies
required adjudication. These results [...] reflect the amount of
Suggestions: subjective judgment inherent in assessment of some of the
... domains of the Cochrane Collaboration, risk-of-bias tool.”
32 Da Costa Strengths: ”The risk of bias tool has been widely embraced by the 95
… systematic review community.”...”The selection of domains of
bias was based on empirical evidence and theoretical
Challenges: considerations…”...”We recently found that the reliability of
Low risk of bias inter-agreement, perhaps due the risk of bias tool might be improved by intensive
to poor training standardized training of raters.”…”…no formal evaluation of
such a training intervention has been performed. We
Suggestions: therefore aim to investigate whether training of raters, with
Better training of reviewers objective and standardized instructions on how to assess risk
Page 61 of 637
of bias, can improve the within and between pairs of rater
reliability of the Cochrane risk of bias tool.”
33 De Bruin Strengths: ”Other sources of bias: Contamination and inappropriate 90
… administration, Stopping early or late for benefit: sample size
adjustments, Scientific malpractice: the fabrication of data,
Challenges: the wilful distortion of data or results, and questionable
Risk of bias assessment is interrelated and research practices…”...”...we suggest that creative
impacts trial feasibility and external validity approaches to blinding and alternative strategies to blinding
should be studied (do they work and under what
Suggestions: conditions?), and that tools for assessing the risk of bias
tools (ie. the tool) should value alternative should consider valuing alternative strategies where blinding
blinding strategies is not an option.”...”...it may not be realistic to
capture...specific [blinding] strategies in general risk of bias
assessment tools.”
34 Dechartres Strengths: ”Our results raize questions about the overall risk of bias, 96
… summarizing risk of bias across domains, as currently defined.
The [CC] risk of bias tool includes methodological
Challenges: characteristics or domains shown to be associated
The overall risk of bias on treatment outcomes individually with treatment outcomes in meta-epidemiologic
is challenging and has not been validated studies.”...”Further research is needed to explore whether
Domains may not have the same risk of bias one can obtain a simple measure of the overall risk of bias for
weight and may be associated with one a given trial and, if so, how.”...”We also recommend assessing
another the influence on treatment outcomes of each domain of the
risk of bias tool separately rather than summarizing these
Suggestions: domains into an overall risk of bias.”
Further research is needed to explore the
overall risk of bias rating
35 Faggion Strengths: ”...susceptibility to bias may differ among 98
A domain-based tool (e.g. the tool) is domains.”...”Authors should clearly state the objective of the
preferable assessment (risk of bias versus quality) and report separately
and in detail other assessments related to overall quality.
Challenges: Authors should report in detail whether the methodological
Susceptibility to bias may differ among quality results were taken into account when MA was
domains performed.”...”Because the assessment of the validity of a
study may involve subjectivity (Higgins et al. 2011a), a
Suggestions: domain-based tool (e.g. Cochrane's risk of bias tool) should
... be preferred. Domain-based tools have the flexibility of
allowing authors to judge whether a specific threatened
domain may in fact bias the estimate.”
36 Gordon Strengths: ”...we share with the Cochrane Collaboration methodologists 72
The tool does not use risk of bias scores a reluctance to provide a risk of bias score that, by its nature,
must make questionable assumptions about the relative
Challenges: extent of bias associated with individual items and fails to
… consider the context of the individual items.”...”...the risk of
bias should be considered in the context of other limitations.
Suggestions: If, for instance, reviewers find themselves in a close-call
... situation with respect to two quality issues.”
37 Hartling Strengths: “Researchers applying the risk of bias tool found it easier to 53
… assess blinding as three separate domains rather than a single
(2nd included item.”…”This study provides evidence that risk of bias due to
publication Challenges: blinding varies depending on the targeted individuals. Risk of
by this Risk of bias due to blinding varies depending bias due to 'other' sources is different from inappropriate
author) on the targeted individuals influence of study sponsor.”
Suggestions:
...
38 Hartling Strengths: ”The risk of bias tool is a more favorable approach for 93
The tool is a favourable approach assessing the internal validity of studies.”...”The factor that
(3rd included was most influential for risk of bias was the potential for
publication Challenges: inappropriate influence of study sponsors. We included this
by this Clear and consistent guidelines are needed variable within the “other sources of bias” domain; however,
author) users of the risk of bias tool may consider examining this
Suggestions: variable separately due to the complex nature of funding
Funding as a separate domain source and its influence on the design, conduct, and reporting
of trials.”…”Clear and consistent guidelines are needed for
other users of the risk of bias tool.”
39 Hartling Strengths: ”The final domain within the Cochrane tool includes an 65
… assortment of study characteristics that may lead to biased
Page 62 of 637
(4th included results, including factors associated with specific designs (eg,
publication Challenges: cross-over trials, cluster trials), blocked randomization in
by this “Small study bias” [is only included in the tools unblinded trials, and baseline imbalances.41,42 Sample size is
author) other bias domain] not included in the tool; however, some evidence suggests
“small study bias,” whereby trials with few participants may
Suggestions: be associated with exaggerated effect estimates.”
...
40 Hartling Strengths: ”...the risk of bias tool may be more appropriate for assessing 101
The tool is appropriate for assessing internal a trial’s internal validity.”...”…the results provide some
(5th included validity preliminary validation on the usefulness of the risk of bias
publication tool to identify studies that may exaggerate treatment
by this Challenges: effects.”...”We found substantial variation in agreement
author) Substantial tool agreement variation across domains of the risk of bias tool.”...”There was low
correlation between overall assessments using the risk of bias
Suggestions: tool compared with [...] the Jadad scale and the Schulz
Careful training and clear guidelines are approach.”...”Overall risk as assessed by the risk of bias tool
required when applying the tool differentiated effect estimates, with more conservative
estimates for low risk studies. Careful training and clear
guidelines are required when applying the tool.”
41 Hayden Strengths: ”In line with the Cochrane Risk of Bias tool for intervention 40
The tool does not use a summated risk of bias studies (15) and the QUADAS-2 (Quality Assessment of
scores Diagnostic Accuracy Studies) tool for diagnostic studies (16),
we recommend against the use of a summated score for
Challenges: overall study quality.”
…
Suggestions:
...
42 Hempel Strengths: ”Where possible, criteria should be selected accordingly, 71
The tools strict criterias particularly for critical appraisal instruments with very strict
criteria, such as the Cochrane Risk of Bias tool [34,35], that
Challenges: routinely result in very imbalanced distributions - given that
… the number of studies with the rare expression of the
moderator has pronounced implication for the statistical
Suggestions: power and can only be compensated for statistically with a
… very large number of trials to ensure sufficient power.”
43 Higgins Strengths: ”We believe our risk of bias tool is one of the most 46
The tool is one of the most comprehensive comprehensive approaches to assessing the potential for bias
tools in randomized trials included in systematic reviews or meta-
analyses.”...”The tool takes longer to complete than other
Challenges: tools.”...”The reliability of the tool has not been extensively
The tool takes longer to complete than other studied.”
tools
The reliability of the tool has not been
extensively studied
Suggestion:
…
44 Hopewell Strengths: “The majority (n=86/105; 82 %) of Cochrane reviews reported 49
The majority of systematic reviews uses the using the Cochrane risk of bias tool; five reported using more
tool than one tool.”...”… Cochrane reviews are more likely to
assess individual methodological components.”...”…it is
Challenges: unclear to what extent such restrictions should include all
… methodological components at high risk of bias…”
Suggestions:
...
45 Ijaz Strengths: ”The idea of using the Cochrane risk of bias tool for all RCTs, 88
The tool might be applicable for all study CCTs, quasi-randomized trials, CBA studies, cohort studies,
designs and all studies with a concurrent control group and a
prospective follow-up seems logical [19]. The tool allows
Challenges: addition of new relevant domains of bias such as adjustment
… for confounding and except for the selection bias domain,
which would probably be at high risk for all studies except
Suggestions: RCTs with adequate allocation concealment, the rest applies
… in the same way to all studies of a prospective controlled
design.”
Page 63 of 637
46 Jo Strengths: ”Validated quality assessment tools for RCTs include the 56
The tool is validated Jadad scale, the van Tulder scale, The Cochrane tool, Newell's
scale, the Scottish Intercollegiate Guidelines Network, and
Challenges: the National Institute for Health and Clinical
… Excellence.”...”...we [...] found by using the van Tulder scale
and the tool that the quality was significantly higher for
Suggestions: articles that had received funding.”
…
47 Johnson Strengths: “…the Cochrane handbook recommends a risk of bias 76
… assessment that overlaps considerably with the Downs and
Black (1998) and the Newcastle–Ottawa (Wells et al., 2000)
Challenges: inventories in its emphases on individual biases.”...”those
Many best-evidence meta-analyses and those using risk of bias nearly always left the items unscaled,
following Cochrane conventions have very consistent with Cochrane advice. None used both risk of bias
small samples because they use and another instrument.”...”Furthermore, many best-
methodological deficits as exclusion criteria evidence meta-analyses and those following Cochrane
conventions have very small samples because they use
Suggestions: methodological deficits as exclusion criteria.”
…
48 Kirkham Strengths: ”Review authors will need to use their judgment regarding 52
The tool raizes awareness of outcome the potential for outcome reporting bias.”…”Adoption of the
reporting bias new Cochrane risk of bias tool, 11 which includes a judgment
of the risk of selective outcome reporting, should also help to
Challenges: raize awareness of outcome reporting bias.”…”…consider the
… potential for bias caused by unpublished studies.”
Suggestions:
…
49 Kirkham Strengths: “The adoption of the new Cochrane risk of bias tool, which 84
The adoption of the tool helps to raize includes a judgment of the risk of selective outcome reporting
(2nd included awareness of outcome reporting bias for included studies, should also help to raize awareness of
publication outcome reporting bias. By looking at only Cochrane
by this Challenges: systematic reviews, we suspect that our study
author) … underestimates bias due to changes in outcome specification
during the systematic process. Cochrane reviews are not only
Suggestion: monitored by a CRG but also the Cochrane Handbook
… provides guidelines which offer some protection against this
type of bias [4].”
50 Lee Strengths: ”The van Tulder scale is more comprehensive because it 82
… includes 11 elements; the addition of the tool, which adds an
assessment of the risk of study bias, increases the reliability
Challenges: of the results through the use of three complementary
The van Tulder scale is more comprehensive assessment tools. Therefore, the results of the evaluation by
than the tool because it includes 11 elements these methods can differ from each other. However, this
study did not show the significant increase in the scores
Suggestion: assessed by all of the assessment methods as well as the
Assess trials with multiple risk of bias scales number of high-quality RCTs.”
51 Liu Strengths: ”In order to improve assessment of risk of bias, we 95
… recommend that the most recent version of the Cochrane risk
of bias tool be used by SR and MA authors. Reviewers should
Challenges: continue to update their knowledge according to the latest
In cases where a Cochrane risk of bias tool is Cochrane Collaboration Handbook versions and other
used, reporting is sometimes incomplete developing methodology and to clearly state which version of
the or handbook was used in their reviews.”...”In reviews
Suggestions: published after 2008 in Chinese journals, Cochrane risk of bias
... tools were not always used. In cases where a Cochrane risk of
bias tool was used, reporting was sometimes incomplete.”
52 Lundh Strengths: ”The Cochrane Handbook is currently being updated to 93
… ensure a more homogenous methodology in its reviews [43].
This revision is based on the acknowledgement of the
Challenges: discrepancies in assessment of methodological quality
The current Cochrane Handbook between the review groups [44], and it will involve
recommendations are not clear about the introduction of a detailed risk-of-bias tool to be used in all
usage of risk-of-bias assessments reviews. The tool will also address bias in selective outcome
reporting [45,46]. Finally, we suggest that the revision should
Suggestions improve recommendations for assessing attrition bias and the
The tool revision should improve usage of the risk-of-bias assessments, as the current
recommendations for assessing attrition bias recommendations are not clear about this.”
Page 64 of 637
and the usage of the risk-of-bias assessments
53 Lundh Strengths: ”...our data suggest that industry sponsorship should be 75

… treated as bias-inducing and industry bias should be treated
(2nd included as a separate domain.”...”Interestingly, the AMSTAR tool for
publication Challenges: methodological quality assessment of systematic reviews
by this The tool is unable to catch bias introduced by includes funding and conflicts of interest as a domain (Shea
author) funding 2007). Methods for reporting, assessing and handling industry
bias and other biases in future systematic reviews must be
developed. Specifically, further methodological research
Suggestions: should focus on how industry bias is handled in Cochrane
Further methodological research should focus reviews.”
on how industry bias is handled
54 Manchikanti Strengths: ”…once the learning curve has passed, reviewers will not only 23
Once the learning curve has passed, reviewers become comfortable but will start appreciating various
will not only become comfortable but will insights of this instrument [the tool].”
start appreciating various insights of the tool
Challenges:
…
Suggestions:
...
55 O’Connor Strengths: ”Risk of bias should be judged for every outcome reported 66
… and the judgment reached may differ among different
outcomes within the same experiment. Assessment of the
Challenges: risk of detection bias for a particular outcome often requires
The risk of bias might differ between different content expertize.”...”To assess internal validity, for each of
outcomes the risk domains, we would propose following the approach
proposed by the Cochrane Collaboration (Higgins et al. 2011)
Suggestions: [ie. using the tool].”
Risk of bias should be judged for every
outcome [proposedly with the tool]
56 Oremus Strengths: ”While the Cochrane Collaboration has stated that quality 81
… scales and scale scores are inappropriate means of
ascertaining study quality, 33 our results are relevant because
Challenges: many researchers continue to use the Jadad Scale and NOS in
… their systematic reviews. Indeed, our work suggests an area
of future research. The Cochrane Collaboration has proposed
Suggestions: a ‘risk of bias’ tool to assess the quality of RCTs.33 The
The tool reliability should be assessed in raters reliability of the risk of bias tool should be assessed in raters
with different experience levels with different levels of experience.”
57 Robertson Strengths: “We achieved fair agreement between reviewers for risk of 57
The tool proved useful for demonstrating bias assessment of non-randomized studies. Although the
conservative effect estimates process was time consuming, using a modified version of the
Cochrane (risk of bias) tool proved useful for demonstrating
Challenges: conservative effect estimates in our systematic review. We
… suggest including more risk of bias levels and further
validation could improve inter-rater agreement.”
Suggestion:
More tool levels and further validation could
improve inter-rater agreement
58 Robinson Strengths: ”Guidance is available on assessing the risk of bias of 33
Guidance is available on assessing the risk of individual studies, though little guidance specifically
bias of individual studies with the tool adldresses how to incorporate this information from existing
reviews into a new systematic review [with the tool].”
Challenges:
Little guidance addresses how to incorporate
risk of bias information from an existing
Cochrane review into a new Cochrane review
Suggestions:
…
59 Roseman Strengths: ”Currently, the Cochrane Collaboration's Risk of Bias tool 48
The tool includes an optional “other bias” includes an optional “other sources of bias” domain, 63 which
domain, which could be used to include meta-analysts could use to include information on COIs
conflicts of interests information [conflicts of interests]. We recommend that the Cochrane
Collaboration consider formalizing the requirement to assess
Page 65 of 637
Challenges: potential bias from COIs [conflicts of interest].”
…
Suggestions:
Funding and COI [conflicts of interest] should
be assessed as risk of bias
60 Sargeant Strengths: ”The Cochrane ‘Risk of Bias'; tool could be easily adapted for 99
The tool can be adapted to veterinary use in veterinary medicine.”...”A useful validation and
medicine reviewer training strategy are to have all of the reviewers
[The tool] is a useful resource for reviewers conduct risk of bias on a selection of articles that pass
relevance screening. Poor agreement between assessors may
Challenges: lead to revisions or rewording of the risk of bias questions or
… may lead to additional training of reviewers in critical
appraisal of the literature. There is an entire section of the
Suggestions: Cochrane Review Handbook devoted to assessing the risk of
… bias (Higgins and Green, 2011), which is a useful resource for
reviewers.”
61 Savović Strengths: “Most respondents thought that risk of bias assessments 94
Most authors likes the risk of bias were better than past approaches to trial quality assessment.
(2nd included standardized approach and find the workload Most authors liked the standardized approach (81%) and the
publication acceptable ability to provide quotes to support judgments (74%). About a
by this third of participants did not like the increased workload, and
author) Challenges: found the wording describing judgments of risk of bias to be
Some find the wording describing judgments unclear. Most authors (75%) thought availability of training
of risk of bias to be unclear materials was sufficient...”…”…respondents identified positive
experiences and perceptions of the risk of bias tool. Revisions
Suggestions: of the tool and associated guidance, and improved provision
… of training, may improve implementation.”
62 Shamliyan Strengths: “Future tools [ie. the tool] should evaluate quality at both the 95
… study and hypothesis levels. One study can examine more
than one risk factor with different degree of bias [86], [87]
Challenges: and [88] examined estimates in racial, ethnic, or other
… subgroups can have a greater degree of bias compared with
the total study estimates [89], [90], [91], [92], [93] and
Suggestions: [94].”...”Future appraisal tools should grade the quality of a
Future tools should evaluate quality at both study as it applies to various subgroups when applicable. We
the study and hypothesis levels concluded that numerical scores are meaningless when
Future tools should grade the quality of a examining the quality of studies in systematic reviews [95].”
study as it applies to various subgroups when
applicable
63 Shrier Strengths: ”Risk of Bias tool may lead to double-counting of bias, and 25
… inappropriate inferences.”...”…[the] current Risk of Bias tool
[is] appropriate for observational studies with slight
Challenges: modifications.”
Risk of Bias tool may lead to double-counting
of bias, and inappropriate inferences
Suggestions:
…
64 Sinha Strengths: “Many studies had an unclear risk of bias.”...”The tool has 37
The tool has changed with blinding of changed over time, with biases related to blinding of
participants and personnel assessed participants and personnel assessed separately from blinding
separately from blinding of outcome of outcome assessment in the current tool, 5 adding clarity
assessment adding clarity for reviewers for reviewers.”
Challenges:
Many “unclear” risk trials
Suggestions:
…
65 Turner Strengths: ”...the Cochrane risk of bias tool suggests blinding of 96
… participants and personnel, and blinding of outcome
assessment be assessed separately. Moreover consideration
Challenges: should be given to the type of outcome (i.e. objective or
Consideration should be given to the type of subjective outcome) when assessing bias…”...”...there has
outcome (i.e. objective or subjective outcome) been some criticism of the Cochrane risk of bias tool [109]
when assessing bias concerning its ease of use and reliability
[110,111]...”…”…[Further studies are needed to explore the
Page 66 of 637
Suggestions: role and effect of] small study effect, reporting bias, higher
Further studies are needed to explore the role risk of bias in single centre studies, or factors related to the
and effect of small study effect, reporting bias, selection of the participants, treatment administration and
higher risk of bias in single centre studies, or care providers’ expertize.”
factors related to the selection of the
participants, treatment administration and
care providers’ expertize
66 Voss Strengths: “...the Cochrane Collaboration Risk of Bias tool represents an 80
The tool represents an accepted standard for accepted standard for systematic reviews.”...”It would be
systematic reviews informative to explore how different ways of deriving a
summary score and more or less rigorous cut-off values might
Challenges: influence the results.”...”Our study emphasizes the
… importance of quality appraisal in systematic reviews, and the
need to develop and test a gold standard tool that is
Suggestions: applicable across a broad range of epidemiological study
It would be informative to explore how designs and addresses all major aspects of internal and
different ways of deriving a summary score external validity.”
might influence results
Need to develop and test a gold standard tool
that is applicable across a broad range of
epidemiological study designs and addresses
all major aspects of internal and external
validity
67 Wells Strengths: ”The next major step is to extend the existing Cochrane Risk 95
… of Bias tool so that it can assess the risk of bias to NRS
included in a review. This initiative involves detailed
Challenges: consideration of how the assessment of each bias domain
… may need to be customized to study
designs.”...”Fundamental research into a number of particular
Suggestions: topics for systematic reviews of NRS is critical. Topics such as
The next major step is to extend the existing the assessment of publication bias, and the need and
Cochrane Risk of Bias tool so that it can assess justification for comprehensive searching require empirical
the risk of bias to non randomized trials evaluation. Empirical research is currently underway on
selective analysis reporting related to RCTs.”
68 Zeng Strengths: ”For primary research, the Collaboration’s recommended tool 61
The tool is widely accepted and recommended for assessing risk of bias of RCT is neither a scale nor a
checklist; it is a domain-based evaluation, in which critical
Challenges: assessments are made separately for different domains
… (5,12). The “Cochrane Collaboration’s tool” is very widely
accepted and recommended. Additionally, the Modified
Suggestion: Jadad Scale and PEDro Scale are also suitable tools for RCT.”
…
Page 67 of 637
Table III: Peripheral remarks on the Cochrane risk of bias tool for randomized clinical trials
Reference First author Selected commentaries from references

69 Bala ”We conclude that RCTs published in higher impact journals were, on average, less prone to risk of bias compared with those
published in lower impact journals in their reported design, conduct, and analysis. There remains a substantial room for
improvement among RCTs published in all Core Clinical Journals. Journal editors and reviewers in higher and lower impact
journals should insist on comprehensive explicit reporting of methodological (concealment of allocation, blinding, and loss to
follow-up) and statistical issues (effect estimates and confidence intervals), including reporting of subgroup analyses
(prespecified subgroup analyses and test of interaction). Users of the medical literature should critically appraize each trial
regardless of the reputation of the publishing journal.”
70 Balk ”We conclude that more research is necessary to better understand the utility of this new translation tool to reduce the risk of
language bias in systematic review. However, in the meantime, it may be worthwhile for EPCs to devote the small amount of
resources and effort necessary to try Google Translate to include non-English articles. It will be important, however, to recognize
that extraction of these articles is more prone to error than extraction of typical English language articles.”
71 Bassler ”In our prior work, we repeatedly referred to the bias associated with stopping RCTs early for efficacy. We now see this as a
misleading characterization. As we will describe below, the overestimates that occur in trials stopped early for benefit are largely
the result of random error. There is a bias, in addition, a bias that stopping early creates, but it is a relatively minor contributor to
the overestimation. There is also potential for bias at the level of the meta-analysis, but this is due to factors other than bias at
the level of the individual study.”...”In our view, the danger of overestimation will be particularly high when all the following
three conditions exist: tRCTs have a relatively small number of events (e.g. <200), there is a substantial difference (e.g. a ratio of
RR < 0.7) in the RRs between the tRCTs and the nontruncated RCTs; the tRCTs have a substantial (>20%) weight in the meta-
analysis. When these conditions exist, systematic reviews should offer sensitivity analyses with and without inclusion of tRCTs.
When the three conditions do not exist, and RCTs have instituted safeguards against bias, and yield precize estimates that are
consistent across trials,25,26 authors of systematic reviews and meta-analyses can, regardless of the presence of tRCTs, have a
high level of confidence in the results.”
72 Berger ”Zambon et al. [1] appraized the quality of a series of meta-analyses, and found that “Internal validity appeared largely robust, as
most (50.5 %) reviews were at low risk for bias.” To conclude that there is a low risk of bias, a comprehensive review is required,
so that all potential biases are considered. Otherwize, we might as well notice that some raindrops have missed us as we run
through the rain, and conclude, on that basis, that therefore we must be dry. This wishful thinking provides a false sense of
security that interferes with required reforms, and is potentially quite harmful.”...”It is rather unnerving, given that randomized
trials are the worst possible design except for all the rest [2], that fully half of the reviews could not meet even these minimal
requirements, but what about the ones that did? We are told nothing about how well or poorly the trials were randomized, or
even if they truly were randomized at all. The risk of bias depends critically on the precize methods of randomization [3], and not
every trial that is labeled as randomized actually is [4]. Nor are we told how successful the masking effort was; the risk of bias is
clearly high if masking is unsuccessful, and the effort should never be confused with completion of the mission (Section 1.8 of
[3]). Beyond that, nothing is mentioned of the myriad numbers of other potential biases, including improper enrichment,
improper surrogate endpoints, changing endpoints, post-randomization exclusions, and analyses whose validity is predicated on
untenable assumptions.”...”Half of the meta-analyses should be dismissed out of hand, because they were unable to meet even
the barest minimal requirements of validity. The other half are at a rather high risk of bias if we know nothing more about them
than that they used the words “randomized” and “masked”, without qualification. And, unfortunately, the risk of bias in future
trials and systematic reviews can only be expected to increase if we see more articles that turn a blind eye and contort
themselves to somehow find something to praize. Valid trials need to address and rule out all the aforementioned potential
biases, and valid appraisals of trial quality and internal validity need to do the same [5].”
73 Berger ”Foley et al. [1] lamented the fact that allocation concealment is described inadequately in two-thirds of the studies”...”Foley et
al. [1] also pointed out that allocation concealment continues to be misunderstood by many investigators”...”When the
(2nd methodology is flawed, it sets a dangerous precedent. Hence, the record needs to be set straight, even when the right
peripheral conclusions are found. In this case, the issue is what it takes to establish the concealment of future allocations. Is it enough to
remark by follow the Schulz and Grimes [3] definition?”...”here are two threats, including both (1) direct observation of the allocation
this author) sequence and (2) prediction of future allocations based on knowledge of the past ones [4] ([5]; Sections 2.5–2.6).”...”In fact one-
third of the trials met only an appalling low standard of pseudo allocation concealment, but it is highly doubtful that even 10%
reported and/or operated with true allocation concealment. Of course, we have no way to know for sure, because Foley et al. [1]
did not enumerate the trials considered and the description of allocation concealment in each. What we do know for sure is that
allocation concealment remains misunderstood even by those who would try to explain it to others. We know that the relevant
literature continues to be ignored when it is convenient for authors to do so. And we know that the vast majority of trials cannot
meet even so low a standard as the wink and the nod required, along with keeping a straight face, when reporting that the trial
had or did or was blessed by allocation concealment. We know that in the future we all need to do better. Patients are depending
on us to do so.”
74 Bialy ”None of the studies had an overall low risk of bias. Most studies had a low risk of bias for the domain of incomplete outcome
data (89%), while 63%, 55% and 46% of trials had low risk of bias for sequence generation, other sources of bias, and blinding of
outcome assessors, respectively.”...”We examined risk of bias among a sample of 208 trials included in 24 neonatal systematic
reviews. The majority of trials had an overall high risk of bias; none had an overall low risk of bias. This is consistent with other
studies that have shown few trials to have low risk of bias overall (12,16,20–22). ”...”Selective reporting of outcomes was a major
contributing factor to the number of trials assessed overall as high risk. Further, trials that were at high or unclear risk of bias for
selective outcome reporting were associated with an overestimate of treatment effects by 47%.”...”For the majority of other
domains, we did not find an association between risk of bias and effect estimates. This may be explained by the fact that most of
the outcomes in our analysis were objective (i.e. 15/25 reviews included mortality in their primary outcome, Table 4). A recent
study by Savovic et al. that combined seven meta-epidemiological studies involving 234 meta-analyses and 1973 trials found an
Page 68 of 637
exaggeration of treatment effects among studies with inadequate or unclear random-sequence generation, allocation
concealment and blinding 32. However, they also showed that the bias was driven largely by trials reporting subjective outcomes
with little influence by trials with objective and mortality outcomes. Another explanation for our findings may be related to
insufficient power to detect differences; in particular, there were often few trials in the low risk of bias category.”
75 Chaimani ”Based on the Cochrane Risk of Bias tool, 19 we sought information regarding the random sequence generation, allocation
concealment, and blinding of patients and outcome assessors.” META-epidemiologic study.
76 Chase ”While meta-analysis is a powerful tool to overcome the variation among studies and arrive at an answer to a particular scientific
question (e.g., does a particular intervention alleviate the symptoms of a dizease?), it is less powerful in its ability to detect
publication bias and the selective presentation of analyses. In the biomedical sciences, such biases not only slow the progression
of science, but they could also result in bringing ineffective or harmful substances to clinical trial, creating considerable financial
and health costs. Thus, it is important to understand just how rampant these biases are.”...”With increasing numbers of humans
afflicted with neurological disorders, millions of animals sacrificed in the name of research, and billions of dollars spent on health
care, it is imperative that biomedical scientists take action to alleviate these biases. Tsilidis and colleagues advocate a number of
such actions, including the development of standard reporting protocols, preregistration of experimental design, and provisioning
of raw data to the broader community, all of which should allow more efficient development of dizease interventions from
animal models to clinical trials.”
77 Chess ”It is important to note the difference between methodological quality and reporting quality. Our study is designed to evaluate
the methodological conduct of studies; however poor reporting can innately make this task difficult. While it is imperative to
decipher between reporting and methodology, it can be tempting to draw similar conclusions from both. This will ultimately
hamper a true risk of bias assessment and must not be carried out.”
78 Chiappelli ”Cochrane organization, now established world-wide and across all continents: the premier entity for generating and
disseminating systematic reviews, and establishing the fundamental research synthesis methodology, including the risk of bias
assessment tool.”...”Most of the scales available for this purpose are derived, or expanded from the original JADAD scale [9], and
are often limited to rating subject randomization, blinding, and drop-out. These domains are hardly representative of the vast
number of criteria that establish the standards of research methodology, design and data analysis. Other available instruments
often suffer from fundamental flaws of reliability, including most importantly standardization of the readers as noted by Hartling
and collaborators in a 2012 AHRQ report (“Inter-rater variability resulted more often from different interpretation of the tool
rather than different information identified in the study reports…”) [10]. The report further states in no uncertain terms the
“need to determine inter-rater reliability and validity in order to support the uptake and use of individual tools that are
recommended by the systematic review community…” (p. 2). In brief, current trends demand an articulated research program for
validating these tools, because of the fundamental importance and relevance of sound assessments of research quality to the
process of obtaining the best available evidence...”
79 Christensen ”There is empirical evidence that OA trials may be affected by selection and detection bias [11]. Allegedly, few patients noted the
taste of fish oil during 12 weeks of taking such capsules three times per day. We argue that a fishy taste in the mouth might
certainly cause detection bias. Assessment of the trial reporting in terms of risk of bias, the use of random assignment, and
subsequent concealment of allocation would qualify as adequate (that is, low risk of selection bias); it seems reasonable that at
baseline the patients in the study groups were similar with respect to prognostic factors. The reporting of double-blinding
supports a low risk of performance bias as the authors state that the manufacturer provided both the Phytalgic® and placebo
capsules and that it claimed that they were identical and indistinguishable. We argue, however, that it might be difficult to hide
the taste of fish oil during a 3-month trial, probably as difficult as it is to hide the taste of ginger [5]. Finally, deviations from
protocol and loss to follow-up often lead to the exclusion of patients after they have been allocated to treatment groups, and this
may introduce attrition bias [12]. We are concerned about the fact that the trial registration was done after study completion
(April 2008). Thus, we would categorize the risk of attrition bias as being at best unclear as there is a possibility that some
patients were excluded from the analyses. Although the authors performed their analyses according to the intention-to-treat
principle on what they claim is the correct sample size, we worry about the fact that the attrition rate was 10% (4/40) in the
placebo group, whereas only 2% (1/41) withdrew from Phytalgic®.”
80 Chung ”There are various types of qualitative assessment tools for RCTs including Campell, Moher, Chalmers, Jadad, van Tulder,
Newell's, and Cochrane. The interesting point in this study is that by using three different tools, we found differences in the
qualitative analysis outcomes of RCTs. The assessment of the quality of trials remains controversial, and there is no consensus on
highly accurate and valid tools (17). However, in this study, efforts were made to overcome such limitations by using three
different tools: the Jadad scale, van Tulder scale, and the tool. These are representative assessment tools used most commonly
nationwide and worldwide. In particular, the Jadad scale has advantages in the simplicity of the assessment questions and ease of
assessment performance, but it does not include assessment items for the most important item of RCT assessment: concealment
of allocation. Therefore, additional analyses were performed using the van Tulder scale and the tool to supplement in this
regard.”
81 Chung ”Some generic quality issues are applicable to all SRs. For example, a comprehensive and transparent search strategy, with
adequate justifications for inclusion or exclusion of specific studies, is needed to ensure an unbiased selection of studies for SRs
(2nd and to improve understanding of how the SR was conducted. Furthermore, searching for unpublished data and comparing them
peripheral with published data could shed some insights on the potential impact of publication bias (175). There is an underlying suspicion
remark by of publication bias against studies having either null or negative outcomes (176). It is important to note that there are no reliable
this author) methods to measure publication bias. Studies have shown that the most frequently used method to assess publication bias
(funnel plots) can be misleading (177–179). Quality assessment of the primary studies is essential for the evaluation of validity
and the overall strength of the conclusions in an SR.”...”Without quality assessments, the validity of the included primary studies
is unclear and the impact of the potential biases in the primary studies on the conclusions of an SR cannot be assessed.”
82 Da Costa ”The Cochrane Collaboration recommends a component approach to the assessment of trial quality and risk of bias, rather than
the use of quality scales such as PEDro [12]. It was wondered whether the approach chosen might affect the conclusions of
Cochrane reviews. To examine this, the review of transcutaneous electrostimulation for osteoarthritis of the knee was revisited
[13]. The PEDro scores were available for 15 of the 16 trials included in the meta-analysis, with scores ranging from three to
Page 69 of 637
eight. These 15 trials contributed 17 comparisons. The Cochrane review had identified only one trial of clearly high quality [14],
with adequate generation of random sequence, concealment of allocation, and blinding of study participants and therapists. This
trial received a high PEDro score of eight; however, other trials that were also highly scored by PEDro lacked adequate
concealment of allocation and blinding of patients or therapists [[15], [16], [17], [18], [19]]. A high PEDro score does thus not
mean that the trial was adequately randomized and blinded.”...”The Cochrane meta-analysis was repeated [13] using the scores
from the PEDro database to identify high-quality trials. Fig. 1 shows the summary effect sizes for pain for all trials, for trials of
higher quality using different PEDro thresholds, and for the high-quality trial identified in the original review [13]. With the PEDro
scale, the beneficial effect of electrostimulation became more prominent as the quality of the trials increased. In contrast, the
estimate from the high-quality trial was close to zero, indicating little benefit of electrostimulation. The previous review
concluded that the evidence was “inconclusive, hampered by the inclusion of only small trials of questionable quality” [13]. It
seems likely that based on the results from the PEDro scale, many reviewers would conclude with the contrary opinion that there
was robust evidence from high-quality trials that electrostimulation had a clinically relevant, beneficial effect on pain in
osteoarthritis of the knee.”...”Greenland [20] described quality scoring as “perhaps the most insidious form of subjectivity
masquerading as objectivity in meta-analysis”: the effects of quality dimensions that are important in a given study and context
are diluted or confounded by the summary quality score, sometimes to the point that quality effects are no longer evident, or
that effects are reversed, as in our example. The PEDro scale and many other quality scales include items that are not in fact
related to the methodology and extent bias was avoided in a trial, but to the quality of reporting. Furthermore, items that are
important for some interventions or outcomes may not be relevant in other situations but will receive the same consideration.
For example, blinding of study participants will be crucial for pain but irrelevant for all-cause mortality.”...”the PEDro database's
inappropriate emphasis on the use of summary scores from a quality scale makes the database less useful than it might be. It is
likely that bias is introduced into systematic reviews and meta-analyses when these scores are used as the main criteria on which
the inclusion or exclusion of trials is based. We suggest that the use of summary scores should be discouraged, and that the
PEDro database be restricted to presenting the scores for individual items of the scale. The addition of items, for example on the
prevention of differential expertize bias, and the removal of items not related to the risk of bias, might further enhance the value
of this important initiative.”
83 Davey ”It is clear that the numbers of studies eligible for meta-analyses are typically very small for all medical areas, outcomes and
interventions covered by Cochrane reviews. This highlights the particular importance of suitable methods for the meta-analysis of
small data sets.”
84 Dechartres ”Results were robust after adjustment for RCT sample size, risk of bias as assessed by the risk of bias tool, variance of the log OR
with consideration of publication bias, and funding source.”...”Sensitivity analyses adjusted for the domains of the risk of bias
tool showed consistent results, with single-center status still significantly associated with larger treatment effect size.”
85 Doi ”Da Costa et al. [1] suggest that the scoring of trials using quality scales be discontinued and try to provide evidence against such
use by citing an example of a scale widely used in trials of physiotherapy. They repeated a Cochrane meta-analysis on
electrostimulation for pain [2] using the scores from the PEDro database to identify higher vs. lower quality trials and pooled
summary effect sizes for pain for all trials, trials of higher quality using different PEDro thresholds, and trials with adequate
randomization, allocation concealment, and blinding. They note that with the PEDro scale, the beneficial effect of
electrostimulation became more prominent as the quality of the trials increased. In contrast, the estimate from a single study
that had adequate randomization, allocation concealment, and blinding (Garland 2007 in Fig. 1) was close to zero, indicating little
benefit of electrostimulation. The implication they derived from this seemingly contradictory result was that quality effects were
reversed, and although some of the blame was attributed to the characteristics of the PEDro scale, a large amount of criticism
was leveled at the use of summary scores[1].”...”In reality, stratification of studies by quality score also stratifies them by
precision and effect sizes. Thus, intermediate quality studies are likely to fall on different sides of the quality threshold with the
use of different thresholds. If these studies happen to have a larger effect size or higher precision, this will markedly affect the
magnitude and direction of the pooled effect under different quality thresholds depending on which side of the threshold a
cutoff assigns such studies.”...”This sort of meta-analysis is thus amenable to serious bias from redistribution of precision and
effect sizes depending on how the intermediate quality studies get assigned.”...”The subsequent variation in precision and effect
sizes in the high- and low-quality groups was what led to discrepancies, and in the latter study too, there was effectively no
quality effect assessed [6]. We conclude therefore that what da Costa et al. [1] report is not PEDro's bias but rather a bias in their
interpretation of effects.”
86 Dossing ”The Cochrane Collaboration's Handbook for Systematic Reviews of Interventions provides guidance to authors to critically
review trial outcome using the risk of bias (risk of bias) assessment tool.5 The risk of bias tool requires authors to evaluate the
well-established strengths and limitations of RCTs, including sequence generation, allocation concealment, blinding of
participants, personnel and outcome assessors, loss to follow-up with failure to apply the intention-to-treat (ITT) principle and
selective outcome reporting.5 7 In the course of meta-epidemiological studies, other sources of bias in RCTs have been identified,
such as significant discrepancies favouring intervention in single (rather than multicentre) trials, in trials with small (rather than
large) sample sizes and in using subjective (rather than objective) outcome measures.3 8–11 Most recently, funding source has
become a distinct possibility as a source of bias, with for-profit organisation funding likely favouring prointervention results,12–
14 however there is an on-going debate as to whether funding should be regarded as a risk of bias item.15 16”...”This study may
point to potential bias and disadvantages in the handling of missing data in RCTs, otherwize known for having a low risk of bias
compared with other study designs.62 SI has been criticized on a theoretical level, but its implication on efficacy outcomes in RA
trials is uncharted. Accordingly, this study may provide empirical evidence that can support or contradict existing critics.
Regardless of our findings one should always be careful when interpreting results from trials where data are missing and consider
the reasons for missing data and potential impact on effect estimates.7 63”
87 Duclos ”Other sources of reporting bias may include design-specific risks of bias, early stopping, baseline imbalance, blocking of
experimental units in unblinded studies and differential diagnostic activity.”...”After carefully reviewing each study for potential
bias, an overall assessment of the evidence for bias and the likely direction(s) and magnitude of the bias(es) should be made. If
many of the studies that constitute the evidence base have a high likelihood of substantial bias, any conclusions should be
treated with circumspection. Studies at high risk for bias may be excluded if the results are deemed unreliable.”
Page 70 of 637
88 Duvendack ”Bias assessment in quasi-experiments is complicated by the nature of the validity assumptions, for example unconfoundedness
or exogeneity (Morgan and Winship 2007). There are a large number of existing tools to assess risk of bias, many of which enable
integrated assessment of experimental and quasi-experimental designs 6 . The tools mostly rely on the assessment of group
comparability in terms of observable covariates. Although some of the tools include vague questions about statistical validity,
none provide further guidance to assess selection (and placement) bias and statistical analysis comprehensively.
Operationalisation of existing tools to assess quasi-experimental designs used in development (including RDD, IV, PSM, DID) may
therefore lead to simplistic and inappropriate study classifications. Adequate assessment of selection bias in quasi-experiments
requires analysis of the methods of counterfactual identification to address selection bias (that is not just whether the study used
random allocation), among other factors including file-drawer effects and the use of appropriate specification tests.”
89 Dwan ”Hahn et al [13] compared the funder stated in protocol to publication. These studies indicate that funding is an important factor
to consider when investigating publication bias and outcome reporting bias, however more work needs to be done to examine
common questions before conclusions regarding the relationship between funding and outcome reporting bias can be
drawn.”...”Another recommendation is to conduct empirical evaluations looking at both ORB and study publication bias in RCTs
to investigate the relative importance of both i.e. which type of bias is the greater problem. The effects of factors such as
funding, i.e. the influence of pharmaceutical industry trials versus non pharmaceutical trials, should also be factored in these
empirical evaluations.”
90 Dwan ”Systematic reviewers need to ensure they access all possible trial documentation, whether it is publicly available or obtained
from the trialists, in order to assess the potential for selective reporting bias for analyses. The Cochrane risk of bias tool is
(2nd currently being updated, and the revized version will acknowledge the possibility of selective analysis reporting in addition to
peripheral selective outcome reporting. Selective analysis reporting generally leads to a reported result that may be biased, so sits more
remark by naturally alongside other aspects of bias assessment of trials, such as randomisation methods, use of blinding, and patient
this author) exclusions. Selective outcome reporting may lead either to bias in a reported result (e.g., if a particular measurement scale is
selected from among several) or to non-availability of any data for a particular outcome (e.g., if no measures for an outcome are
reported). The latter sits more naturally alongside consideration of “publication bias” (suppression of all information about a
trial).”
91 Elkhadem ”Quality of included studies was determined using Cochrane Collaboration risk of bias tool, while publication bias was not
assessed.”...”The authors' recommendation should be considered with caution. First, three out of eight included studies
performed adequate sequence generation, allocation concealment and blinding while the five remaining studies showed a high
risk of bias. Nevertheless, all eight studies were included in the meta-analysis, which poses a threat on the validity of the final
pooled estimate.”
92 Elkhadem ”Quality of reporting trials was assessed using Cochrane Collaboration Risk of Bias tool. Trials with high risk of bias were excluded
from meta-analysis. Four split mouth RCTs discussing bond failure risk were excluded due to inadequate or unclear
(2nd randomisation procedures. These four trials included 1458 brackets in the SEP group and 1460 brackets in the AE group. No
peripheral sensitivity analysis including trials with high risk of bias was performed; hence their effect on the direction of the results cannot
remark by be concluded.”
this author)
93 Engebretson ”…studies were analysed separately according to risk of bias evaluation. Heterogeneity between studies was calculated in order
to determine whether publication bias was significant.”...”For a meta-analysis to be high quality evidence, each of the trials
included must also be of high quality, and that means each must have sufficient statistical power on its own.”
94 Eriksen ”In addition, most of the positive findings were commercially funded ([16, 67]), which involves a risk of bias. To address this
problem, we performed a sensitivity analysis restricted only to studies with a low risk of bias.”
95 Faggion ”Most domains were judged to be at unclear risk of bias and therefore it is not possible to determine the degree of
(un)biasedness of the described treatment effects. Note that risk of bias and quality of reporting should be considered distinct
from each other.”...”Although direct contact with authors of the study might be an attempt for clarifying dubious or lack of
information, this does not guarantee the accuracy of information provided (Haahr and Hróbjartsson 2006). We therefore adopted
a conservative approach for assessing the domains; that is, we considered “unclear entries” as unclear risk of bias, although many
of these domains would probably be scored at high risk of bias.”
96 Ferreira ”Taking into account the set of 4,581 studies, only one study (0.22%) was classified as presenting low risk of bias for the four
dimensions assessed and 77 potential RCTs (98.72 %) were classified as presenting high risk of bias.”
97 Fleming ”As expected, the quality of Cochrane reviews was significantly better than non-Cochrane SRs. An area of particular concern in
relation to non-Cochrane reviews was the failure to register reviews at the outset. Registration of Cochrane reviews is mandatory
with publication of a protocol a priori. Use of a protocol pre-specifies the objectives and methodology reducing the risk of biased
post hoc decisions.”…”In the present review, a problem with classification arose in respect of risk of bias assessment. Inclusion of
a quality assessment is advocated in the AMSTAR guidelines as part of a comprehensive review process. However, QUORUM
guidelines (Moher et al., 1999), which were advocated as a template for reporting prior to the advent of PRISMA (Liberati et al.,
2009), recommended the use of methodological quality assessment rather than risk of bias assessment. Consequently, where
assessment of methodological quality was undertaken, this was taken to be synonymous with risk of bias assessment in the
present review. It should be noted, however, that risk of bias assessment is considered best reporting practice presently; this
approach should therefore be used until the PRISMA guidance is superseded.”
98 Flodgren ”CR has three additional domains in risk of bias-table: ”Similar baseline outcome measures”, ”Similar baseline characteristics”
Page 71 of 637
and ”Protection against contamination”. CR uses ”Other Bias” for: ”Only less than half of eligible hospitals agreed to participate
which creates a greater risk of selection bias since the hospitals that declined to participate were different from the others (small
and rural)” (High risk).”
99 Flores-Mir ”…a point that I would like to raize is the inclusion of only RCTs. Due to the fact that the quality of the included RCTs was mainly
poor, why not to include non-RCTs? Would they provide additional data that may be clinically useful, even considering the
increased risk of bias among lower level of evidence studies? Sometimes a well-conducted non-randomized clinical trial can
provide sound evidence in comparison to a poorly conducted RCT.”
100 Forbes ”An attempt to blind participants and personnel does not always ensure successful blinding in practice. For example, for many
blinded drug trials, the side effects of the drugs can reveal group allocation, unless the study compares two rather similar
interventions (eg, drugs with similar side effects, or uses an active placebo.6 It has been suggested that it would be useful to ask
trial participants at the end of the trial to guess which treatment they have received,7 ,8 and some reviews of such reports have
been published.7 ,9 Evidence of correct guesses exceeding 50% would suggest that blinding may have been broken. However,
responses may simply reflect the patients' experiences in the trial. A good outcome will tend to be more often attributed to an
active treatment, and a poor outcome to a placebo.10 Risk of bias may be high for some outcomes and low for others. For
example, knowledge of the assigned intervention may impact on behavioural outcomes (eg, number of visits to their physicians),
while not impacting on physiological outcomes or mortality. Thus, assessments of risk of bias resulting from lack of blinding may
need to be made separately for different outcomes. Rather than assessing risk of bias for each outcome separately, it is often
convenient to group outcomes with similar risks of bias. For example, there may be a common assessment for all subjective
outcomes (eg, quality of life) that is different from objective outcomes (eg, blood work).”
101 Foster ”Bias — that is, the systematic distortion of the estimates due to poor design, conduct, or analysis of a trial5 — is particularly an
issue for pragmatic trials of complex interventions where ‘real world’ estimates are paramount. Maximising generalisability and
access to the key target group is important by careful attention to how the trial is communicated to potentially eligible patients,
and recruitment methods that avoid reliance on busy clinicians (for example, using Read-Code activated electronic tags, or
mailed invitation or population screening surveys of registered patients). Where the unit of randomisation is not the patient
(cluster randomized trials) using such methods will also lessen selection bias, as clinicians in the control group may be less
enthusiastic in recruiting patients, leading to differential selection bias.”...”The choice of control intervention is a difficult issue in
trials of complex interventions. Trials with waiting list controls, usual care, or ongoing ‘stable’ medication provide intervention
estimates which reflect the combined specific and non-specific effects that will accrue in practice, and are more likely to show
between-group differences,6 but are less likely to motivate eligible patients to participate, and make attrition bias more likely.
Conversely, using a ‘credible’ control or comparison intervention can help control for the known non-specific effects of complex
interventions, ensure that the trial results are not simply explained by regression to the mean, natural history of the complaint or
attention from a health professional, but provide an underestimate of the total effect of the intervention. More than one control
group can be useful to tease out some of these issues,7 but equally it can be very difficult to separate fully the specific and non-
specific effects of an intervention.8 Performance bias is a particular issue since an integral component of complex interventions is
frequently patient–practitioner interaction or the ‘therapeutic relationship’. Complex interventions are rarely in standardized
formats like a simple pill, rather most are individualized depending on presenting features, history, and response to the
intervention. Factors such as the initial starting ‘dose’, the intensity of treatment progression, the frequency of sessions and
contact time, degree of adherence required of the patient, and delivery as monotherapy or in parallel with other interventions
may vary.”
102 Freemantle ”To avoid bias, monitoring should be conducted in confidence and the results not communicated to study staff, investigators, or
patients. Unfortunately, this does not always happen in device trials and investigator led trials. Further guidance is needed to
bring all trials of human subjects up to adequate standards for safety monitoring.”...”ICH E9 provides unusually strongly worded
guidance: “All staff involved in the conduct of the trial should remain blind to the results of such analyses, because of the
possibility that their attitudes to the trial will be modified and cause changes in the characteristics of patients to be recruited or
biases in treatment comparisons.2”...”Trials funded by commercial organisations are typically protected by confidentiality
agreements, and clinical trials units may choose not to share their standard operating procedures, making it difficult to ascertain
current practice in academic trials.”
103 Frosi ”Many outcomes were not mentioned in trial reports meaning that clinical judgment was needed to decide whether the outcome
of interest was likely to have been measured for a particular trial. A limitation of this study was that we did not contact trialists to
determine whether outcomes were measured if they were not mentioned although any uncertainties in classifications were
confirmed by contact with review authors. Our decision not to contact trialists in this study was a pragmatic one. The most recent
trial published for inclusion in this study was published over 5 years ago (median publication date 1999), meaning that there
would be obvious difficulties in locating most trialists. Nevertheless, the reliability of systematic reviews can be improved if more
attention is paid to outcome data missing from the source trial reports. If data are missing, reviewers should be encouraged to at
least attempt to contact the trialists or study sponsors to confirm whether the outcome was measured and analyzed and, if so,
obtain the results and update the review meta-analysis accordingly with the newly obtained data. Reviewers should also be
encouraged to complete the Cochrane risk of bias. A new version of the Cochrane risk of bias tool that includes a section on “bias
in selection of reported result,” which is informed by the ORBIT study and is set to be launched in 2014 [21]. If obtaining outcome
data is not feasible or successful then rather than do nothing, review authors are encouraged to apply a sensitivity analysis to
assess the impact of ORB on an individual review.”
104 Garcia ”…this review compared the results from the high quality trials to those from the low quality trials. Not surprisingly, the result
from only the low quality trials indicated a protective effect against PTB as well as low birth weight from the periodontal
treatment. They also noted publication bias that may have favoured reporting of positive trial results, particularly from smaller
studies.”
105 Ghaeminia ”The heterogeneity among studies was tested and the sensitivity analyses and Intention to Treat analyses were performed. A
quality assessment according to the Cochrane Reviewers' Handbook was performed and the authors concluded that the included
studies had unclear or high risk of bias and were therefore of poor to medium quality. However, the authors failed to describe
possible confounding of their results due to the low quality of the included studies. If bias is present in some of the studies, meta-
Page 72 of 637
analysis will simply compound the errors, and produce a ‘false result that may be interpreted as having more credibility’. The
authors failed to describe or discuss possible differences in patient characteristics at baseline of the included studies. For
example, smoking and poor oral hygiene are significant risk factors for post-operative infection and alveolitis. None of the
included studies reported these parameters, which may bias the results, especially if there was no randomisation. If,
hypothetically, there were fewer smokers in the coronectomy group compared to the extraction group, the incidence of post-
operative infection and alveolitis would be higher in the extraction group. Another important potential bias is the surgeon's skill
and experience.”...”When the primary analysis is based on all studies, we would suggest incorporating the assessments of risk of
bias into measures of the quality of evidence, for example using the GRADE system. This can help to ensure that judgments of risk
of bias, as well as other factors affecting the quality of evidence, such as imprecision and publication bias, are appropriately taken
into consideration when interpreting the results.”
106 Goodman ”Metabias poses a real challenge for comparative effectiveness research and evidence-based health care because it is typically
neither discernable nor explainable from an individual study report, but is manifest only when looking at collections of those
studies. These studies are in turn used to create treatment recommendations and practice guidelines.”…” This has led some to
explore whether industry sponsorship by itself should be considered a bias, or by our criteria, a metabias (11–13).”...”Without an
independent gold standard for the truth, one cannot even say for sure which class of studies is biased...”...”To look askance at the
findings of every single-center (or industry-funded) study may be a form of unfair “methodological profiling” because most such
studies will be innocent of bias-inducing infractions. On the other hand, if we notice such an effect, can we ignore it?
Understanding whether these findings are caused by a procedural problem in the conduct of randomized, controlled trials is
critical to knowing how these studies should be handled in evidence syntheses and to formulating remedies. This finding, if
confirmed, shows us that we have more work to do in understanding when we can rely on published medical evidence and what
to do when we cannot.”
107 Greenhalgh ”Box 1: Crisis in evidence based medicine? The evidence based “quality mark” has been misappropriated by vested interests. The
volume of evidence, especially clinical guidelines, has become unmanageable. Statistically significant benefits may be marginal in
clinical practice. Inflexible rules and technology driven prompts may produce care that is management driven rather than patient
centred.”...”Evidence based guidelines often map poorly to complex multimorbidity.”...”To support real evidence based
medicine, and in particular to reassure policy makers, clinicians, and the public that research and the guidance derived from it
can be trusted, 63 the infrastructure for research and guideline development must show the highest standards of probity.
Independent funding of national bodies for medical research is crucial.”...”In relation to producing usable evidence, we need to
identify how to balance gold standard systematic reviews with pragmatic, rapid reviews that gain in timeliness and accessibility
what they lose in depth and detail.65 In the same vein, we need research on how and in what circumstances to trade detail for
brevity in developing guidelines. We need to develop decision aids that support clinicians and patients to clarify the goals of care,
raize and answer questions about the quality and completeness of evidence, and understand and contextualize estimates of
benefit and harm. We also need to improve both the usefulness and ease of use of these and other evidence based tools
(models, scores, algorithms, and so on) including the intellectual, social, and temporal demands they make on users and the
resource implications for the healthcare organisation and system.”...”in relation to the collective effort to prevent the
misappropriation of the evidence based quality mark, a key research priority remains the study of hidden biases in sponsored
research—for example, by refining the statistical techniques for challenging findings that appear too good to be
true.”...”…evidence based medicine has not resolved the problems it set out to address (especially evidence biases and the
hidden hand of vested interests), which have become subtler and harder to detect. Furthermore, contemporary healthcare’s
complex economic, political, technological and commercial context has tended to steer the evidence based agenda towards
populations, statistics, risk, and spurious certainty.”
108 Hamm ”...there was more uncertainty surrounding identification of problems with allocation concealment, incomplete outcome data,
and “other sources of bias” (see Additional file 2). Despite this range of awareness of issues relevant to bias, 94.2% of
respondents felt confident in their ability to evaluate the quality of published trials.”...”While most survey respondents indicated
that bias is a problem, the interview data suggested that trialists often do not have the knowledge to first, recognize, and second,
address bias in their studies. They often mentioned a lack of formal training, instead relying on skills learned on the
job.”...”Addressing specific biases, survey and interview respondents reported challenges with blinding most frequently, which
included the cost of providing a placebo, difficulties in blinding non-pharmacological interventions, and blinding all relevant
parties, including parents.”...”The major barriers to minimizing risk of bias in trials were related to awareness and environment.
With little emphasis on research methodology in clinical curricula, many investigators are not adequately prepared to design
trials with high levels of internal validity or to recognize and attend to issues as they arize.”
109 Hamm ”There is a body of evidence suggesting that pediatric randomized controlled trials (RCTs) are susceptible to methodological
limitations, and a substantial proportion of the studies conducted are at a high risk of bias [5]–[14], increasing the likelihood that
(2nd treatment effects are being exaggerated. In two evaluations that assessed the overall risk of bias of pediatric RCTs using the
peripheral Cochrane Collaboration’s Risk of Bias tool, more than 90% of studies were at high or unclear risk of bias, and these trials reported
remark by larger effect estimates than studies at low risk of bias [12], [14]. Guidance on rigorous trial conduct and reporting is available in
this author) abundance, demonstrating the negative impact of design elements such as improper sequence generation, allocation
concealment, and blinding [15]–[25]; as is research on specific challenges inherent to trials in child health, for example,
recruitment and consent [26]–[30]. However, a research-practice gap persists between what is known about bias and how RCTs
are conducted, indicating a need for KT research in this population.”
110 Hansen ”There remain additional possible contributors to bias which are not currently standard components for the Cochrane risk of Bias
tool.”
111 Hartling ”The difficulties in interpreting study design labels and the consequent difficulties in reaching agreement in assigning these labels
to individual studies are consistent with those of other researchers. These issues have led some authors to direct systematic
reviewers to focus on features of designs rather than on design labels when assessing studies for inclusion and evaluating
potential risk of bias.3”...”The appropriate classification of studies by design is a critical step in a systematic review in order to
guide inclusion, risk of bias assessments, pooling of studies for analysis, interpretation of results, and grading the body of
evidence. We believe that a tool such as the one tested in this study would be useful to guide this process, although application
Page 73 of 637
of the requires several considerations in order to optimize agreement and reliability among reviewers.”...”Finally, in order to
inform this field more broadly, work is needed to quantify the bias associated with design labels and the differences between
studies that they help to identify.”
112 Hartling ”...we would recommend that trials not be excluded from SRs and/or meta-analyses based on high or unclear risk of bias
assessments. Rather, risk of bias should be explored as a potential source of heterogeneity where there is substantial variation
(2nd observed in effect estimates across studies.”...”…we found that a high proportion of our sample of trials was at high or unclear
peripheral risk of bias for many domains [...] only 3% were considered low risk of bias...”...”From an epidemiological perspective, there may
remark by be no difference in how typical biases (e.g., selection, performance, detection, attrition, reporting) operate in trials based on
this author) population characteristics.”
113 He ”The quality assessment involved whether the randomization methods, allocation concealment and blinding were adequate or
not based the study reported.”...”Many studies have showed that RCTs not using randomization, allocation concealment or
blinding exaggerate estimates of effect to various extents. Compared with the RCTs using blinding, the RCTs not using blinding
yield 17% larger estimates of treatment effects and in trials with subjective outcomes, effect estimates are exaggerated by 25%.
Compared with the RCTs using adequate allocation concealment, RCTs using unclear or inadequate concealment of allocation
exaggerate estimates of effect by 30%-41% [10-14]. These showed that compared with other ”flaws”, unclear or inadequate
allocation concealment will cause a larger bias, which highlights the importance of allocation concealment. This study indicates
that the adequate allocation concealment takes up the smallest proportion (7%) of the three assessed aspects. Although the
adequate randomization methods accounted for a larger proportion (12%) than allocation concealment, there are also some
investigations which showed that only 6.8% of the RCTs published in Chinese journals were deemed authentic randomized trials
[19]. So the quality of the TCM RCTs in this study may be overstated.”
114 Herrera ”The risk of bias was considered as medium to high. In addition, all studies had a limited sample size (from 10 to 33 patients).”
115 Hopp ”Other bias refers to any other source of bias that might be specific to the particular study design, related to fraudulent data or
other sources of bias.[2]”
116 Hróbjartsson ”We encourage extensive pretrial testing of blinding procedures and explicit reporting of who was in the blinded condition and
the methods used to ensure blinding.”...”To facilitate communication of blinding procedures in randomized clinical trials, the
traditional terminology uses the phrases “single-blind,” “double-blind,” and sometimes “triple-blind.” These terms derive from
the very early days of clinical trial development and have become deeply rooted. Unfortunately, they are ambiguous.”...”In
general, traditional blinding terminology does not serve as a means of unambiguous scientific communication and should be
abandoned, or at least complemented by an explicit reporting of who exactly was blinded.”...”Common phrases used to describe
key trial persons can be ambiguous as well (Tabl e 1). A broad category such as “investigator” is imprecize. Confusion may also
arize in trials in which the same physical person fulfills the role of health-care provider, outcome assessor, and data analyst; in
trials that rely only on patient-reported outcomes; or in trials in which the same person both adjusts dosage and assesses
outcomes. Furthermore, outcome adjudicators, i.e., those who decide whether a patient experienced an outcome of interest, are
sometimes called “judicial outcome assessors” or “secondary assessors.” These terms may be easily confused with “outcome
assessors,” that is, the persons who assess patients and provide outcome data, some-times called “data collectors” or “primary
data collectors.” Given the variable terminologies in use, any description of blinding should be very precize with regard to
whether and how all key trial persons were blinded.”
117 Innes ”Unfortunately, there were only six studies that met the inclusion criteria and their quality was not high. The main limitations
seem to be with data collection, both in the methods used and in what was collected/reported. There were no sample size
calculations nor was population representative sampling carried out. In fact, most of the studies were carried out in dental
schools, with ‘convenience sampling’. Many of the studies did not report exactly where the dental anomaly was, ie whether it
was on the cleft side or non-cleft side, and this limits interpretation of the data. In addition, the authors had to impute some of
the non-reported data, based on what was reported. One other methodological factor that has to be taken into consideration is
that the authors used a controversial approach to the meta-analysis, weighting studies rated as being at moderate risk of bias
more than those rated as being at high risk of bias.” FROM ORIGINAL ARTICLE: ”We performed a quality assessment of the
remaining studies to control for influence bias, to gain insight into potential comparisons, and to guide interpretation of findings
(Higgins and Green, 2005). Selected articles were assessed in accordance with the modified criteria of Loney et al. (1998). Seven
criteria were analyzed, and a methodologic scoring system was used to rate the quality of the papers. The authors recommended
weights for each item for the scoring system. Thus six criteria were assigned a score of 1 point. Only one was assigned a score of
3 points because it evaluated the distribution of dental anomalies according to the region or according to the arch segment/class
of teeth, making 9 the maximum score possible. After this, researchers classified the studies into three categories with scores “A”
to “C” according to predetermined criteria for method and performance. To obtain score “A,” low risk of bias, the study should
present 8 to 9 points in the methodologic scoring system; to obtain score “B,” moderate risk of bias, the study should present 5
to 7 points; and to obtain score “C,” high risk of bias and poor quality, it should present 1 to 4 points. Studies assigned the higher
scores (“A” and “B”) were weighed more heavily when the meta-analysis was performed.”
118 Jakobsen “The bias risk domains generation of allocation sequence, allocation concealment, blinding of participants and treatment
providers, blinding of outcome assessors, incomplete outcome data (see paragraph below), selective outcome reporting, and
industry funding have been shown to be of particular importance [13, 58–64].”
119 Jo ”…we found, using the van Tulder scale and the tool, that the number of high-quality articles was significantly higher for articles
that had received funding. We also found a significant difference in RCT quality for blinding, which was noted using all three
tools. This highlights the importance of using blinding to enhance the quality of an RCT.”...”The results presented here should be
interpreted within the study limitations. First, no one representative assessment tool is available for the qualitative analysis of an
RCT. Also, there is not one tool that can assess all of the items listed in the CONSORT statement. However, three representatively
used tools for quality assessment of RCTs that covered the majority of items within the CONSORT statement were used to
supplement this limitation. Second, because of the nature of the manual searching and evaluation used in this study, assessor
bias may have influenced the selection and/or assessment process. This limitation was minimized through the use of two
reviewers who independently extracted and assessed the RCTs, as well as the use of a third reviewer who moderated any
Page 74 of 637
discrepancies.”
120 Johnston ”Studies focusing on PROs often use a number of instruments to measure the same, or similar constructs. This situation creates a
risk of reporting bias. It is possible for investigators to measure a number of outcomes, and only report those that showed
significant effects. Methodologists have long suspected the existence of outcome reporting bias [21,22], and systematic
investigations comparing RCT protocols and their subsequent publications have provided estimates of its magnitude [23-25].
Investigators have examined a random sample of 156 completed Cochrane reviews that included 10 or more studies [26]. They
found that a median of 46% of the review’s eligible trials (IQR: 20 to 75%, range: 2 to 100%) contributed to the pooled estimates.
Thus, approximately half of the RCTs identified by the Cochrane reviews did not contribute to the pooled effect size in their meta-
analyses. Furthermore, they found a correlation between effect size and the number of studies included (the fewer the studies,
the larger the effect size) and this effect appeared strongest in studies using continuous outcomes (the correlation between the
percentage of trials included in a meta-analysis and the SMD was -0.18 (95% CI: -0.35 to -0.01, p = 0.04). When analyses included
less than 20% of eligible studies the mean effect size was 0.64 and when they included over 80% of the eligible studies the mean
effect size was 0.31. These results demonstrate just how frequently studies fail to provide data for meta-analyses, and provide
support for the existence of reporting bias in which investigators are inclined to selectively report results with larger effects.
Systematic reviews focusing on PROs should be alert to this problem. When only a small number of eligible studies have reported
a particular outcome, particularly if it is a salient outcome that one would expect conscientious investigators to measure,
reviewers should note the possibility of reporting bias and consider rating down confidence in estimates of effect in their
summary of findings table [27].”
121 Jørgensen ”We found that meta-analyses with non-profit or no support are of better methodological quality on average than those with
industry support. Lack of allocation concealment and blinding, and high attrition rates in randomized controlled trials may bias
results of meta-analyses, but if the authors fail to describe these details, the reader is not able to judge if the meta-analysis is
reliable. Most industry-supported meta-analyses failed on these counts; this agrees with results we have published previously
[8].”...”Industry support was defined as authorship, provision of grants to authors of the meta-analysis, or other major assistance
such as help with the statistical analysis. We did not consider provision of references or unpublished trial reports as
support.”...”Our definition of industry support does not distinguish between different amounts of support, and our judgement of
support is based on details reported in the meta-analyses. This can theoretically lead to misclassification of the support, as
industry support may range from very little to generous, and details about some types of support may be lacking more often than
others. However, the definition is operational and we believe that it includes the most important types of industry support. Lack
of details or transparency in meta-analyses may also have led to misjudgement of the methodological quality, and it has been
argued that the methodological superiority of Cochrane reviews can be explained by the fact that there are no word limits in the
Cochrane Library. However, the methodological quality of Cochrane reviews published in regular journals do not seem to differ
from Cochrane reviews published in The Cochrane Library [16,17]. Furthermore, important methodological details should always
be made available in journals with a word limit, either in the article itself, or in material on the journal's website.”
122 Kalha ”The flow diagram effectively summarizes the issue of evidence related to orthodontics with a screening of 1572 records,
assessment of 117 full text articles for eligibility with 17 studies being eventually included in a quantitative analysis. The
assessment of risk of bias is impressive and adds to the strength of the review.”
123 Kirkham ”The reliability of systematic reviews can be improved if more attention is paid to outcome data missing from the source trial
reports. If data are missing, reviewers should be encouraged to contact the trialists to confirm whether the outcome was
measured and analysed and, if so, obtain the results. If this approach is not feasible or successful, as often is the case, then rather
than do nothing, review authors are encouraged to apply a sensitivity analysis to assess the impact of outcome reporting bias on
an individual review. If the results are not robust to outcome reporting bias, the review conclusions may need to be adjusted. The
multivariate meta-analysis approach offers one such sensitivity analysis to adjust for outcome reporting bias when there is
missing trial data for many review outcomes. Our recommendation to reviewers would be to use the multivariate meta-analysis
approach if one is reasonably confident about the correlation estimates between outcomes (either from actual data, IPD from a
single study or the Pearson estimate) or use an alternative univariate adjustment approach, for example the bound for maximum
bias [6] if one is not confident about the correlations between outcomes. Where the multivariate approach is desirable but
estimates of correlation are imprecize or clinically unexpected (as the negative correlation was in the beta-lactam example), one
can consider clinical or biological reasoning to inform the correlation, or consider sensitivity analyses over a range of sensible
values.”
124 Koretz ”...none of the trials of early enteral nutrition qualified as low risk of bias. Thus, the trials were analyzed with three other
approaches: 1) Meta-analyses of trials containing at least three adequate domains (‘more robust’) were compared to meta-
analyses of trials with fewer domains being so graded (‘less robust’). 2) Trials were grouped by Jadad scores13 (e-Table 3) ≥3 and
≤2; meta-analyses were conducted for each group. 3) For each individual domain, meta-analyses of trials judged to be adequate
were compared to meta-analyses of trials judged to be not adequate (inadequate or unclear).”...”In our review, the single domain
that failed to follow the typical pattern of showing more favorable effects in trials with more bias was intention-to-treat.”...”We
required 100% followup for a judgment of adequate intention-to-treat.”...”Our use of a point scale to assess the overall risk of
bias assumes that each domain has an equivalent effect. This is probably not true and was the reason for us to look at each
domain individually. However, when there are multiple risks of bias, any attempt to demonstrate an effect of bias from only one
domain will be confounded by the other residual risks of bias diluting the effect of the single domain being considered, especially
when there are few trials in total. This insensitivity in assessing individual domains may have been why systematic error appeared
to be less influential on the results involving infectious morbidity.”
125 Kraglund ”The designation of high risk of bias was awarded because of lack of blinding in one study, incomplete outcome data in two
studies, and selective reporting of data in seven studies.”...”The authors also carried out subgroup analyses and sensitivity
analysis to ensure robust conclusions by excluding studies of high or unclear risk of bias.”
126 Krauth ”Although a sample size calculation is not a risk of bias criterion, it is an important characteristic to consider in evaluating an
overall body of evidence.”...”Although some risk of bias criteria have been investigated primarily in human studies, they warrant
consideration for animal studies. Reviews of clinical studies have shown that study funding sources and financial ties of
Page 75 of 637
investigators (including university- or industry-affiliated investigators) are associated with favorable research outcomes for the
sponsors (Lundh et al. 2011). In that study, favorable research outcomes were defined as either increased effect sizes for drug
efficacy studies, or decreased effect sizes for studies of drug harm. Selective reporting of outcomes and failure to publish entire
studies is considered an important source of bias in clinical studies; however, little is known about the extent of this bias in
animal research (Hart et al. 2012; Rising et al. 2008).”...”Further research should consider potential interactions between criteria
for assessing risk of bias. Existing instruments have tested the association of study design criteria on effect size using univariate
models. Multiple regression models should be used to ascertain the relationship between a study design criterion and effect size
when taking into account other criteria in the model. Covariance between methodological criteria should also be examined. For
example, randomized studies may be less likely to omit blinding than nonrandomized studies (van der Worp et al. 2010). Knowing
the relative importance of these criteria will provide additional support for inclusion of specific criteria in risk of bias assessment
instruments.”...”The most complex instrument had 25 criteria (Agerstrand et al. 2011).”...”As happened for clinical research,
reporting of animal research is likely to improve if risk of bias assessments become more common.”
127 Krauth ”…we recommended the use of empirically tested criteria and we pointed out criteria that have been shown to be a risk of bias.
We caution against gathering judgments on how to assess study quality and propose that evidence should guide such
(2nd evaluations. We propose an empirically based approach—as opposed to consensus-based opinion of experts—as this would
peripheral provide a more unbiased evaluation of the data.”
remark by
this author)
128 Lambert ”The risks of bias were difficult to evaluate. The overall quality of the studies was limited due to unclear risk of bias. Due to the
small number of studies, it is not possible to assess publication bias. ”
129 Langford ”We used the ‘other bias’ domain to note any additional concerns relating to study quality that did not fit into any of the previous
five domains. For example, in this domain we included concerns about recruitment bias, baseline imbalances between groups, or
selective reporting of subgroup analyses. We assessed the overall quality of the body of evidence for each outcome using the
GRADE approach (Schünemann 2011). Using this method, randomized trial evidence can be downgraded from high to moderate,
low or very low quality on the basis of five factors: limitations in design or implementation (often indicative of high risk of bias);
indirectness of evidence; unexplained heterogeneity; imprecision of results; or high probability of publication bias.”
130 Li ”The assessment of the risk of bias and its consideration in the network meta-analysis is far more challenging than in
conventional meta-analysis. Risk of bias refers to the problems with the design and execution of individual trials that raize
questions about the validity of their findings [6]. A fundamental difference between a conventional pair-wize meta-analysis and
network meta-analysis is that a conventional pair-wize meta-analysis yields only one pooled effect estimate whereas a network
meta-analysis yields more than one pooled effect estimate. Thus, while bias in the effect estimate from any single trial affects a
single pooled effect estimate in a conventional meta-analysis, it may affect several pooled effect estimates obtained in a network
meta-analysis. For example (Figure 1), the risk of bias for trials contributing to the direct comparison within a network may be
low (e.g., all A vs. C trials described adequate masking), but the risk of bias for trials contributing to the indirect comparison may
be high (e.g., some A vs. B or B vs. C trials reported no masking). In addition, the risk of bias may differ across different regions
within the network of interventions being examined. Future methodological research should address ways to deal with such
variation in risk of bias between direct and indirect comparisons and across the network. Specifically, such research may examine
the impact of risk of bias in an individual trial on the network meta-analytic effect estimates, identify the biases specific to the
network meta-analysis context that need to be considered, develop methods to assess, summarize and present the variation in
risk of bias across the network, and use empirical research to postulate guidance for network meta-analysts on incorporating bias
assessments in statistical analyses. Finally, methodological research may also examine whether network meta-analysis offers a
potential method for identifying and adjusting for biases within included trials [10,15,23].”...”Factors such as the total number of
trials in a network, number of trials with more than two comparison arms, heterogeneity (i.e., clinical, methodological, and
statistical variability within direct and indirect comparisons), inconsistency (i.e., discrepancy between direct and indirect
comparisons), and bias may influence effect estimates obtained from network meta-analyses. Heterogeneity, inconsistency, and
bias may propagate through a network of trials, and may affect the estimates differentially across regions of the network. A
range of methods to detect, quantify and deal with heterogeneity, inconsistency, and bias has been proposed [10-12,15,23].”
131 Lim ”Abstract: tools for assessing methodological quality or risk of bias in randomized controlled trials (RCTs) and non-randomized
studies (NRS) were reviewed. The van Tulder scale and Cochrane's assessment of risk of bias are the two most useful
methodological quality evaluation tools for RCTs. Cochrane's tool includes sequence generation, allocation of sequence
concealment, blinding, incomplete outcome data, selective outcome reporting, and other potential sources of bias. The Cochrane
Collaboration Group recommends the Downs and Black instrument and the Newcastle-Ottawa Scale for evaluating the quality of
NRS. In conclusion, this study offers useful information to physicians about tools for assessing the quality of evidence in clinical
guidelines. Further research is needed to provide an essential core for evidence-based decision making regarding levels and/or
grades of recommendations.”
132 Louis ”…three of them had low risk of bias.24 ,25 ,35 When pooled, studies with high risk of bias showed a benefit of IVIg in reducing
ET (3 studies, 110 neonates, RR 0.21, 95% CI 0.10 to 0.45, I2 = 0%), whereas studies with low risk of bias did not show statistically
significant difference (3 studies, 190 neonates, RR 0.82, 95% CI 0.53 to 1.26, I2 = 0%)”
133 Lutomski ”…we support that journal editorial guidelines and reviewers request data on missingness if they are not aptly discussed in an
original article.”...”As highlighted by Sterne et al,3 when incorrectly performed or when underlying assumptions are not met,
multiple imputation may in fact introduce bias into the analysis that can potentially lead to faulty conclusions.”...”Adapted from
earlier work that demonstrated the utility of causal diagrams to theoretically identify confounding,4 Daniel et al5 have
demonstrated the same theory can be applied to determine whether a complete case analysis or multiple imputation will
produce biased estimates of association.”...”…we would like to emphasize that missing data analysis requires careful
consideration to ensure the most accurate results. In this regard, causal diagrams can serve as a useful aid for choosing the
preferable method for treating missing data.”
134 Ma [Recommends the tool for especially ‘open’ randomized clinical trials.]
Page 76 of 637
135 MacLennan ”Background: In instances where randomized controlled trials (RCT) are impossible or have not been conducted, clinical
recommendations and decision-making must rely on other evidence. If systematic reviewers decide to include non-randomized
studies (NRS), it is imperative to use a standard method to assess and communicate the risk of bias (risk of bias) in NRS.
Objectives: To pilot a risk of bias tool for NRS and make it commensurate with GRADE. Methods: An extended version of the
Cochrane RCT risk of bias tool was applied to NRS. This included an additional item on the risk of findings of an NRS being
explained by confounding. Each pre-specified confounding factor was assessed on the precision of measurement, baseline
imbalance, and quality of case-mix adjustment, on 5-point scales. Imbalance was judged by clinical consensus, while other items
were assessed by two independent reviewers. Mean 'adjustment' scores per outcome across studies were used to determine the
quality of evidence according to GRADE. The tool was applied to 33 NRS retrieved for a systematic review of surgical
interventions for localized renal cancer. Results: The initial 5-point scale was unwieldy and lead to disagreement among
reviewers. We created scoring guidelines and re-piloted. Risk of bias scores were tabulated rather than aggregated to indicate
where likely biases were located. All NRS were rated as either 'low' or 'very low' on GRADE; however, determining an appropriate
cut-off required considerable judgement. Conclusions: Compared with risk of bias assessment in RCT, assessment of NRS was
more difficult and increased required time and expertize resources. In areas where the quality of studies is known to be very low,
the added time and complexity may make the assessment not worthwhile. Presentation of the large amount of information
generated by this tool is challenging. Further research needs to strike a balance of making a 'brief' and 'easy' version while
addressing complex methodological issues inherent in NRS.”
136 Matthews ”The major flaw in this review is the application of an adaptation of the Newcastle-Ottawa Scale (NOS) to determine risk-of-bias
for included trials (http://www.ohri.ca/programs/clinical_epidemiology/nosgen.pdf). Risk-of-bias assessment tools are used in
systematic reviews in a number of ways - as a threshold for inclusion of studies; as a possible explanation for differences in
results between subgroups of studies; by performing sensitivity analyses where only some of the studies are included; or by using
a qualitative score as a weight in a meta-analysis of the results. However, empirical research does not support the use of these
scales other than to describe the potential biases of each included trial. In fact, the Cochrane Collaboration, the group
responsible for the majority of published systematic reviews in health literature, advizes against the use of scales.1”
137 McCay ”By singling out industry ties, 1 the BMJ proposes to systematically discriminate against a particular group of people without
evidence based justification. This is concerning not only from an ethical perspective but also because it ignores the wider issue of
conflicts of interest, introduces bias, deters productive collaborations, marginalizes industry doctors rather than incentivising
their academic and ethical rigour, and deprives readers of relevant, diverse healthcare perspectives. Few editorials or clinical
reviews exist in a vacuum, entirely free of influence or motivation, and current conflict of interest procedures are insufficient to
understand these influences or interpret their impact. Let us improve the scope, detail, and prominence of reporting and
discussion of conflicts of interest using the principles of shared decision making to facilitate interpretation of how particular
conflicts might affect a viewpoint. Thus readers will be exposed to wider perspectives and be able to make up their own minds,
rather than have their access to the full healthcare picture restricted paternalistically.”
138 Millett ”A key question that must be asked when looking at all Cochrane reviews is whether the review has been undertaken in a way to
avoid bias and whether the data and results from the included studies are valid.10 Conclusions may be distorted if bias is
introduced to the review process at any stage. For example, incomplete searching for all relevant studies may lead to publication
bias; there is strong association between publication and studies reporting significant or positive findings in medicine11 and also
within orthodontics.12 With regard to studies included within the review, these need to be assessed to ensure that they are
undertaken and reported in a manner that is free of bias (systematic error).10 Outcome reporting bias (ORB) may also be
relevant, e.g. in a study identified as relevant to a review, where data exist on the outcome of interest but only selected findings
have been given based on the results.13 Although systematic reviews have emphasized ‘assessment of methodological quality’ of
included studies, the Cochrane handbook recommends a focus on ‘risk of bias assessment’ instead.10 The justifications are four-
fold: it targets whether the results of included studies should be credible; despite being carried out to the highest standards, a
study may have risk of bias due to, e.g. impracticality/impossibility of blinding of participants/study personnel; some quality
markers, e.g. a power calculation is unlikely to directly affect risk of bias;14 it overcomes ambiguity between quality of reporting
and quality of the underlying research.10”...”There are many tools available; most are scales where quality components are
scored and combined to give a summary score; or checklists of specific questions.15 As it is impossible to know the extent or true
risk of bias in a given study, possible validation of any proposed tool is limited. The Cochrane-Collaboration recommends neither
of the aforementioned methods; rather a domain-based evaluation (Yes/No/Unclear) is used where Yes indicates a ‘low risk of
bias’ for the following six components: sequence generation, allocation concealment, blinding of participants/personnel and
outcome assessors, incomplete outcome data, selective outcome reporting and other sources of bias.10,16 The issue of bias
linked to poor randomization and blinding has been discussed extensively but the discussions about ORB are relatively
new.16,17”
139 Moher ”Strong evidence of outcome reporting bias was recently reported within clinical trials [23,24]. Our results suggest that some
aspect of selective outcome reporting bias might also exist within non-Cochrane reviews. Only about one-quarter of them
reported a primary outcome, of which half report statistical significance in favour of this outcome (versus 14.4% for Cochrane
reviews). This issue requires further investigation.”
140 Morrison ”We found no evidence of systematic bias from the use of language restrictions in SRs/MAs in conventional medicine. There were
conflicting findings about the methodological and reporting quality of English-language versus languages other than English trials.
These findings do not rule out the potential for language bias when language restrictions are used. Searches should include
languages other than English studies when resources and time are available to minimize the risk of a biased summary effect.
More research, in different medical specialties, will provide better evidence on the effect of language restriction on systematic
reviews.”
141 Mullan ”To help reduce reporting bias, and potentially allow for exploration about the impact of author contact on bias and the results of
systematic reviews, journals and the Cochrane Collaboration should implement reporting recommendations for author contact in
systematic reviews.”...”Author contact adds to the burden of conducting a systematic review. However, it can improve the quality
of the review by: (1) confirming the accuracy of the extracted data; and (2) overcoming the data-driven reporting choices that
authors and journals may have made. Lack of reporting or incomplete reporting of an important outcome de facto excludes an
Page 77 of 637
eligible study from analysis, a contributor to reporting bias if the outcome was in fact measured. Reporting bias can lead to
overestimation of the treatment effect [3], and Chan and Altman have found that more than 20% of outcomes measured in
parallel group trials were incompletely reported [2]. When are the benefits worth the costs and burden of contacting authors?
Arguably, reviewers should contact authors when the review can yield stronger inferences as a result of this process. The extent
to which reviewers can make this determination a priori remains uncertain and may represent an area of fruitful methodological
investigation.”...”The inclusion of reporting recommendations for author contact in systematic reviews and the enforcement of
these requirements by journals and the Cochrane Collaboration may allow for methodological explorations about the impact of
author contact on bias and the results of systematic reviews. Essential elements to report include: the number of studies for
which authors were contacted, the information requested, any important details of the method of eliciting information, and the
response of authors to the request. When pertinent, authors should report the impact that information obtained from authors
had on review results (i.e., using sensitivity analyses).”
142 Murad ”Ideally, systematic reviewers will evaluate and report the risk of bias for each of the important outcomes measured in each
individual study. There is no one correct way to assess the risk of bias.26 Review authors can use detailed checklists or focus on a
few key aspects of the study. Different study designs require the use of different instruments (eg, for randomized clinical trials,
the Cochrane Risk of Bias tool27). A judgment about the overall risk of bias for all of the included studies may then result in
decreasing the confidence in estimates.5”
143 Murad ”Although this systematic review has accounted for selection, performance, detection and attrition bias for all the identified
studies, grave weakness is still there. As with many systematic reviews, the limited number of well-conducted RCT might reduce
(2nd the effectiveness of this review in its ability to provide valuable scientific evidence to support or contribute to changes in our
peripheral conventional clinical practice. Although efforts were made to avoid language bias and to examine the reference lists, no attempt
remark by was made to hand-search any journals and the only attempt to identify unpublished trials was made by contacting investigators
this author) of the included study.”...”The major weaknesses of this systematic review is that the result is derived from the inclusion of one
sole industry sponsored trial, which failed to provide sufficient information about the randomisation procedure. This trial could
therefore be considered to fall into the category of poor reporting. The assumption of inadequate randomisation method or bias
interference generally indicates inadequate methods.”
144 Naci ”While some aspects of clinician–industry interactions may be beneficial, the normalization of such relationships in clinical
settings creates the potential for serious risks for patients and health care systems. Yet it may be unrealistic to expect that
clinicians can be taught individually how to interact with industry ethically or to detect and avert bias. Social science researchers
suggest that the rational choice view of conflict of interest does not reflect the evidence, arguing that judgments are subject to a
“self-serving” bias that is both unconscious and unintentional [50]. The problem of self-serving bias suggests that clinician
education will not be effective in mitigating unconscious biases, nor will disclosure be an effective means to counteract biases
[50]. Further, even clinicians who consciously seek to avoid interactions with industry may fail because of the ubiquitous nature
of marketing and promotional materials [51] and the strength of practice and social norms [26]. Although education alone may
be ineffective, the ethical implications of such interactions could be problematized for clinicians during professional and
continuing education, and issues such as the introduction of bias into clinical decision-making could be addressed at an
institutional or regulatory level. ”
145 Nankervis ”…only 5% of the recent eczema trials were registered correctly and with enough detail to assess outcome-reporting bias for the
primary outcome.”...”The fact that some investigators chose to preregister their trial could be an indicator of trial quality, and
this was explored using the four key domains known to be associated with a high risk of bias (Higgins and Altman, 2008). In this
sample of eczema RCTs, there was a suggestion that trial quality might be improved in registered trials, but this was only
significant for the domain of allocation concealment, and is possibly limited by the modest sample size of our survey.”
146 Palys ”The authors are correct that the Jadad score, whether with 13, 11, or 5 points (as is most common), is efficient.1 That is to say
that a score for trial quality can be whipped up with little effort. However, would it not be even more efficient to simply toss a
coin and award full marks if that coin lands face up?”...” Flawed studies invite biases that distort findings, generally towards more
favorable ones that suit the preferences of the experimenters and all of their conflicts of interest. When these studies go on to
inform medical decisions, patients are denied the basic right of informed decision making because they are basing their decisions
on information that is not only misleading, but also known to be misleading. The emperor's new clothes must finally be exposed
for what they are, and the emperor's new clothing inspectors are part of the problem, and not part of the solution. We need new
inspectors, ones who will actually inspect the clothing instead of simply tossing a coin (or, equivalently, using the Jadad score).
Only then can we expect better clothes (or medical studies, as the case may be).”
147 Pearson ”While the rigour of studies included in public health systematic reviews is routinely assessed using a quality appraisal tool,1–3
with the exception of the Cochrane risk of bias tool,1 none assesses potential bias to a subsequent review from missing or
incompletely reported results. ”...”The aims of this paper were to demonstrate the use of a formal tool (Outcome Reporting Bias
in Trials (ORBIT)10) to appraize studies included in a systematic review of public health interventions for the existence of
outcome reporting bias and to assess the impact that this bias (where present) had on the evidence that was used to inform the
development of guidance.”...”The aims of this paper were to demonstrate the use of a formal tool (Outcome Reporting Bias in
Trials (ORBIT)10) to appraize studies included in a systematic review of public health interventions for the existence of outcome
reporting bias and to assess the impact that this bias (where present) had on the evidence that was used to inform the
development of guidance. ”...”Retrospective assessment of outcome reporting bias as part of critical appraisal tools used in
public health18 19 is also possible but will require development of a tool such as ORBIT to be more suitable for the study designs
and reporting found in evaluations of public health interventions.”
148 Pildal ”Most conclusions favouring an intervention would lose support if trials with unclear or inadequate allocation concealment were
excluded from the meta-analysis. This may seem too radical, especially since the bias associated with these trials appears to be
smaller and less consistent than previously thought. Furthermore, the remaining trials might still be affected by other sources of
bias, for instance selective reporting of significant outcomes.89 Yet, results of meta-analyses should always be accompanied by
sensitivity analyses presenting the results with and without trials with unclear or inadequate bias protection. While sensitivity
analyses will allow the reader to gauge the possible impact of bias, decisions still have to be made whether or not the
investigated interventions should be implemented. To guide such decisions, further research on the size and direction of different
Page 78 of 637
types of bias under different circumstances is warranted. In addition, steps to prevent bias and avoid uncertainty regarding the
level of bias protection should be taken. First, the gatekeepers of trial protocols (primarily drug-regulatory authorities and
research ethics committees) should insist on description of methods to ensure allocation concealment and sanction only
protocols with adequate methods. Secondly, trial protocols should be publicly available to facilitate critical appraisal of trials and
thirdly, the CONSORT statement, 91 which requires explicit and appropriate reporting on measures taken to protect a trial
against bias, should be broadly enforced.”...”Two thirds of conclusions drawn from meta-analyses loose support if only trials with
reported adequate allocation concealment are relied upon. The impact of reported allocation concealment and double-blinding
on the treatment effect estimate is smaller and less consistent than previously thought. It would be too radical to routinely only
rely on trials with reported adequate allocation concealment.”
149 Poulsen ”The experimental design of the trials also differed: nine of the 12 trials used the split-mouth design and the remaining three
used a parallel-group design. The methodological problems associated with the split-mouth design have been reviewed recently,
1 and the possibility of a carry-over effect from one side of the mouth to the other has been pointed out as an important risk-of-
bias. To what extent this factor has been taken into account in the trials is not reported.”
150 Purgato ”25 scales that had been used to assess the validity or ‘quality’ of randomized trials. These scales included between 3 and 57
items and rating scales were likely to include criteria that did not directly relate to study validity (Cipriani et al., 2009). The
Cochrane Collaboration recommends simple approaches for assessing methodological quality using a specific tool for evaluating
the risk of bias. This tool – developed between 2005 and 2007 by a working group of methodologists – is a domain-based
evaluation.”...”Theoretically, it is expected high coherence between the quality of reporting and the methodological quality. In
practice, it is possible to see clinical trials with a high methodological quality that are poorly reported, but also high quality study
reports of clinical trials with a poor methodological quality.”...”Even though to assess quality of RCTs is a complex issue, in order
to have a rough idea of the quality of a RCT we recommend to have a careful look at the table reporting the baseline clinical and
sociodemographic characteristics of the patients allocated to the new and reference treatment. It is possible to check how many
patients were truly randomized (a small sample size generally correlates with low study quality) and whether there are
imbalances in the randomization procedures that may be suggestive of high risk of bias.”
151 Rasines ”For the assessment of risk of bias of the cross-sectional surveys with retrospective assessment of exposure, the authors
developed their own criteria. None of the studies filled all the considered criteria.”
152 Reeves ”Non-randomized studies vary in their design features (Reeves et al., 2008; Higgins et al., 2012). These features make the studies
more or less susceptible to bias. In NRS, compared with in RCTs, attrition is often worse (and poorly reported); intervention and
outcome assessments are less likely to be conducted according to standardized protocols, and outcomes are rarely assessed blind
to the group allocation. Too often, these limitations of NRS are seen as part of doing an NRS with the consequence that their
implications for risk of bias to a study (Higgins and Altman, 2008), and the way in which limitations vary across studies, are not
properly considered. For example, some users of evidence may consider NRS that investigate long-term outcomes to have ‘better
quality’ than randomized trials of short-term outcomes, simply on the basis of their relevance without appraising their risk of
bias. Schünemann et al. (2013) describe how such a judgement can be made more systematically.”...”The Cochrane Collaboration
has typically sought to summarize reliable evidence rather than the best available evidence (i.e. with an absolute threshold rather
than a conditional rule for inclusion). Should it continue to do this? In what circumstances should NRS be included in Cochrane
reviews? How acceptable is it for this decision to depend on (i) the anticipated needs of certain stakeholders; (ii) knowledge of
how much randomized evidence exists; and (iii) the suspected magnitude of effect size? Should a common tool be used to assess
the risk of bias in RCTs and non-randomized studies? What sort of tool would allow differentiation in risks of bias: (i) between
RCTs and non-randomized studies, and (ii) between different types of NRS? How should review authors draw distinctions
between different types of non-randomized evidence, particularly in relation to determining eligibility for the review? Are design
labels (e.g. cohort study) or design features (e.g. prospective identification of participants) preferable? When should review
authors be encouraged to use different study designs to assess unintended (e.g. adverse) effects compared with intended
effects?”
153 Reveiz “Overall, the proportion of adequate reporting varied by methodologic item from the Cochrane Collaboration Risk of Bias tool:
random sequence generation (weighted proportion 5.7%, 95% CI 3.0–8.4%), allocation concealment (1.4%, 0–2.8%), blinding
(41%, 35–47% including open label RCTs; 8.4%, 4.1–13% excluding open label RCTs), primary outcomes (66%, 60–72%), secondary
outcomes (46%,40–52%) and harms outcomes (5%, 2–8%) (Table S1). Weighted proportions were calculated using data from
Table 2. Most records reported no useful information for allocation concealment (97.9%) and harm (89.5%), and had insufficient
detail for blinding (86.2%, excluding open label RCTs) and primary outcome measures (32%).”
154 Richards ”While the authors reported the assessment of risk of as low for seven of the included studies in the discussion, they indicate
that if they had included allocation concealment (a key component of bias protection) only one of the studies would be
considered to have a low risk of bias. While I agree that this may be a reporting issue rather than a study conduct it does form an
important element of assessing potential bias.”
155 Rubinstein ”...only in the last period (2010 ) did more than half the studies have a low risk of bias, namely, 68%.”...”…performance bias is but
one construct of methodological quality, and studies that do not “blind” their patients or practitioners may still be considered to
have a low risk of bias even if other constructs are not entirely conducted well.”...”Blinding (or masking) of subjects in trials of
nonpharmacological interventions remains a problem, and SMT is certainly no exception. It is questionable to what degree
blinding is relevant. Although adequate blinding is absolutely necessary when comparing an intervention to a sham treatment,
this is much less relevant for pragmatic studies where experimental interventions are typically compared with standard
therapies, such as “usual care.” – “Therein also belies a problem because it assumes that all criteria weigh equally, and one could
argue that randomization, for example, specifically treatment allocation, is perhaps the most important feature of trials.12”
156 Runnels ”It was also clear from responses to the survey questions that respondents, a majority of whom had expertize in systematic
review methodology, recognized the many limitations to the quality of evidence in primary studies, systematic reviews, and
meta-analyses. These perspectives were evident in comments about poor quality protocols; the lack of transparency in clinical
trial design; inadequate reporting of outcomes, including adverse events; in the challenges of conducting sub-group analyses; and
in addressing the many forms of methods bias in appraisal of evidence. As Ioannidis and others have shown, seemingly rigorous
methods have considerable strengths, “but can also lead to wrong or misleading answers” [54], p. 169]. For example, the
Page 79 of 637
outcomes of meta-analyses on the same topic may differ widely due to many factors, including that those conducting/sponsoring
the meta-analyses may choose different data sources, search strategies, inclusion/exclusion criteria for eligible populations, and
have different, sometimes conflicting interests that may influence how results are presented and interpreted [54].”
157 Saini ”The classification system used in this study has been presented and applied during a workshop that we developed and delivered
at international Cochrane colloquiums. The feedback from this workshop supported the practical application of our classification
system, and many participants were able to relate their own experiences to the types of scenarios that are captured in the
classification. Following the application of the classification system, the Cochrane risk of bias tool is currently being updated to
include the assessment of bias in both randomized controlled trials and non-randomized studies. The proposed new structure of
the risk of bias tool considers selective outcome reporting as being analogous to publication bias (non-reporting of whole
studies). It is planned that this form of bias will be appraized outside the risk of bias tool…”
158 Santaguida “When assessing limitations in studies of medical tests, systematic reviewers should select validated criteria that examine the risk
of systematic error. Systematic reviewers should categorize the risk of bias for individual studies as “low,” “medium,” or “high.”
Two reviewers should independently assess individual criteria as well as global categorization. Reviewers should establish
methods for determining an overall categorization for the study limitations a priori and document these decisions clearly.”
159 Savović ”Bias associated with specific reported study design characteristics of RCTs leads to exaggeration of intervention effect estimates
and increases in between-trial heterogeneity. For each of the three characteristics assessed, these effects appeared greatest for
subjectively assessed outcome measures. Assessments of the risk of bias in trial results should account for these findings.
Downweighting trials at high risk of bias in future meta-analyses, based on these empirical findings, could be an alternative to
completely excluding such trials from meta-analyses, resulting in a smaller loss of precision.”
160 Sequeira- “We suggest that the use of a tool such as AMSTAR in the pre-publication and editorial appraisal of systematic reviews would
Byron help to ensure that those that are published are of high quality and are reported clearly, completely and in such a way that this
quality is readily apparent.”
161 Shadrick ”Cochrane reviews are well known for their methodological rigour and minimisation of bias and this one certainly doesn't
disappoint. It describes in comprehensive detail the methodology used and design of the review. The Cochrane Collaboration
statistical guidelines were adhered to for data analysis with heterogeneity and bias also thoroughly assessed, although
insufficient studies were identified to investigate reporting bias. Considerable effort was undertaken to contact the authors of
the studies to confirm details missing from the methodology of the trials. The lack of long-term data and small sample sizes made
it difficult for the authors to draw conclusions other than the need for more high quality long-term research into this area.”
162 Sidney ”Risk of Bias: Unclear Items: Trends over time for studies scoring items as “unclear” are depicted in Figure 4. In more than half of
all studies, 3 items (ie, treatment allocation, selective reporting, and cointerventions) were rated unclear. For many items, there
appears to be a trend toward better reporting over time, which is most pronounced for items related to selection
bias.”...”Nevertheless, performance bias is but one construct of methodological quality, and studies that do not “blind” their
patients or practitioners may still be considered to have a low risk of bias even if other constructs are not entirely conducted
well.”
163 Simon ”Establish the blind is one thing, but maintaining throughout the follow-up is another. For this, we will sometimes use false
adaptations doses. Such adaptations will be centralized. In this case, it is possible to generate false biological results. Using an
assessor blinded endpoint is always useful (assessor distinguished therapist). If the assessor is not blind, the bias induced
wherever endpoint is now well documented. [8] It will therefore avoid contact between Therapist and the non-blind evaluator.
When necessary, the patient will be informed that he must not speak with the evaluator. In addition, it is helpful to use a
centralized evaluation further investigations. In all cases, unblinding procedure should be described in the protocol, especially in
emergency situations.”...”Entering a final question: should assess the success of the blind? In reality, this assessment of the blind
be criticized. [9] Often evaluation focuses more on blind prejudices about the effectiveness of treatment on the blind itself. In
addition, the repetition of these evaluations will effectively focus the participants on blind issues. Finally, being able to guess
treatment (not certain) is not equivalent to the fact of knowing for sure: this methodological flaw probably not enough to bring
the test at an open trial. This is why the 2010 version of Consort statement abandoned blind evaluation of methods [4] (although
the latter was explicitly mentioned in recommendation 11 of the 2001 version). Nevertheless, it must accurately describe the
method for the blind. Most of the time, these methods are insufficiently described, either in the article or in the protocol of [10]
study. For our part, we do the same finding in preeclampsia prevention trials (personal data).”
164 Smaïl- ”...Cochrane review authors describe clinical implications only after describing the quality of evidence and the balance of benefits
Faugero and harms.”
165 Sola ”Background: The Cochrane Handbook claims that the evaluation of the validity of included studies is essential for the analysis,
interpretation and conclusions of systematic reviews. Therefore, reviews should include not only a risk of bias assessment but
also should discuss how the limitations in the design and execution could affect the validity of their conclusions. Objectives: To
evaluate to what extent the risk of bias assessment is properly considered to formulate the conclusions in set of published
systematic reviews. Methods: Two reviewers will independently assess a set of systematic reviews used to develop a practice
guideline on the surgical management of femoral fractures. We will use AMSTAR (Shea 2007), paying special attention to the
relevant items that address whether the reviews documented any formal quality assessment (item 7), and if this assessment was
used accurately to formulate the reviews' conclusions (item 8). We will estimate an agreement coefficient between these two
items, but will also qualitatively check the extent to which the quality of information was used to draw conclusions in the
reviews. Results: So far, we have assessed 15 systematic reviews (5 Cochrane vs. 10 non Cochrane). Although 9 of these reviews
properly considered the limits in design and execution of the included trials to formulate their conclusions, the rest (6 reviews,
40%) did not assess the quality of the included studies or did not use the quality assessment to discuss its impact on the review
results. At this moment we are increasing the sample of assessed reviews to accurately estimate if these preliminary results could
be considered significant. Conclusions: There still exists clear room for improvement in the way that the quality assessment of
included studies is used to draw conclusions in systematic reviews. This issue is of special relevance for the usefulness of reviews
in the process of knowledge translation, given the importance of bias assessment for allowing an appropriate evaluation of the
confidence in the estimates derived from reviews.”
Page 80 of 637
166 Song ”Exclusion of non-English language studies appears to result in a particularly high risk of bias in some areas of research such as
complementary and alternative medicine.”...”Risk of bias was marked as ‘high’ if the efforts taken to minimize publication bias
were partial or insufficient, publication bias was not discussed and the authors’ conclusions were positive. Risk of bias was
‘moderate’ if partial efforts were taken to minimize bias, publication bias was probably considered, and the author’s conclusions
might have been positive with cautious interpretation. Risk of bias was ‘low’ if partial or sufficient efforts were taken to minimize
bias.”...”Risk of bias was marked as ‘high’ if the efforts taken to minimize publication bias were partial or insufficient, publication
bias was not discussed and the authors’ conclusions were positive. Risk of bias was ‘moderate’ if partial efforts were taken to
minimize bias, publication bias was probably considered, and the author’s conclusions might have been positive with cautious
interpretation. Risk of bias was ‘low’ if partial or sufficient efforts were taken to minimize bias.”
167 Souza ”Asymmetrical in funnel plots are linked to publication bias although there are other sources of asymmetry that have to be
considered, including other dissemination biases, differences in the quality of smaller studies, the existence of true
heterogeneity, and chance. Asymmetry in funnel plots may be an indicator that a more detailed investigation should be carried
out on the presence of heterogeneity, such as sensitivity analysis.”
168 Spuls ”Systematic reviews should not be restricted to English-language publications, to prevent bias.”
169 Szczesniak ”Studies of various quality levels can either mask or have an adverse effect. The quality of included studies, for example the study
of Xu et al., is low and could have biased the results, as reported in the Cochrane Collaboration guidelines.2 The authors should
have evaluated the methodology more accurately; they declare no publication bias in their funnel plot for mortality; at a closer
inspection, however, a marked asymmetry in the lower left-hand corner is evident, suggesting markedly biased results.”...”…we
believe the methodology employed in the article biased the results, which are against the current evidence of literature.
Therefore, the findings of this meta-analysis should be interpreted with strong caution.”
170 Tendal ”Meta-analyses of randomized clinical trials are crucial for making evidence based decisions. However, trial reports often present
the same data in multiple forms when reporting different intervention groups, time points, and outcome measures.1 Although
this multiplicity has always been a challenge in meta-analyses, its potential as a source of bias has received little
attention.”...”Multiplicity of data in trial reports might lead to biased decisions about which data to include in meta-analyses and
hence threaten the validity of their results. In this study, we empirically assessed whether selecting between multiple time
points, scales, and treatment groups affected SMD results in a randomly selected sample of Cochrane reviews.”...”A typical
statement, which allowed for a potentially biased choice regarding the selection of a time point, was: “All outcomes were
reported for the short term (up to 12 weeks), medium term (13 to 26 weeks), and long term (more than 26 weeks).”7 ”
171 Thombs ”Studies of the accuracy of depression screening tools rarely exclude already diagnosed or treated patients, a potential bias that
is not evaluated in systematic reviews and meta-analyses. This may result in inflated accuracy estimates on which clinical practice
and preventive care guidelines are often based, a problem that takes on greater importance as the rate of diagnosed and treated
depression in the population increases.”
172 Thombs “Studies of the accuracy of screening tools for depression rarely exclude patients who already have a diagnosis or are receiving
treatment, a potential bias that is not evaluated in systematic reviews and meta-analyses. This can result in inflated accuracy and
(2nd estimates of the yield of new cases on which clinical practice and preventive care guidelines are often based, a problem that
peripheral takes on greater importance as the rate of diagnosed and treated depression in the population increases.”
remark by
this author)
173 Tricco ”...the impact of this potential bias can be addressed through sensitivity analyses, which explores whether results of the review
are robust to differences in the trials, such as methodology (eg, examining studies with and without allocation concealment
analyzed separately) and populations examined.”...”Reviewers should consider both study- and review-level limitations. If the
conduct or reporting of included studies is poor, the review conclusions may be biased and this should be stated explicitly.
Furthermore, knowledge syntheses themselves can be susceptible to bias.”...”Although much attention has been paid to
enhancing the quality of systematic reviews, relatively little attention has been paid to the format for presenting the review.
Because the reporting of systematic reviews tends to focus on methodological rigor more than clinical context, they often do not
provide crucial information for clinicians. In one study, the researchers found that of systematic reviews published in ACP JC and
EBM Journal (journals of secondary publication), less than 15% of these had sufficient information about the intervention to
allow clinicians or policy makers to implement it [78].”
174 Tricco ”We identified SRs examining several biases, yet further investigation into the following is warranted: 1) place of publication bias,
2) country bias, 3) search bias, 4) citation bias, 5) multiple publication bias, and 6) outcome reporting bias (a SR is planned; Dr. P.
(2nd Williamson, personal communication). The SRs themselves should be updated, as new evidence may have emerged. Although not
peripheral a mandate of this review, we believe it is important to explore whether common SR practices do in fact decrease bias, such as
remark by having two people independently screen potentially relevant material and scanning the reference lists of the included studies in a
this author) SR.”...”Our findings recommend including unpublished material in SRs, updating SRs periodically, searching more than one
database, hand searching for additional material, using the Cochrane HSSS to locate RCT reports, and assessing for publication
bias. Further examination of the other types of bias identified in our SR is warranted and the existing empirical evidence should
be systematically reviewed.”
175 Tugwell ”When is the optimal time to introduce new health care technologies into routine care? Many clinical epidemiologists and
approval agencies will suggest recommending their introduction only after consistent evidence of benefit is found from two
pragmatic, well-designed, and well-executed randomized controlled trials (RCTs).”...”Integrating global research evidence with
locally relevant evidence and contextual factors is rarely a transparent process.”...”Surrogate markers are very attractive as
endpoints in trials – they have been popular since blood pressure was shown to predict morbidity and mortality, and a reduction
in blood pressure was shown to reduce the death and cardiovascular morbidity. They are so much easier to measure, the trials so
much easier to power, and hence so much cheaper. However, their indiscriminate use can cause enormous harm.”...”Response
shift is the focus of an interesting study by Galenkamp et al. Why does self-rated health not decline as we get older? What could
explain why ‘‘older olds'’ have been reported to rate their health more positively than ‘‘younger olds.’’ Given similar levels of
limitations and chronic conditions, the authors examined 3 different components known to contribute to response shift
Page 81 of 637
(reprioritization, reconceptualization, and recalibration). They conclude that recalibration is probably the most responsible. To
assess and to evaluate possible effects arising from web-based data collection on the results of a study, Mayr et al sought to
assess and evaluate possible effects arising from selection bias and additional response bias. They did find a selection bias as their
sample was found to be better educated, more often living in a partnership, more often female, and older than the general
population. They also found a substantial response bias as participants using the internet were younger, better educated, and
more often male compared with participants preferring the paper-and-pencil version. Despite this bias, the method of data
collection had no direct effect on the results of various self-report instruments after adjusting for the 3 characteristics defining
the response bias.”
176 Turner ”Review methods: assessing risk of bias: The risk of bias assessment within included studies varied considerably across the
samples. For example, although 28% (37/131) of CAM SRs and 17% (30/175) of control SRs used the Cochrane Risk of Bias tool
[19], 83% of CAM reviews used a tool identified as relatively less frequently used [20] (e.g., MINORS [21], Downs and Black [22],
Zaza [23]). Self-developed tools were used in 4% (5/131) of CAM reviews and 11% (19/175) of control SRs. Of the CAM reviews,
19/131 reviews used more than one tool (Table 3).”...”Risk of bias assessment within included studies varied considerably across
the samples; 28% of CAM SRs and 17% of control SRs used the Cochrane Risk of Bias tool [19]. These findings are consistent with
other research [27]. Moreover, 83% of CAM reviews used less prominent tools and self-developed tools were used in 4% of CAM
reviews and 11% of control SRs. There are a substantial number of methods used to assess the quality of primary studies in both
samples of SRs. This is consistent with previous research which reported of 177 reviews, 38% defined a method of quality
assessment, within which 74 different methodological items and 26 different scales were identified [21].”
177 Turner ”In our analyses, we have modelled total between-study heterogeneity, which is likely to comprize a mixture of variation caused
by true diversity among the protocols for the original studies, variation caused by biases and unexplained variation. Assuming
(2nd that a conventional random-effects model will be used in many future meta-analyses, it is appropriate to focus on total between-
peripheral study heterogeneity in our predictive findings. However, it would be preferable to separate variation attributable to biases from
remark by other sources of between-study variation. In later versions of the CDSR, this will become possible once the recently introduced
this author) Cochrane risk-of-bias tool21 has been implemented in a large number of systematic reviews. Our existing hierarchical model for
the data from all available meta-analyses could be extended to incorporate the bias model proposed by Welton et al.22 This
would allow us to adjust for the bias attributable to a potential source (e.g. inadequate allocation concealment) in all studies
judged to be at high risk. In principle, the model could be extended further to adjust for multiple sources of bias simultaneously.
Results from this analysis could provide useful information about the degree to which one would expect between-study
heterogeneity to reduce, on average, if meta-analysts chose to adjust for known sources of bias, for example, by using empirical
evidence or elicited opinion on biases.22,23”
178 Turner ”When at least two adequately powered studies are available in meta-analyses reported by Cochrane reviews, underpowered
studies often contribute little information, and could be left out if a rapid review of the evidence is required. However,
(3rd underpowered studies made up the entirety of the evidence in most Cochrane reviews.”
peripheral
remark by
this author)
179 Van Driel ”Our study shows that searching for unpublished trials in Cochrane reviews does not give a high yield and that the
methodological quality of unpublished trials raizes concern. We found that only 11.9% of all recent Cochrane reviews included
references to unpublished data. In an earlier literature review of 150 meta-analyses indexed in MEDLINE (1988–1999), 30.7%
included unpublished data in their analysis [8]. Egger et al. report that only 7% of all trials included in their sample of 159 meta-
analyses were unpublished [10] The attention on publication bias in the past decade and recommendations to search for
unpublished material to minimize the risk of publication bias [9], [16] and [17] apparently have not resulted in more frequent
inclusion of unpublished data in Cochrane reviews. Egger et al. found that funnel plot asymmetry, as a proxy of publication bias,
was reduced when unpublished trials were included, but excluding them from the meta-analysis had only relatively small effects
on the estimates of treatment and their precision [10]. However, the funnel plot may not be the most reliable method to detect
publication bias [18].”...”Our review shows that less than 10% of Cochrane reviews include unpublished data and references to
unpublished studies make up only a small proportion of all included studies. The fact that a third of these “unpublished”
references could be located as journal publications suggests that not including them in the review before formal publication
would merely delay the evidence synthesis. The poor methodological transparency and quality of the trials that remain
unpublished is an important concern for the validity of the reviews. Cochrane reviewers should be aware of this issue and
perhaps they could add to the transparency of their review by performing sensitivity analyses of trial quality. Some articles of
high quality may remain unpublished because they show negative or indifferent results [40]. However, extensive searching has
not uncovered these trial reports in Cochrane Reviews and reviewers need to weigh the importance of searching for (additional)
unpublished studies in each individual review against the risk of introducing bias. It may be better to invest in improving
(reporting of) methodological quality and regular updating of existing reviews that include good quality trials than in extensive
searches for unpublished data.”
180 Viswanathan ”While we were able to identify a set of questions for the most common observational study designs, we were not able to
establish consensus on required items for each type of design, as we had originally intended. Whether our suggested approach
applies to other quasi-experimental designs such as controlled clinical trials or pre-post studies of public health interventions also
requires empirical assessment. More work is required to establish consensus on type of study designs and specific sources of bias
for each design.”
181 Vollenweider ”We believe that a way forward would be not only to inform investigators what ought to be carried out and reported (as is laid
out in the CONSORT statement) but also to have a better explanation of how the features of trial design help reduce bias. A
clinical trial is no different from any epidemiological study; the primary concern should be minimization of confounding, selection
bias and measurement error that lead to information bias. There seems to be too little awareness of the problems of selection
bias and missing data. A number of studies have found that many trials do not report or use inappropriate methods to conceal
the random allocation [38]. Similarly, we found that many trials do not report on the handling of missing data and ways to deal
with it. It is difficult to quantify the bias that results from different ways of (not) dealing with missing data because individual
Page 82 of 637
patient data from many different trials would be needed to investigate this. To improve reporting on missing data, editors and
reviewers should require investigators to follow the CONSORT statement and report on their efforts to minimize bias from
missing data and to report when they were unable to do so explicitly.”
182 Wong ”the findings from this review should be interpreted with caution, because the methodologic assessment of the quality of the
included studies showed an unclear risk of bias in five out of the ten included papers, and a high risk of bias in four studies.”
183 Yavchitz ”However, despite these initiatives, the quality of reporting of abstracts remains questionable [9, 10, 22–24]. A recent study
showed that despite systematic reviews including primary studies with high risk of bias, just over half included a risk of bias
assessment in the interpretation of results in the abstract [13]. Consequently, adding a limitations section could be useful to
enhance readers’ awareness and improve their interpretation. However, a limitations section in the abstract is recommended by
only a few journals and for systematic reviews in the PRISMA statement for abstracts [12].”
References
1. Armijo-Olivo, Susan, Maria Ospina, Bruno R. da Costa, Matthias Egger, Humam Saltaji, Jorge Fuentes, Christine Ha,
and Greta G. Cummings. “Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When
Applying the Cochrane Risk of Bias tool in Physical Therapy Trials.” PloS One 9, no. 5 (2014): e96920.
doi:10.1371/journal.pone.0096920.
2. Bero, Lisa A. “Why the Cochrane Risk of Bias tool Should Include Funding Source as a Standard Item.” The Cochrane
Database of Systematic Reviews 12 (2013): ED000075.
3. Boutron, Isabelle, and Philippe Ravaud. “Classification Systems to Improve Assessment of Risk of Bias.” Journal of
Clinical Epidemiology 65, no. 3 (March 2012): 236–38. doi:10.1016/j.jclinepi.2011.09.006.
4. De Bruin, Marijn. “Risk of Bias in Randomized Controlled Trials of Health Behaviour Change Interventions: Evidence,
Practices and Challenges.” Psychology & Health 30, no. 1 (January 2, 2015): 1–7. doi:10.1080/08870446.2014.960653.
5. Hartling, Lisa, Michele P. Hamm, Andrea Milne, Ben Vandermeer, P. Lina Santaguida, Mohammed Ansari, Alexander
Tsertsvadze, Susanne Hempel, Paul Shekelle, and Donna M. Dryden. “Testing the Risk of Bias tool Showed Low
Reliability between Individual Reviewers and across Consensus Assessments of Reviewer Pairs.” Journal of Clinical
Epidemiology 66, no. 9 (September 2013): 973–81. doi:10.1016/j.jclinepi.2012.07.005.
6. Hróbjartsson, Asbjørn, Isabelle Boutron, Lucy Turner, Douglas G. Altman, and David Moher. “Assessing Risk of Bias in
Randomized Clinical Trials Included in Cochrane Reviews: The Why Is Easy, the How Is a Challenge.” The Cochrane
Database of Systematic Reviews 4 (2013): ED000058. doi:10.1002/14651858.ED000058.
7. Ivers, Noah M, Andrea C Tricco, Monica Taljaard, Ilana Halperin, Lucy Turner, David Moher, and Jeremy M Grimshaw.
“Quality Improvement Needed in Quality Improvement Randomized Trials: Systematic Review of Interventions to
Improve Care in Diabetes.” BMJ Open 3, no. 4 (April 9, 2013). doi:10.1136/bmjopen-2013-002727.
8. Jefferson, Tom, Mark A Jones, Peter Doshi, Chris B Del Mar, Rokuro Hama, Matthew J Thompson, Igho Onakpoya, and
Carl J Heneghan. “Risk of Bias in Industry-Funded Oseltamivir Trials: Comparison of Core Reports versus Full Clinical
Study Reports.” BMJ Open 4, no. 9 (September 30, 2014). doi:10.1136/bmjopen-2014-005253.
9. Katikireddi, Srinivasa Vittal, Matt Egan, and Mark Petticrew. “How Do Systematic Reviews Incorporate Risk of Bias
Assessments into the Synthesis of Evidence? A Methodological Study.” Journal of Epidemiology and Community
Health 69, no. 2 (February 1, 2015): 189–95. doi:10.1136/jech-2014-204711.
10. Morissette, Kate, Andrea C Tricco, Tanya Horsley, Maggie H Chen, and David Moher. “Blinded versus Unblinded
Assessments of Risk of Bias in Studies Included in a Systematic Review.” In Cochrane Database of Systematic Reviews.
John Wiley & Sons, Ltd, 1996. http://onlinelibrary.wiley.com/doi/10.1002/14651858.MR000025.pub2/abstract.
11. Moustgaard, Helene, Segun Bello, Franklin G. Miller, and Asbjørn Hróbjartsson. “Subjective and Objective Outcomes
in Randomized Clinical Trials: Definitions Differed in Methods Publications and Were Often Absent from Trial
Reports.” Journal of Clinical Epidemiology 67, no. 12 (December 2014): 1327–34. doi:10.1016/j.jclinepi.2014.06.020.
12. Roseman, M., E. H. Turner, J. Lexchin, J. C. Coyne, L. A. Bero, and B. D. Thombs. “Reporting of Conflicts of Interest
from Drug Trials in Cochrane Reviews: Cross Sectional Study.” BMJ 345, no. aug16 3 (August 16, 2012): e5155–e5155.
doi:10.1136/bmj.e5155.
13. Savović, Jelena, Laura Weeks, Jonathan A. C. Sterne, Lucy Turner, Douglas G. Altman, David Moher, and Julian P. T.
Higgins. “Evaluation of the Cochrane Collaboration’s tool for Assessing the Risk of Bias in Randomized Trials: Focus
Groups, Online Survey, Proposed Recommendations and Their Implementation.” Systematic Reviews 3 (2014): 37.
doi:10.1186/2046-4053-3-37.
Page 83 of 637
14. Sterne, Jonathan A. C. “Why the Cochrane Risk of Bias tool Should Not Include Funding Source as a Standard Item.”
The Cochrane Database of Systematic Reviews 12 (2013): ED000076.
15. Vale, Claire L., Jayne F. Tierney, and Sarah Burdett. “Can Trial Quality Be Reliably Assessed from Published Reports of
Cancer Trials: Evaluation of Risk of Bias Assessments in Systematic Reviews.” BMJ (Clinical Research Ed.) 346 (2013):
f1798.
16. Abou-Setta A, Dryden D, Hamm M, Moher D, Klassen T, Hartling L. Assessment of trial risk of bias among Cochrane
reviews: a cross-sectional analysis. Poster presentation at the 19th Cochrane Colloquium; 2011 Oct 19-22; Madrid,
Spain. Cochrane Database of Systematic Reviews, Supplemen.
17. Akl, Elie A., Xin Sun, Jason W. Busse, Bradley C. Johnston, Matthias Briel, Sohail Mulla, John J. You, et al. “Specific
Instructions for Estimating Unclearly Reported Blinding Status in Randomized Trials Were Reliable and Valid.” Journal
of Clinical Epidemiology 65, no. 3 (March 2012): 262–67. doi:10.1016/j.jclinepi.2011.04.015.
18. Anette Blümle, Erik von Elm, Gerd Antes, Joerg J. Meerpohl. Measurement and assessment of study quality and
reporting quality. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen Volume 108, Issues 8–9,
2014, Pages 495–503
19. Armijo-Olivo, Susan, Carla R. Stiles, Neil A. Hagen, Patricia D. Biondo, and Greta G. Cummings. “Assessment of Study
Quality for Systematic Reviews: A Comparison of the Cochrane Collaboration Risk of Bias tool and the Effective Public
Health Practice Project Quality Assessment tool: Methodological Research.” Journal of Evaluation in Clinical Practice
18, no. 1 (February 2012): 12–18. doi:10.1111/j.1365-2753.2010.01516.x.
20. Armijo-Olivo, Susan, Greta G. Cummings, Jorge Fuentes, Humam Saltaji, Christine Ha, Annabritt Chisholm, Dion
Pasichnyk, and Todd Rogers. “Identifying Items to Assess Methodological Quality in Physical Therapy Trials: A Factor
Analysis.” Physical Therapy 94, no. 9 (September 1, 2014): 1272–84. doi:10.2522/ptj.20130464.
21. Armijo-Olivo, Susan, Jorge Fuentes, Maria Ospina, Humam Saltaji, and Lisa Hartling. “Inconsistency in the Items
Included in tools Used in General Health Research and Physical Therapy to Evaluate the Methodological Quality of
Randomized Controlled Trials: A Descriptive Analysis.” BMC Medical Research Methodology 13 (September 17, 2013):
116. doi:10.1186/1471-2288-13-116.
22. Armijo-Olivo, Susan, Jorge Fuentes, Todd Rogers, Lisa Hartling, Humam Saltaji, and Greta G Cummings. “How Should
We Evaluate the Risk of Bias of Physical Therapy Trials?: A Psychometric and Meta-Epidemiological Approach towards
Developing Guidelines for the Design, Conduct, and Reporting of RCTs in Physical Therapy (PT) Area: A Study
Protocol.” Systematic Reviews 2 (September 26, 2013): 88. doi:10.1186/2046-4053-2-88.
23. Armstrong, Rebecca, Elizabeth Waters, Maureen Dobbins, John N. Lavis, Mark Petticrew, and Robin Christensen.
“Knowledge Translation Strategies for Facilitating Evidence-Informed Public Health Decision Making among Managers
and Policy-Makers.” In Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd, 1996.
http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD009181/abstract.
24. Bafeta, Aïda, Agnes Dechartres, Ludovic Trinquart, Amélie Yavchitz, Isabelle Boutron, and Philippe Ravaud. “Impact of
Single Centre Status on Estimates of Intervention Effects in Trials with Continuous Outcomes: Meta-Epidemiological
Study.” BMJ : British Medical Journal 344 (2012). doi:10.1136/bmj.e813.
25. Barkhordarian, Andre, Peter Pellionisz, Mona Dousti, Vivian Lam, Lauren Gleason, Mahsa Dousti, Josemar Moura, and
Francesco Chiappelli. “Assessment of Risk of Bias in Translational Science.” Journal of Translational Medicine 11
(August 8, 2013): 184. doi:10.1186/1479-5876-11-184.
26. Berkman, Nancy D., P. Lina Santaguida, Meera Viswanathan, and Sally C. Morton. “The Empirical Evidence of Bias in
Trials Measuring Treatment Differences,” September 2014. http://www.ncbi.nlm.nih.gov/books/NBK253183/.
27. Bero, Lisa. “Bias Related to Funding Source in Statin Trials.” BMJ (Clinical Research Ed.) 349 (2014): g5949.
28. Buchberger, Dr Barbara, E. von Elm, G. Gartlehner, H. Huppertz, G. Antes, J. Wasem, and J. J. Meerpohl. “Bewertung
des Risikos für Bias in kontrollierten Studien.” Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz
57, no. 12 (December 1, 2014): 1432–38. doi:10.1007/s00103-014-2065-6.
29. Cho, Hee Ju, Jae Hoon Chung, Jung Ki Jo, Dong Hyuk Kang, Jeong Man Cho, Tag Keun Yoo, and Seung Wook Lee.
“Assessments of the Quality of Randomized Controlled Trials Published in International Journal of Urology from 1994
to 2011.” International Journal of Urology 20, no. 12 (2013): 1212–19. doi:10.1111/iju.12150.
30. Chung, Vincent CH, Robin ST Ho, Xinyin Wu, Daisy HY Fung, Xin Lai, Justin CW Wu, and Samuel YS Wong. “Are Meta-
Analyses of Chinese Herbal Medicine Trials Trustworthy and Clinically Applicable? A Cross-Sectional Study.” Journal of
Page 84 of 637
Ethnopharmacology 162 (March 13, 2015): 47–54. doi:10.1016/j.jep.2014.12.028.
31. Crocetti, Michael T., Diane D. Amin, and Roberta Scherer. “Assessment of Risk of Bias Among Pediatric Randomized
Controlled Trials.” Pediatrics 126, no. 2 (August 1, 2010): 298–305. doi:10.1542/peds.2009-3121.
32. Da Costa, Bruno R., Nina M. Resta, Brooke Beckett, Nicholas Israel-Stahre, Alison Diaz, Bradley C. Johnston, Matthias
Egger, Peter Jüni, and Susan Armijo-Olivo. “Effect of Standardized Training on the Reliability of the Cochrane Risk of
Bias Assessment tool: A Study Protocol.” Systematic Reviews 3 (2014): 144. doi:10.1186/2046-4053-3-144.
33. De Bruin, Marijn, Jim McCambridge, and Jan M. Prins. “Reducing the Risk of Bias in Health Behaviour Change Trials:
Improving Trial Design, Reporting or Bias Assessment Criteria? A Review and Case Study.” Psychology & Health 30, no.
1 (January 2, 2015): 8–34. doi:10.1080/08870446.2014.953531.
34. Dechartres A, Altman DG, Trinquart L, Boutron I, and Ravaud P. “Association between Analytic Strategy and Estimates
of Treatment Outcomes in Meta-Analyses.” JAMA 312, no. 6 (August 13, 2014): 623–30. doi:10.1001/jama.2014.8166.
35. Faggion, Clovis M., Fahd Huda, and Jason Wasiak. “Use of Methodological tools for Assessing the Quality of Studies in
Periodontology and Implant Dentistry: A Systematic Review.” Journal of Clinical Periodontology 41, no. 6 (2014): 625–
31. doi:10.1111/jcpe.12251.
36. Gordon H. Guyatt, Andrew D. Oxman, Gunn Vist, Regina Kunz, Jan Brozek, Pablo Alonso-Coello, Victor Montori, Elie A.
Akl, Ben Djulbegovic, Yngve Falck-Ytter, Susan L. Norris, John W. Williams Jr., David Atkins, Joerg Meerpohl, Holger J.
Schünemann, GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias), Journal of Clinical
Epidemiology, Volume 64, Issue 4, April 2011, Pages 407-415, ISSN 0895-4356,
http://dx.doi.org/10.1016/j.jclinepi.2010.07.017.
37. Hartling L, Bialy L, Armijo-Olivo S, Ha C, Lacaze-Masmonteil T, Vandermeer B, Dryden DM. Modifications to the Risk of
Bias tool: a case study of 204 trials. Oral presentation at the 19th Cochrane Colloquium; 2011 Oct 19-22; Madrid,
Spain. Cochrane Database of Systematic Reviews, Supplement. CMR-16534.
38. Hartling, L., Bond, K., Vandermeer, B., Seida, J., Dryden, D. M., & Rowe, B. H. (2011). Applying the Risk of Bias tool in a
Systematic Review of Combination Long-Acting Beta-Agonists and Inhaled Corticosteroids for Persistent Asthma. PLoS
ONE, 6(2), e17242. doi:10.1371/journal.pone.0017242
39. Hartling, L., M. Hamm, T. Klassen, A.-W. Chan, M. Meremikwu, V. Moyer, S. Scott, D. Moher, M. Offringa, and for the
StaR Child Health Group. “Standard 2: Containing Risk of Bias.” PEDIATRICS 129, no. Supplement (June 1, 2012): S124–
31. doi:10.1542/peds.2012-0055E.
40. Hartling, Lisa, Maria Ospina, Yuanyuan Liang, Donna M Dryden, Nicola Hooton, Jennifer Krebs Seida, and Terry P
Klassen. “Risk of Bias versus Quality Assessment of Randomized Controlled Trials: Cross Sectional Study.” BMJ : British
Medical Journal 339 (2009). doi:10.1136/bmj.b4012.
41. Hayden, Jill A., Danielle A. van der Windt, Jennifer L. Cartwright, Pierre Côté, and Claire Bombardier. “Assessing Bias in
Studies of Prognostic Factors.” Annals of Internal Medicine 158, no. 4 (February 19, 2013): 280–86. doi:10.7326/0003-
4819-158-4-201302190-00009.
42. Hempel, Susanne, Jeremy NV Miles, Marika J Booth, Zhen Wang, Sally C Morton, and Paul G Shekelle. “Risk of Bias: A
Simulation Study of Power to Detect Study-Level Moderator Effects in Meta-Analysis.” Systematic Reviews 2
(November 28, 2013): 107. doi:10.1186/2046-4053-2-107.
43. Higgins, Julian P T, Douglas G Altman, Peter C Gøtzsche, Peter Jüni, David Moher, Andrew D Oxman, Jelena Savović,
Kenneth F Schulz, Laura Weeks, and Jonathan A C Sterne. “The Cochrane Collaboration’s tool for Assessing Risk of Bias
in Randomized Trials.” BMJ : British Medical Journal 343 (2011). doi:10.1136/bmj.d5928.
44. Hopewell, Sally, Isabelle Boutron, Douglas G. Altman, and Philippe Ravaud. “Incorporation of Assessments of Risk of
Bias of Primary Studies in Systematic Reviews of Randomized Trials: A Cross-Sectional Study.” BMJ Open 3, no. 8
(August 1, 2013): e003342. doi:10.1136/bmjopen-2013-003342.
45. Ijaz, Sharea, Jos H. Verbeek, Christina Mischke, and Jani Ruotsalainen. “Inclusion of Nonrandomized Studies in
Cochrane Systematic Reviews Was Found to Be in Need of Improvement.” Journal of Clinical Epidemiology 67, no. 6
(June 2014): 645–53. doi:10.1016/j.jclinepi.2014.01.001.
46. Jo, Jung Ki, Jae Hoon Chung, Kyu Shik Kim, Jeong Woo Lee, Seung Wook Lee, and International Evidence-Based
Medicine Research (IEMR) Group. “Randomized Controlled Trials in the Journal of Sexual Medicine: A Quality
Assessment and Relevant Clinical Impact.” The Journal of Sexual Medicine 11, no. 4 (2014): 894–900.
doi:10.1111/jsm.12455.
Page 85 of 637
47. Johnson, Blair T., Robert E. Low, and Hayley V. MacDonald. “Panning for the Gold in Health Research: Incorporating
Studies’ Methodological Quality in Meta-Analysis.” Psychology & Health 30, no. 1 (January 2, 2015): 135–52.
doi:10.1080/08870446.2014.953533.
48. Kirkham, J. J, K. M Dwan, D. G Altman, C. Gamble, S. Dodd, R. Smyth, and P. R Williamson. “The Impact of Outcome
Reporting Bias in Randomized Controlled Trials on a Cohort of Systematic Reviews.” BMJ 340, no. feb15 1 (February
15, 2010): c365–c365. doi:10.1136/bmj.c365.
49. Kirkham, Jamie J., Doug G. Altman, and Paula R. Williamson. “Bias Due to Changes in Specified Outcomes during the
Systematic Review Process.” PLoS ONE 5, no. 3 (March 22, 2010): e9810. doi:10.1371/journal.pone.0009810.
50. Lee, Jeong Woo, Jae Hoon Chung, Jung Ki Jo, and Seung Wook Lee. “Analysis of Randomized Controlled Trials in
Rheumatology International from 1981 to 2012: Methodological Assessment.” Rheumatology International 34, no. 9
(September 1, 2014): 1187–93. doi:10.1007/s00296-014-2963-9.
51. Liu, Yali, Shengping Yang, Junjie Dai, Yongteng Xu, Rui Zhang, Huaili Jiang, Xianxia Yan, and Kehu Yang. “Risk of Bias
tool in Systematic Reviews/Meta-Analyses of Acupuncture in Chinese Journals.” PLoS ONE 6, no. 12 (December 9,
2011). doi:10.1371/journal.pone.0028130.
52. Lundh, Andreas, and Peter C Gøtzsche. “Recommendations by Cochrane Review Groups for Assessment of the Risk of
Bias in Studies.” BMC Medical Research Methodology 8 (April 21, 2008): 22. doi:10.1186/1471-2288-8-22.
53. Lundh, Andreas, Sergio Sismondo, Joel Lexchin, Octavian A Busuioc, and Lisa Bero. “Industry Sponsorship and
Research Outcome.” In Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd, 1996.
http://onlinelibrary.wiley.com/doi/10.1002/14651858.MR000033.pub2/abstract.
54. Manchikanti, Laxmaiah, Joshua A. Hirsch, Steven P. Cohen, James E. Heavner, Frank J. E. Falco, Sudhir Diwan, Mark V.
Boswell, et al. “Assessment of Methodologic Quality of Randomized Trials of Interventional Techniques: Development
of an Interventional Pain Management Specific Instrument.” Pain Physician 17, no. 3 (June 2014): E263–90.
55. O’Connor, Annette M., and Jan M. Sargeant. “Critical Appraisal of Studies Using Laboratory Animal Models.” ILAR
Journal 55, no. 3 (January 1, 2014): 405–17. doi:10.1093/ilar/ilu038.
56. Oremus, Mark, Carolina Oremus, Geoffrey B. C. Hall, and Margaret C. McKinnon. “Inter-Rater and Test–retest
Reliability of Quality Assessments by Novice Student Raters Using the Jadad and Newcastle–Ottawa Scales.” BMJ
Open 2, no. 4 (January 1, 2012): e001368. doi:10.1136/bmjopen-2012-001368.
57. Robertson C, Ramsay C, Gurung T, Mowatt G, Pickard R, Sharma P. Assessing the quality of non-randomized
comparative studies: our experience of using the Cochrane Collaboration's risk of bias tool. Poster presentation at the
19th Cochrane Colloquium; 2011 Oct 19-22; Madrid, Spain. Cochrane Database of Systematic Reviews, Supplement.
58. Robinson, Karen A, Evelyn P Whitlock, Maya E Oneil, Johanna K Anderson, Lisa Hartling, Donna M Dryden, Mary
Butler, et al. “Integration of Existing Systematic Reviews into New Reviews: Identification of Guidance Needs.”
Systematic Reviews 3 (June 23, 2014): 60. doi:10.1186/2046-4053-3-60.
59. Roseman M, Milette K, Bero LA, and et al. “REporting of Conflicts of Interest in Meta-Analyses of Trials of
Pharmacological Treatments.” JAMA 305, no. 10 (March 9, 2011): 1008–17. doi:10.1001/jama.2011.257.
60. Sargeant, J. M., and A. M. O’Connor. “Conducting Systematic Reviews of Intervention Questions II: Relevance
Screening, Data Extraction, Assessing Risk of Bias, Presenting the Results and Interpreting the Findings.” Zoonoses and
Public Health 61 (2014): 39–51. doi:10.1111/zph.12124.
61. Savovic J, Altman D, Higgins J, Moher D, Sterne J, Turner L, Weeks L. Risk of bias tool evaluation: process and results.
Oral presentation at the Joint Cochrane and Campbell Colloquium; 2010 Oct 18-22; Keystone, Colorado, USA.
Database of Systematic Reviews, Supplement
62. Shamliyan, Tatyana, Robert L. Kane, and Stacy Dickinson. “A Systematic Review of tools Used to Assess the Quality of
Observational Studies That Examine Incidence or Prevalence and Risk Factors for Dizeases.” Journal of Clinical
Epidemiology 63, no. 10 (October 2010): 1061–70. doi:10.1016/j.jclinepi.2010.04.014.
63. Shrier I.
http://methods.cochrane.org/sites/methods.cochrane.org/files/Why%20Obs%20and%20DAG%20and%20Meta-
analysis.pdf
64. Sinha, Yashwant K., Jonathan C. Craig, Premala Sureshkumar, Andrew Hayen, and Jo-anne E. Brien. “Risk of Bias in
Randomized Trials of Pharmacological Interventions in Children and Adults.” The Journal of Pediatrics 165, no. 2
(August 2014): 367–71.e1. doi:10.1016/j.jpeds.2014.03.058.
Page 86 of 637
65. Turner, Lucy, Isabelle Boutron, Asbjørn Hróbjartsson, Douglas G Altman, and David Moher. “The Evolution of
Assessing Bias in Cochrane Systematic Reviews of Interventions: Celebrating Methodological Contributions of the
Cochrane Collaboration.” Systematic Reviews 2 (September 23, 2013): 79. doi:10.1186/2046-4053-2-79.
66. Voss, Peer H., and Eva A. Rehfuess. “Quality Appraisal in Systematic Reviews of Public Health Interventions: An
Empirical Study on the Impact of Choice of tool on Meta-Analysis.” Journal of Epidemiology and Community Health
67, no. 1 (January 1, 2013): 98–104. doi:10.1136/jech-2011-200940.
67. Wells, George A, Beverley Shea, Julian PT Higgins, Jonathan Sterne, Peter Tugwell, and Barnaby C Reeves. “Checklists
of Methodological Issues for Review Authors to Consider When Including Non-Randomized Studies in Systematic
Reviews.” Research Synthesis Methods 4, no. 1 (2013): 63–77. doi:10.1002/jrsm.1077.
68. Zeng, Xiantao, Yonggang Zhang, Joey S.W. Kwong, Chao Zhang, Sheng Li, Feng Sun, Yu-Ming Niu, and Liang Du. “The
Methodological Quality Assessment tools for Pre-Clinical and Clinical Studies, Systematic Review and Meta-Analysis,
and Clinical Practice Guideline: A Systematic Review.” Journal of Evidence-Based Medicine, 2015, n/a – n/a.
doi:10.1111/jebm.12141.
69. Bala, Malgorzata M., Elie A. Akl, Xin Sun, Dirk Bassler, Dominik Mertz, Filip Mejza, Per Olav Vandvik, et al.
“Randomized Trials Published in Higher vs. Lower Impact Journals Differ in Design, Conduct, and Analysis.” Journal of
Clinical Epidemiology 66, no. 3 (March 2013): 286–95. doi:10.1016/j.jclinepi.2012.10.005.
70. Balk, Ethan M., Mei Chung, Nira Hadar, Kamal Patel, Winifred W. Yu, Thomas A. Trikalinos, and Lina Kong Win Chang.
Accuracy of Data Extraction of Non-English Language Trials with Google Translate. AHRQ Methods for Effective Health
Care. Rockville (MD): Agency for Healthcare Research and Quality (US), 2012.
http://www.ncbi.nlm.nih.gov/books/NBK95238/.
71. Bassler, Dirk, Victor M. Montori, Matthias Briel, Paul Glasziou, Stephen D. Walter, Tim Ramsay, and Gordon Guyatt.
“Reflections on Meta-Analyses Involving Trials Stopped Early for Benefit: Is There a Problem and If So, What Is It?”
Statistical Methods in Medical Research 22, no. 2 (April 1, 2013): 159–68. doi:10.1177/0962280211432211.
72. Berger, Vance W. “Internal Validity and the Risk of Bias: A Case for a Comprehensive Review.” Journal of Anesthesia
26, no. 5 (June 1, 2012): 802–3. doi:10.1007/s00540-012-1420-8.
73. Berger, Vance W., and Anh-Chi Do. “Allocation Concealment Continues to Be Misunderstood.” Journal of Clinical
Epidemiology 63, no. 4 (April 1, 2010): 468–69. doi:10.1016/j.jclinepi.2009.09.004.
74. Bialy, Liza, Ben Vandermeer, Thierry Lacaze-Masmonteil, Donna M. Dryden, and Lisa Hartling. “A Meta-
Epidemiological Study to Examine the Association between Bias and Treatment Effects in Neonatal Trials.” Evidence-
Based Child Health: A Cochrane Review Journal 9, no. 4 (2014): 1052–59. doi:10.1002/ebch.1985.
75. Chaimani et al. Effects of study precision and risk of bias in networks of interventions: a network meta-
epidemiological study. meta-epidemiological study 2013
76. Chase, Jonathan M. “The Shadow of Bias.” PLoS Biology 11, no. 7 (July 2013). doi:10.1371/journal.pbio.1001608.
77. Chess, Laura E, and Joel Gagnier. “Risk of Bias of Randomized Controlled Trials Published in Orthopaedic Journals.”
BMC Medical Research Methodology 13 (June 9, 2013): 76. doi:10.1186/1471-2288-13-76.
78. Chiappelli, Francesco, André Barkhordarian C Phil, Rashi Arora, Linda Phi, Amy Giroux, Molly Uyeda, Jason Kung, and
Manisha Ramchandani. “Reliability of Quality Assessments in Research Synthesis: Securing the Highest Quality
Bioinformation for HIT.” Bioinformation 8, no. 14 (July 21, 2012): 691–94. doi:10.6026/97320630008691.
79. Christensen R. Is Phytalgic® a goldmine for osteoarthritis patients or is there something fishy about this nutraceutical?
A summary of findings and risk-of-bias assessment. Arthritis Research & Therapy 2010, 12:105
80. Chung, Jae Hoon, Dong Hyuk Kang, Jung Ki Jo, and Seung Wook Lee. “Assessing the Quality of Randomized Controlled
Trials Published in the Journal of Korean Medical Science from 1986 to 2011.” Journal of Korean Medical Science 27,
no. 9 (September 2012): 973–80. doi:10.3346/jkms.2012.27.9.973.
81. Chung, Mei, Ethan M Balk, Stanley Ip, Gowri Raman, Winifred W Yu, Thomas A Trikalinos, Alice H Lichtenstein,
Elizabeth A Yetley, and Joseph Lau. “Reporting of Systematic Reviews of Micronutrients and Health: A Critical
appraisal1234.” The American Journal of Clinical Nutrition 89, no. 4 (April 2009): 1099–1113.
doi:10.3945/ajcn.2008.26821.
82. Da Costa, Bruno R., Roger Hilfiker, and Matthias Egger. “PEDro’s Bias: Summary Quality Scores Should Not Be Used in
Meta-Analysis.” Journal of Clinical Epidemiology 66, no. 1 (January 1, 2013): 75–77.
doi:10.1016/j.jclinepi.2012.08.003.
Page 87 of 637
83. Davey, Jonathan, Rebecca M Turner, Mike J Clarke, and Julian PT Higgins. “Characteristics of Meta-Analyses and Their
Component Studies in the Cochrane Database of Systematic Reviews: A Cross-Sectional, Descriptive Analysis.” BMC
Medical Research Methodology 11 (November 24, 2011): 160. doi:10.1186/1471-2288-11-160.
84. Dechartres, Agnes, Isabelle Boutron, Ludovic Trinquart, Pierre Charles, and Philippe Ravaud. “Single-Center Trials
Show Larger Treatment Effects Than Multicenter Trials: Evidence From a Meta-Epidemiologic Study.” Annals of
Internal Medicine 155, no. 1 (July 5, 2011): 39–51. doi:10.7326/0003-4819-155-1-201107050-00006.
85. Doi, Suhail A. R., and Jan J. Barendregt. “Not PEDro’s Bias: Summary Quality Scores Can Be Used in Meta-Analysis.”
Journal of Clinical Epidemiology 66, no. 8 (August 2013): 940–41. doi:10.1016/j.jclinepi.2013.03.001.
86. Dossing, Anna, Simon Tarp, Daniel E Furst, Christian Gluud, Joseph Beyene, Bjarke B Hansen, Henning Bliddal, and
Robin Christensen. “Interpreting Trial Results Following Use of Different Intention-to-Treat Approaches for Preventing
Attrition Bias: A Meta-Epidemiological Study Protocol.” BMJ Open 4, no. 9 (September 25, 2014).
doi:10.1136/bmjopen-2014-005297.
87. Duclos, P., D. N. Durrheim, A. L. Reingold, Z. A. Bhutta, K. Vannice, and H. Rees. “Developing Evidence-Based
Immunization Recommendations and GRADE.” Vaccine 31, no. 1 (December 17, 2012): 12–19.
doi:10.1016/j.vaccine.2012.02.041.
88. Duvendack, Maren, Jorge Garcia Hombrados, Richard Palmer-Jones, and Hugh Waddington. “Assessing ‘what Works’
in International Development: Meta-Analysis for Sophisticated Dummies.” Journal of Development Effectiveness 4,
no. 3 (September 1, 2012): 456–71. doi:10.1080/19439342.2012.710642.
89. Dwan, Kerry, Douglas G. Altman, Juan A. Arnaiz, Jill Bloom, An-Wen Chan, Eugenia Cronin, Evelyne Decullier, et al.
“Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias.” PLoS ONE 3, no.
8 (August 28, 2008). doi:10.1371/journal.pone.0003081.
90. Dwan, Kerry, Douglas G. Altman, Mike Clarke, Carrol Gamble, Julian P. T. Higgins, Jonathan A. C. Sterne, Paula R.
Williamson, and Jamie J. Kirkham. “Evidence for the Selective Reporting of Analyses and Discrepancies in Clinical
Trials: A Systematic Review of Cohort Studies of Clinical Trials.” PLoS Medicine 11, no. 6 (June 24, 2014).
doi:10.1371/journal.pmed.1001666.
91. Elkhadem A. Question: How effective is fluoride toothpaste in preventing dental caries in the primary dentition of
preschool children? Dos Santos AP, Nadanovsky P, de Oliveira BH. A systematic review and meta-analysis of the
effects of fluoride toothpastes on the prevention of dental caries in the primary dentition of preschool children.
Community Dent Oral Epidemiol 2012; doi: 10.1111/j.1600-0528.2012.00708.x. [Epub ahead of print] PubMed PMID:
22882502.
92. ElKhadem A. Question: When bonding orthodontic appliances are 1-step bonding approaches more effective than 2-
step approaches? Fleming PS, Johal A, Pandis N. Self-etch primers and conventional acid-etch technique for
orthodontic bonding: a systematic review and meta-analysis. Am J Orthod Dentofacial Orthop. 2012; 142: 83–94.
93. Engebretson S. Question: Does non-surgical periodontal treatment improve glycaemic control in diabetic patients?
Corbella S, Francetti L, Taschieri S, De Siena F, Fabbro MD. Effect of periodontal treatment on glycemic control of
patients with diabetes: A systematic review and meta-analysis. J Diabetes Investig 2013; 4: 502–509.
94. Eriksen, P., Bartels, E. M., Altman, R. D., Bliddal, H., Juhl, C. and Christensen, R. (2014), Risk of Bias and Brand Explain
the Observed Inconsistency in Trials on Glucosamine for Symptomatic Relief of Osteoarthritis: A Meta-Analysis of
Placebo-Controlled Trials. Arthritis Care Res, 66: 1844–1855. doi: 10.1002/acr.22376
95. Faggion, Clovis Mariano, Nikolaos Nikitas Giannakopoulos, and Stefan Listl. “Risk of Bias of Animal Studies on
Regenerative Procedures for Periodontal and Peri-Implant Bone Defects – a Systematic Review.” Journal of Clinical
Periodontology 38, no. 12 (2011): 1154–60. doi:10.1111/j.1600-051X.2011.01783.x.
96. Christiane Alves FerreiraI; Carlos Alfredo Salles LoureiroII; Humberto SaconatoIII; Álvaro Nagib AtallahIV. Assessing
the risk of bias in randomized controlled trials in the field of dentistry indexed in the Lilacs (Literatura Latino-
Americana e do Caribe em Ciências da Saúde) database. Sao Paulo Med. J. vol.129 no.2 São Paulo Mar. 2011
97. Fleming, Padhraig S., Jadbinder Seehra, Argy Polychronopoulou, Zbys Fedorowicz, and Nikolaos Pandis. “Cochrane
and Non-Cochrane Systematic Reviews in Leading Orthodontic Journals: A Quality Paradigm?” The European Journal
of Orthodontics, April 16, 2012, cjs016. doi:10.1093/ejo/cjs016.
98. Flodgren G, Parmelli E, Doumit G, Gattellari M, O’Brien MA, Grimshaw J, Eccles MP. Local opinion leaders: effects on
professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2011, Issue 8. Art. No.:
CD000125. DOI: 10.1002/14651858.CD000125.pub4.
Page 88 of 637
99. Flores-Mir C. Question: How effective are orthodontic appliances in distalising upper first molars in children and
adolescents? Jambi S, Thiruvenkatachari B, O'Brien KD, Walsh T. Orthodontic treatment for distalising upper first
molars in children and adolescents. Cochrane Database Syst Rev 2013; 10: Art. No.: CD008375. DOI:
10.1002/14651858.CD008375.pub2.
100. Forbes, Dorothy. “Blinding: An Essential Component in Decreasing Risk of Bias in Experimental Designs.” Evidence
Based Nursing 16, no. 3 (July 1, 2013): 70–71. doi:10.1136/eb-2013-101382.
101. Foster, Nadine, and Paul Little. “Methodological Issues in Pragmatic Trials of Complex Interventions in Primary Care.”
The British Journal of General Practice 62, no. 594 (January 2012): 10–11. doi:10.3399/bjgp12X616238.
102. Freemantle, N., and F. V. d. Werf. “Keeping Patients Safe While Avoiding Bias in Randomized Trials.” BMJ 344, no.
may14 1 (May 14, 2012): e3297–e3297. doi:10.1136/bmj.e3297.
103. Frosi, Giacomo, Richard D. Riley, Paula R. Williamson, and Jamie J. Kirkham. “Multivariate Meta-Analysis Helps
Examine the Impact of Outcome Reporting Bias in Cochrane Rheumatoid Arthritis Reviews.” Journal of Clinical
Epidemiology. Accessed March 1, 2015. doi:10.1016/j.jclinepi.2014.11.017.
104. Garcia R. Question: Does the treatment of periodontitis and gingivitis during pregnancy reduce the number of
preterm births? Polyzos NP, Polyzos IP, Zavos A, Valachis A, Mauri D, Papanikolaou EG, Tzioras S, Weber D, Messinis
IE. Obstetric outcomes after treatment of periodontal dizease during pregnancy: systematic review and meta-
analysis. BMJ 2010; 341: c701
105. Ghaeminia H. Question: In patients with impacted third molars does coronectomy when compared to conventional
removal result in fewer complications? Long H, Zhou Y, Liao L, Pyakurel U, Wang Y, Lai W. Coronectomy vs. total
removal for third molar extraction: a systematic review. J Dent Res 2012; 91: 659–665.
106. Goodman, Steven, and Kay Dickersin. “Metabias: A Challenge for Comparative Effectiveness Research.” Annals of
Internal Medicine 155, no. 1 (July 5, 2011): 61–62. doi:10.7326/0003-4819-155-1-201107050-00010.
107. Greenhalgh, T., J. Howick, N. Maskrey, and for the Evidence Based Medicine Renaissance Group. “Evidence Based
Medicine: A Movement in Crisis?” BMJ 348, no. jun13 4 (June 13, 2014): g3725–g3725. doi:10.1136/bmj.g3725.
108. Hamm, Michele P, Shannon D Scott, Terry P Klassen, David Moher, and Lisa Hartling. “Do Health Care Institutions
Value Research? A Mixed Methods Study of Barriers and Facilitators to Methodological Rigor in Pediatric Randomized
Trials.” BMC Medical Research Methodology 12 (October 18, 2012): 158. doi:10.1186/1471-2288-12-158.
109. Hamm, Michele P., Terry P. Klassen, Shannon D. Scott, David Moher, and Lisa Hartling. “Education in Health Research
Methodology: Use of a Wiki for Knowledge Translation.” PLoS ONE 8, no. 5 (May 31, 2013).
110. Hansen, Julie B., Carsten B. Juhl, Isabelle Boutron, Peter Tugwell, Elizabeth A. T. Ghogomu, Jordi Pardo Pardo, Tamara
Rader, et al. “Assessing Bias in Osteoarthritis Trials Included in Cochrane Reviews: Protocol for a Meta-
Epidemiological Study.” BMJ Open 4, no. 10 (2014): e005491. doi:10.1136/bmjopen-2014-005491.
111. Hartling, Lisa, Kenneth Bond, Krystal Harvey, P. Lina Santaguida, Meera Viswanathan, and Donna M. Dryden.
Developing and Testing a tool for the Classification of Study Designs in Systematic Reviews of Interventions and
Exposures. AHRQ Methods for Effective Health Care. Rockville (MD): Agency for Healthcare Research and Quality
(US), 2010. http://www.ncbi.nlm.nih.gov/books/NBK52670/.
112. Hartling, Lisa, Michele P. Hamm, Ricardo M. Fernandes, Donna M. Dryden, and Ben Vandermeer. “Quantifying Bias in
Randomized Controlled Trials in Child Health: A Meta-Epidemiological Study.” PLoS ONE 9, no. 2 (February 4, 2014).
113. He, Jia, Liang Du, Guanjian Liu, Jin Fu, Xiangyu He, Jiayun Yu, and Lili Shang. “Quality Assessment of Reporting of
Randomization, Allocation Concealment, and Blinding in Traditional Chinese Medicine RCTs: A Review of 3159 RCTs
Identified from 260 Systematic Reviews.” Trials 12, no. 1 (May 13, 2011): 122. doi:10.1186/1745-6215-12-122.
114. Herrera. Question: Does photodynamic therapy (PDT) improve periodontal outcomes compared to scaling and root
planing (SRP) or no treatment? Azarpazhooh A, Shah PS, Tenenbaum HC, Goldberg MB. The effect of photodynamic
therapy for periodontitis: A Systematic Review and Meta-Analysis. J Periodontol 2010; 81: 4–14.
115. Hopp, Lisa. “Risk of Bias Reporting in Cochrane Systematic Reviews.” International Journal of Nursing Practice, 2014,
n/a – n/a. doi:10.1111/ijn.12252.
116. Hróbjartsson, A, and I Boutron. “Blinding in Randomized Clinical Trials: Imposed Impartiality.” Clinical Pharmacology &
Therapeutics 90, no. 5 (November 2011): 732–36. doi:10.1038/clpt.2011.207.
Page 89 of 637
117. Innes N. Question: Are individuals born with isolated oral clefts more likely to have dental anomalies than unaffected
individuals? Tannure PN, Oliveira CA, Maia LC, Vieira AR, Granjeiro JM, De Castro Costa M. Prevalence of dental
anomalies in nonsyndromic individuals with cleft lip and palate: A systematic review and meta-analysis. Cleft Palate
Craniofac J 2012; 49: 194–200.
118. Jakobsen, Janus Christian, Jørn Wetterslev, Per Winkel, Theis Lange, and Christian Gluud. “Thresholds for Statistical
and Clinical Significance in Systematic Reviews with Meta-Analytic Methods.” BMC Medical Research Methodology
14, no. 1 (November 21, 2014). doi:10.1186/1471-2288-14-120.
119. Jo, Jung Ki, Jae Hoon Chung, Kyu Shik Kim, Soo Hyun Song, and Seung Wook Lee. “Reporting of Randomized
Controlled Trials in Andrology Journals: A Quality Assessment.” The Journal of Sexual Medicine, 2015, n/a – n/a.
doi:10.1111/jsm.12784.
120. Johnston, Bradley C, Donald L Patrick, Jason W Busse, Holger J Schünemann, Arnav Agarwal, and Gordon H Guyatt.
“Patient-Reported Outcomes in Meta-Analyses – Part 1: Assessing Risk of Bias and Combining Outcomes.” Health and
Quality of Life Outcomes 11 (July 1, 2013): 109. doi:10.1186/1477-7525-11-109.
121. Jørgensen, Anders W, Katja L Maric, Britta Tendal, Annesofie Faurschou, and Peter C Gøtzsche. “Industry-Supported
Meta-Analyses Compared with Meta-Analyses with Non-Profit or No Support: Differences in Methodological Quality
and Conclusions.” BMC Medical Research Methodology 8 (September 9, 2008): 60. doi:10.1186/1471-2288-8-60.
122. Kalha AS. Question: What is the best orthodontic treatment approach for prominent upper front teeth?
Thiruvenkatachari B, Harrison JE, Worthington HV, O'Brien KD. Orthodontic treatment for prominent upper front
teeth (Class II malocclusion) in children. Cochrane Database Syst Rev 2013; 11: Art. No.: CD003452. DOI:
10.1002/14651858.CD003452.pub3.
123. Kirkham, Jamie J., Richard D. Riley, and Paula R. Williamson. “A Multivariate Meta-Analysis Approach for Reducing the
Impact of Outcome Reporting Bias in Systematic Reviews.” Statistics in Medicine 31, no. 20 (2012): 2179–95.
doi:10.1002/sim.5356.
124. Koretz, Ronald L., and Timothy O. Lipman. “The Presence and Effect of Bias in Trials of Early Enteral Nutrition in
Critical Care.” Clinical Nutrition 33, no. 2 (April 2014): 240–45. doi:10.1016/j.clnu.2013.06.006.
125. Kraglund F. Question: Are triclosan/copolymer containing fluoride toothpastes more effective than fluoride
toothpastes for control of caries, plaque and gingivitis in children and adults? Riley P, Lamont T. Triclosan/copolymer
containing toothpastes for oral health. Cochrane Database Syst Rev 2013; 12: Art. No.: CD010514. DOI:
10.1002/14651858.CD010514.pub2.
126. Krauth, David, Tracey J. Woodruff, and Lisa Bero. “Instruments for Assessing Risk of Bias and Other Methodological
Criteria of Published Animal Studies: A Systematic Review.” Environmental Health Perspectives 121, no. 9 (September
2013): 985–92. doi:10.1289/ehp.1206389.
127. Krauth, David, Tracey J. Woodruff, and Lisa Bero. “Instruments for Assessing Risk of Bias and Other Methodological
Criteria: Krauth et Al. Respond.” Environmental Health Perspectives 122, no. 3 (March 2014): A67.
doi:10.1289/ehp.1307727R.
128. Lambert. Question: What is the effectiveness of fluoride supplements (tablets, drops, lozenges and chewing gums) for
preventing dental caries in children? Tubert-Jeannin S, Auclair C, Amsallem E, et al. Fluoride supplements (tablets,
drops, lozenges or chewing gums) for preventing dental caries in children. Cochrane Database Syst Rev 2011; 12:
CD007592
129. Langford, Rebecca, Christopher P Bonell, Hayley E Jones, Theodora Pouliou, Simon M Murphy, Elizabeth Waters, Kelli
A Komro, Lisa F Gibbs, Daniel Magnus, and Rona Campbell. “The WHO Health Promoting School Framework for
Improving the Health and Well-Being of Students and Their Academic Achievement.” In Cochrane Database of
Systematic Reviews. John Wiley & Sons, Ltd, 1996.
http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD008958.pub2/abstract.
130. Li, Tianjing, Milo A Puhan, Swaroop S Vedula, Sonal Singh, and Kay Dickersin. “Network Meta-Analysis-Highly
Attractive but More Methodological Research Is Needed.” BMC Medicine 9 (June 27, 2011): 79. doi:10.1186/1741-
7015-9-79.
131. Lim, Sun Mi, Ein Soon Shin, Sun Hee Lee, Kyung Hwa Seo, Yu Min Jung, and Ji Eun Jang. “tools for assessing quality and
risk of bias by levels of evidence.” Journal of the Korean Medical Association 54, no. 4 (2011): 419.
doi:10.5124/jkma.2011.54.4.419.
Page 90 of 637
132. Louis, Deepak, Kiran More, Sapna Oberoi, and Prakesh S. Shah. “Intravenous Immunoglobulin in Isoimmune
Haemolytic Dizease of Newborn: An Updated Systematic Review and Meta-Analysis.” Archives of Dizease in Childhood
- Fetal and Neonatal Edition 99, no. 4 (July 1, 2014): F325–31. doi:10.1136/archdischild-2013-304878.
133. Lutomski JE, Donders AT, and Melis RF. “CAusal Diagrams to Better Understand Missingness.” JAMA Pediatrics 168,
no. 2 (February 1, 2014): 187–187. doi:10.1001/jamapediatrics.2013.3650.
134. MA Jie,LIU Ying,ZHONG Lai-ping,ZHANG Chen-ping,ZHANG Zhi-yuan. Comparison between Jadad scale and Cochrane
collaboration's tool for assessing risk of bias on the quality and risk of bias evaluation in randomized controlled trials
(Department of Oral & Maxillofacial-Head & Neck Oncology,Ninth People's Hospital,College of Stomatology,Shanghai
Jiao Tong University School of Medicine;Shanghai Key Laboratory of Stomatology.Shanghai 200011,China, China
Journal of Oral and Maxillofacial Surgery 2012-05
135. MacLennan S, Imamura M, Dahm P, Neuberger M, Reeves B, MacLennan G, Omar MI, McClinton S, Griffiths L, N'Dow
J. Assessing risk of bias in non-randomized studies and incorporating GRADE: initial experience with a new Cochrane
'Risk of bias' tool under development. Poster presentation at the 19th Cochrane Colloquium; 2011 Oct 19-22; Madrid,
Spain. Cochrane Database of Systematic Reviews, Supplement.
136. Matthews D. Weak evidence to support benefit of periodontal maintenance therapy in prevention of tooth loss.
Chambrone L, Chambrone D, Lima LA, Chambrone LA. Predictors of tooth loss during long-term periodontal
maintenance: a systematic review of observational studies. J Clin Periodontol. 2010; 37: 675–684.
137. McCay, L. “Ban against Industry Ties Introduces Bias and Obscures Whole View.” BMJ 343, no. sep06 4 (September 6,
2011): d5602–d5602. doi:10.1136/bmj.d5602.
138. Millett, Declan. “Bias in Systematic Reviews?” Journal of Orthodontics 38, no. 3 (September 1, 2011): 158–60.
doi:10.1179/14653121141407.
139. Moher, David, Jennifer Tetzlaff, Andrea C Tricco, Margaret Sampson, and Douglas G Altman. “Epidemiology and
Reporting Characteristics of Systematic Reviews.” PLoS Medicine 4, no. 3 (March 2007).
doi:10.1371/journal.pmed.0040078.
140. Morrison, Andra, Julie Polizena, Don Husereau, Kristen Moulton, Michelle Clark, Michelle Fiander, Monika
Mierzwinski-Urban, Tammy Clifford, Brian Hutton, and Danielle Rabb. “THE EFFECT OF ENGLISH-LANGUAGE
RESTRICTION ON SYSTEMATIC REVIEW-BASED META-ANALYSES: A SYSTEMATIC REVIEW OF EMPIRICAL STUDIES.”
International Journal of Technology Assessment in Health Care 28, no. 02 (April 2012): 138–44.
doi:10.1017/S0266462312000086.
141. Mullan, Rebecca J., David N. Flynn, Bo Carlberg, Imad M. Tleyjeh, Celia C. Kamath, Matthew L. LaBella, Patricia J.
Erwin, Gordon H. Guyatt, and Victor M. Montori. “Systematic Reviewers Commonly Contact Study Authors but Do so
with Limited Rigor.” Journal of Clinical Epidemiology 62, no. 2 (February 2009): 138–42.
doi:10.1016/j.jclinepi.2008.08.002.
142. Murad M, Montori VM, Ioannidis JA, and et al. “How to Read a Systematic Review and Meta-Analysis and Apply the
Results to Patient Care: Users’ Guides to the Medical Literature.” JAMA 312, no. 2 (July 9, 2014): 171–79.
doi:10.1001/jama.2014.5559.
143. Murad M. No available evidence to assess the effectiveness of bonded amalgams. Fedorowicz Z, Nasser M, Wilson N.
Adhesively bonded versus non-bonded amalgam restorations for dental caries. Cochrane Database Syst Rev 2009;
issue 4.
144. Naci, Huseyin, Sofia Dias, and A E Ades. “Industry Sponsorship Bias in Research Findings: A Network Meta-Analysis of
LDL Cholesterol Reduction in Randomized Trials of Statins.” BMJ : British Medical Journal 349 (October 3, 2014).
doi:10.1136/bmj.g5741.
145. Nankervis, Helen, Akerke Baibergenova, Hywel C. Williams, and Kim S. Thomas. “Prospective Registration and
Outcome-Reporting Bias in Randomized Controlled Trials of Eczema Treatments: A Systematic Review.” Journal of
Investigative Dermatology 132, no. 12 (December 2012): 2727–34. doi:10.1038/jid.2012.231.
146. Palys, Kaitlin E., and Vance W. Berger. “A Note on the Jadad Score as an Efficient tool for Measuring Trial Quality.”
Journal of Gastrointestinal Surgery 17, no. 6 (December 12, 2012): 1170–71. doi:10.1007/s11605-012-2106-0.
147. Pearson, Mark, and Jaime Peters. “Outcome Reporting Bias in Evaluations of Public Health Interventions: Evidence of
Impact and the Potential Role of a Study Register.” Journal of Epidemiology and Community Health 66, no. 4 (April 1,
2012): 286–89. doi:10.1136/jech.2010.122465.
Page 91 of 637
148. Pildal, J., A. Hróbjartsson, K. J. Jørgensen, J. Hilden, D. G. Altman, and P. C. Gøtzsche. “Impact of Allocation
Concealment on Conclusions Drawn from Meta-Analyses of Randomized Trials.” International Journal of Epidemiology
36, no. 4 (August 1, 2007): 847–57. doi:10.1093/ije/dym087.
149. Poulsen S. Available evidence does not support use of oxalates for dentine hypersensitivity. Cunha-Cruz J, Stout JR,
Heaton LJ, Wataha JC; Northwest PRECEDENT Dentin hypersensitivity and oxalates: a Systematic Review. J Dent Res
2011; 90: 304–310. Epub 2010 Dec 29.
150. Purgato, Marianna. “Assessing Risk of Bias in Randomized Controlled Trials.” Epidemiologia E Psichiatria Sociale 19,
no. 4 (2010): 296–97.
151. Rasines G. Using a fluoridated supplement with a high fluoride concentration in children aged under 6 years may
increase the risk of fluorosis. Wong MCM, Glenny AM, Tsang BWK, Lo ECM, Worthington HV, Marinho VCC. Topical
fluoride as a cause of dental fluorosis in children. Cochrane Database Syst Rev 2010; issue 1
152. Reeves, Barnaby C., Julian P. T. Higgins, Craig Ramsay, Beverley Shea, Peter Tugwell, and George A. Wells. “An
Introduction to Methodological Issues When Including Non-Randomized Studies in Systematic Reviews on the Effects
of Interventions.” Research Synthesis Methods 4, no. 1 (2013): 1–11. doi:10.1002/jrsm.1068.
153. Reveiz, Ludovic, An-Wen Chan, Karmela Krleža-Jerić, Carlos Eduardo Granados, Mariona Pinart, Itziar Etxeandia, Diego
Rada, Monserrat Martinez, Xavier Bonfill, and Andrés Felipe Cardona. “Reporting of Methodologic Information on
Trial Registries for Quality Assessment: A Study of Trial Records Retrieved from the WHO Search Portal.” PLoS ONE 5,
no. 8 (August 31, 2010): e12484. doi:10.1371/journal.pone.0012484.
154. Richards D. Limited evidence suggests that mouthrinses may be effective in reducing oral malodour
155. Rubinstein, Sidney M., Rik van Eekelen, Teddy Oosterhuis, Michiel R. de Boer, Raymond W. J. G. Ostelo, and Maurits
W. van Tulder. “The Risk of Bias and Sample Size of Trials of Spinal Manipulative Therapy for Low Back and Neck Pain:
Analysis and Recommendations.” Journal of Manipulative and Physiological Therapeutics 37, no. 8 (October 2014):
523–41. doi:10.1016/j.jmpt.2014.07.007.
156. Runnels, Vivien, Sari Tudiver, Marion Doull, and Madeline Boscoe. “The Challenges of Including Sex/gender Analysis in
Systematic Reviews: A Qualitative Survey.” Systematic Reviews 3 (April 10, 2014): 33. doi:10.1186/2046-4053-3-33.
157. Saini, Pooja, Yoon K Loke, Carrol Gamble, Douglas G Altman, Paula R Williamson, and Jamie J Kirkham. “Selective
Reporting Bias of Harm Outcomes within Studies: Findings from a Cohort of Systematic Reviews.” BMJ : British
Medical Journal 349 (November 21, 2014). doi:10.1136/bmj.g6501.
158. Santaguida, P. Lina, Crystal M. Riley, and David B. Matchar. “Chapter 5: Assessing Risk of Bias as a Domain of Quality
in Medical Test Studies.” Journal of General Internal Medicine 27, no. 1 (May 31, 2012): 33–38. doi:10.1007/s11606-
012-2030-8.
159. Savović, J., He Jones, Dg Altman, Rj Harris, P. Jűni, J. Pildal, B. Als-Nielsen, et al. “Influence of Reported Study Design
Characteristics on Intervention Effect Estimates from Randomized Controlled Trials: Combined Analysis of Meta-
Epidemiological Studies.” Health Technology Assessment (Winchester, England) 16, no. 35 (September 2012): 1–82.
doi:10.3310/hta16350.
160. Sequeira-Byron, Patrick, Zbys Fedorowicz, Vanitha A. Jagannath, and Mohammad Owaize Sharif. “An AMSTAR
Assessment of the Methodological Quality of Systematic Reviews of Oral Healthcare Interventions Published in the
Journal of Applied Oral Science (JAOS).” Journal of Applied Oral Science 19, no. 5 (October 2011): 440–47.
doi:10.1590/S1678-77572011000500002.
161. Shadrick V. Facemask therapy between ages six to ten years may lead to short term improvements for Class III
malocclusions. Watkinson S, Harrison JE, Furness S, Worthington HV. Orthodontic treatment for prominent lower
front teeth (Class III malocclusion) in children. Cochrane Database Syst Rev 2013; 9: Art. No. CD003451. DOI:
10.1002/14651858.CD003451.pub2.
162. Sidney et al. The Risk of Bias and Sample Size of Trials of Spinal Manipulative Therapy for Low Back and Neck Pain:
Analysis and Recommendations. Journal of Manipulative and Physiological Therapeutics. Volume 37, Issue 8, October
2014, Pages 523–541
163. Simon, E. -G., C. -J. Arthuis, and F. Perrotin. “L’aveugle Dans Les Essais Contrôlés Randomisés.” Gynécologie
Obstétrique & Fertilité 41, no. 2 (February 2013): 144–46. doi:10.1016/j.gyobfe.2012.12.005.
164. Smaïl-Faugeron, Violaine, Hélène Fron-Chabouis, and Frédéric Courson. “Methodological Quality and Implications for
Practice of Systematic Cochrane Reviews in Pediatric Oral Health: A Critical Assessment.” BMC Oral Health 14 (April 9,
Page 92 of 637
2014): 35. doi:10.1186/1472-6831-14-35.
165. Sola I, Martinez L, Roque M, Bonfill X. Do systematic reviews really accurately use the risk of bias assessment to draw
conclusions? Poster presentation at the Joint Cochrane and Campbell Colloquium; 2010 Oct 18-22; Keystone,
Colorado, USA.
166. Song, F, S Parekh, L Hooper, Yk Loke, J Ryder, Aj Sutton, C Hing, Cs Kwok, C Pang, and I Harvey. “Dissemination and
Publication of Research Findings: An Updated Review of Related Biases.” Health Technol Assess 14, no. 8 (February
2010). doi:10.3310/hta14080.
167. Souza, João P, Cynthia Pileggi, and José G Cecatti. “Assessment of Funnel Plot Asymmetry and Publication Bias in
Reproductive Health Meta-Analyses: An Analytic Survey.” Reproductive Health 4 (April 16, 2007): 3.
doi:10.1186/1742-4755-4-3.
168. Spuls, Phyllis I., and Alexander Nast. “Evaluation of and Perspectives on Guidelines: What Is Important?” Journal of
Investigative Dermatology 130, no. 10 (October 2010): 2348–49. doi:10.1038/jid.2010.247.
169. Szczesniak, Anna, Bartosz Gorny, Mariusz Kowalewski, Michalina Kolodziejczak, and Natalia Kotlarek. “Things to Avoid
in Meta-Analysis.” European Journal of Preventive Cardiology 20, no. 3 (June 1, 2013): 513–513.
doi:10.1177/2047487313484123.
170. Tendal, Britta, Eveline Nüesch, Julian P T Higgins, Peter Jüni, and Peter C Gøtzsche. “Multiplicity of Data in Trial
Reports and the Reliability of Meta-Analyses: Empirical Study.” BMJ : British Medical Journal 343 (2011).
doi:10.1136/bmj.d4829.
171. Thombs B, Arthurs E, El-Baalbaki G, Meijer A, Ziegelstein R, Steele R. Forecasting yesterday's weather: the risk of
spectrum bias from the inclusion of already diagnosed/treated patients in studies of depression screening tools. Oral
presentation at the 19th Cochrane Colloquium; 2011 Oct 19-22; Madrid, Spain [abstract]. Cochrane Database of
Systematic Reviews, Supplement
172. Thombs, B. D., E. Arthurs, G. El-Baalbaki, A. Meijer, R. C. Ziegelstein, and R. J. Steele. “Risk of Bias from Inclusion of
Patients Who Already Have Diagnosis of or Are Undergoing Treatment for Depression in Diagnostic Accuracy Studies
of Screening tools for Depression: Systematic Review.” BMJ 343, no. aug18 1 (August 18, 2011): d4825–d4825.
doi:10.1136/bmj.d4825.
173. Tricco, Andrea C., Jennifer Tetzlaff, and David Moher. “The Art and Science of Knowledge Synthesis.” Journal of
Clinical Epidemiology 64, no. 1 (January 2011): 11–20. doi:10.1016/j.jclinepi.2009.11.007.
174. Tricco, Andrea C., Jennifer Tetzlaff, Margaret Sampson, Dean Fergusson, Elize Cogo, Tanya Horsley, and David Moher.
“Few Systematic Reviews Exist Documenting the Extent of Bias: A Systematic Review.” Journal of Clinical
Epidemiology 61, no. 5 (May 2008): 422–34. doi:10.1016/j.jclinepi.2007.10.017.
175. Tugwell, Peter, André Knottnerus, and Leanne Idzerda. “How Many Risks-of-Bias and Comorbidity Scales Do We
Need?” Journal of Clinical Epidemiology 65, no. 9 (September 2012): 919–20. doi:10.1016/j.jclinepi.2012.07.001.
176. Turner, Lucy, James Galipeau, Chantelle Garritty, Eric Manheimer, L. Susan Wieland, Fatemeh Yazdi, and David
Moher. “An Evaluation of Epidemiological and Reporting Characteristics of Complementary and Alternative Medicine
(CAM) Systematic Reviews (SRs).” PLoS ONE 8, no. 1 (January 14, 2013). doi:10.1371/journal.pone.0053536.
177. Turner, Rebecca M, Jonathan Davey, Mike J Clarke, Simon G Thompson, and Julian PT Higgins. “Predicting the Extent
of Heterogeneity in Meta-Analysis, Using Empirical Data from the Cochrane Database of Systematic Reviews.”
International Journal of Epidemiology 41, no. 3 (June 2012): 818–27. doi:10.1093/ije/dys041.
178. Turner, Rebecca M., Sheila M. Bird, and Julian P. T. Higgins. “The Impact of Study Size on Meta-Analyses: Examination
of Underpowered Studies in Cochrane Reviews.” PLoS ONE 8, no. 3 (March 27, 2013).
179. Van Driel, Mieke L., An De Sutter, Jan De Maeseneer, and Thierry Christiaens. “Searching for Unpublished Trials in
Cochrane Reviews May Not Be Worth the Effort.” Journal of Clinical Epidemiology 62, no. 8 (August 2009): 838–44.e3.
doi:10.1016/j.jclinepi.2008.09.010.
180. Viswanathan, Meera, Nancy D. Berkman, Donna M. Dryden, and Lisa Hartling. Assessing Risk of Bias and Confounding
in Observational Studies of Interventions or Exposures: Further Development of the RTI Item Bank. AHRQ Methods
for Effective Health Care. Rockville (MD): Agency for Healthcare Research and Quality (US), 2013.
http://www.ncbi.nlm.nih.gov/books/NBK154461/.
181. Vollenweider, Daniela, Cynthia M Boyd, and Milo A Puhan. “High Prevalence of Potential Biases Threatens the
Page 93 of 637
Interpretation of Trials in Patients with Chronic Dizease.” BMC Medicine 9 (June 13, 2011): 73. doi:10.1186/1741-
7015-9-73.
182. Wong. Systemic nucleoside antiviral agents may be effective in prevention of recurrent herpes labialis. Rahimi H,
Mara T, Costella J, Speechley M, Bohay R. Effectiveness of antiviral agents for the prevention of recurrent herpes
labialis: a systematic review and meta-analysis. Oral Surg Oral Med Oral Pathol Oral Radiol. 2012; 113: 618–627.
183. Yavchitz, Amélie, Philippe Ravaud, Sally Hopewell, Gabriel Baron, and Isabelle Boutron. “Impact of Adding a
Limitations Section to Abstracts of Systematic Reviews on Readers’ Interpretation: A Randomized Controlled Trial.”
BMC Medical Research Methodology 14, no. 1 (November 24, 2014). doi:10.1186/1471-2288-14-123.
Page 94 of 637
Paper 2
Index of the HPV vaccine study programmes
Page 95 of 637
DECLARATION OF CO-AUTHORSHIP
Information on PhD student:
Name of PhD student Lars Jørgensen
E-mail lj@cochrane.dk; jcl649@sund.ku.dk; larsjorgensens@gmail.com
Date of birth 4 February 1987
Work place Nordic Cochrane Centre
Principal supervisor Peter C. Gøtzsche
Title of PhD thesis:
Benefits and harms of the human papillomavirus (HPV) vaccines
This declaration concerns the following article:
Jørgensen L, Gøtzsche PC and Jefferson T. Index of the human papillomavirus (HPV) vaccine industry clinical study
programmes and non-industry funded studies: a necessary basis to address reporting bias in a systematic review.
Syst Rev. 2018 Jan 18;7(1):8. https://doi.org/10.1186/s13643-018-0675-z.
The PhD student’s contribution to the article:

(please use the scale (A,B,C) below as benchmark*)
(A,B,C)
1. Formulation/identification of the scientific problem that from theoretical questions need to be C
clarified. This includes a condensation of the problem to specific scientific questions that is judged
to be answerable by experiments
2. Planning of the experiments and methodology design, including selection of methods and method C
development
3. Involvement in the experimental work C
4. Presentation, interpretation and discussion in a journal article format of obtained data C
*Benchmark scale of the PhD student’s contribution to the article

A. refers to: Has contributed to the co-operation 0-33 %
B. refers to: Has contributed considerably to the co-operation 34-66 %
C. refers to: Has predominantly executed the work independently 67-100 %
Signature of the co-authors:

Date: Name: Title: Signature:
1 Peter C. Gøtzsche DrMedSci,
November professor
2018
1 Tom Jefferson MD, MSc.
November MRCGP,
2018 FFPHM
Page 96 of 637
Signature of the PhD student and the principal supervisor:
Date: 1 November 2018 Date: 1 November 2018
PhD student: Principal supervisor:
2
Page 97 of 637
Jørgensen et al. Systematic Reviews (2018) 7:8
DOI 10.1186/s13643-018-0675-z
RESEARCH Open Access
Index of the human papillomavirus (HPV)

vaccine industry clinical study programmes
and non-industry funded studies: a
necessary basis to address reporting bias in
a systematic review
Lars Jørgensen* , Peter C. Gøtzsche and Tom Jefferson
Abstract
Background: Unabridged access to drug industry and regulatory trial registers and data reduces reporting bias in
systematic reviews and may provide a complete index of a drug’s clinical study programme. Currently, there is no
public index of the human papillomavirus (HPV) vaccine industry study programmes or a public index of non-
industry funded studies.
Methods: By cross-verification via study programme enquiries to the HPV vaccine manufacturers and regulators
and searches of trial registers and journal publication databases, we indexed clinical HPV vaccine studies as a basis
to address reporting bias in a systematic review of clinical study reports.
Results: We indexed 206 clinical studies: 145 industry and 61 non-industry funded studies. One of the four HPV
vaccine manufacturers (GlaxoSmithKline) provided information on its study programme. Most studies were cross-
verified from two or more sources (160/206, 78%) and listed on regulatory or industry trial registers or journal
publication databases (195/206, 95%)—in particular, on ClinicalTrials.gov (176/195, 90%). However, study results
were only posted for about half of the completed studies on ClinicalTrials.gov (71/147, 48%). Two thirds of the
industry studies had a study programme ID, manufacturer specific ID, and national clinical trial (NCT) ID (91/145,
63%). Journal publications were available in journal publication databases (the Cochrane Collaboration’s Central
Register of Controlled Trials, Google Scholar and PubMed) for two thirds of the completed studies (92/149, 62%).
Conclusion: We believe we came close to indexing complete HPV vaccine study programmes, but only one of
the four manufacturers provided information for our index and a fifth of the index could not be cross-verified.
However, we indexed larger study programmes than those listed by major regulators (i.e., the EMA and FDA that
based their HPV vaccine approvals on only half of the available trials). To reduce reporting bias in systematic
reviews, we advocate the registration and publication of all studies and data in the public domain.
Keywords: Human papillomavirus vaccine, Index, Study programme, Clinical study reports, Reporting bias
* Correspondence: lj@cochrane.dk
Nordic Cochrane Centre, Rigshospitalet 7811, Blegdamsvej 9, 2100
Copenhagen, Denmark
© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to
the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Page 98 of 637
Background (HPV) vaccines as a basis for a systematic review of clin-

Healthcare guidelines often rely on drug manufacturers’ ical study reports (see our protocol at PROSPERO:
studies, regulators’ assessments and independent CRD42017056093 [27]).
researchers’ systematic reviews of these studies. Drug man-
ufacturers usually conduct study programmes in agreement Methods
with the drug regulators’ pre-approval requirements. How- Our study involved indexing the HPV vaccine industry
ever, more than half of all studies are never published [1, 2] study programmes and non-industry funded clinical stud-
and the published studies’ intervention effects are often ex- ies using a six-step process that focuses on identifying un-
aggerated in comparison to the unpublished studies [3–5]. published studies (see Additional file 1 for a detailed
This introduces reporting bias that undermines the validity description of steps 1 to 6 and Additional file 2 for the e-
of systematic reviews. To address reporting bias in system- mail correspondence with the HPV vaccine manufacturers
atic reviews, it is necessary to use industry and regulatory in step 6).
trial registers and trial data—in particular, the drug manu- In step 1, we corresponded with the EMA and obtained a
facturers’ complete study programmes with their corre- list of their holdings of clinical study reports and Module
sponding clinical study reports. 2.5 of the common technical document (CTD) for one
A clinical study report of over a thousand pages in length HPV vaccine (Gardasil 9) that listed all studies in the Garda-
may be condensed into a ten-page journal publication, i.e., sil 9 study programme. This gave us a basic study list with
a compression factor of 100 [6, 7]. The criteria used to se- industry study identifiers that usually consist of a prefix that
lect the resulting fraction of available data in journal publi- identifies the HPV vaccine being tested (for example, HPV-
cations are unknown and journal publications are often xxx for Cervarix and V50x-xxx for Gardasil studies). In
misleading, especially in relation to the reporting of harms mid-2017, we were granted access to EMA’s holdings of
of drug interventions [7–14]. Consequently, some re- Modules 2.5 of two other HPV vaccines (Cervarix and Gar-
searchers now rely on study programmes and clinical study dasil), but we have not received the modules yet.
reports as their primary or only source of information for In step 2, we expanded the basic list by searching 45 trial
their systematic reviews [7, 13, 15]. The first systematic re- registers: eight industry trial registers (where the manufac-
view of influenza antivirals showed the feasibility of this ap- turers had been involved or possibly involved in one or
proach [15]. The review demonstrated the importance of more HPV vaccine studies), 32 international and regional
using clinical study reports and the shortcomings of relying trial registers chosen according to their level of impact
only on journal publication database searches (for example, (e.g., https://clinicaltrials.gov and http://apps.who.int/trial-
in PubMed). Although the use of clinical study reports research/ are used globally and were considered high im-
duces the risk of reporting bias, clinical study reports pact) or where one or more HPV vaccine studies had been
themselves may still be subject to significant reporting bias conducted (e.g., we searched the Chinese Clinical Trial
when compared to their underlying data [6, 12, 16]. Registry: http://www.chictr.org.cn/index.aspx, since several
Currently, there is no easily accessible source to access HPV vaccine studies had been conducted in China) and
study programmes. However, drug manufacturers’ common five regulatory registers (where the regulators had been in-
technical documents (CTDs) contain lists of the studies in volved or possibly involved with the assessment or ap-
a study programme that support marketing authorization proval of one or more HPV vaccines, e.g., EMA and FDA).
applications (MAAs) to regulatory authorities (such as the In step 3, we searched the HPV vaccines’ regulatory drug
European Medicines Agency, EMA, or the Food and Drug approval packages (DAPs) from FDA and the HPV vac-
Administration, FDA). Since 2010 and 2015, it has been cines’ European Public Assessment Reports (EPARs) from
possible for researchers to obtain common technical docu- EMA to identify studies possibly not listed in the 45 trial
ments and marketing authorization applications from the registers.
EMA via policies 0043 and 0070, respectively. In step 4, we conducted searches of three journal publica-
Nonetheless, accessing study programmes and unpub- tion databases (the Cochrane Collaboration’s Central Regis-
lished studies and obtaining clinical study reports from ter of Controlled Trials, Google Scholar, and PubMed) to
the industry and regulators can prove difficult [13, 17–20]. identify studies possibly not listed in the 45 trial registers.
Therefore, it is not surprising that only a minority of sys- We also searched WikiLeaks for HPV vaccine studies (or
tematic reviews include unpublished studies (10–20%) related information). When possible, we matched indexed
[21, 22] or searches of trial registers (10–50%) [23–26]. studies to their corresponding journal publications.
Searches of trial registers may lead to the identification of In step 5, we added studies listed in recent HPV vaccine
additional eligible studies in about 20–60% of systematic regulatory and independent reviews to verify the studies’
reviews [23–26]. To identify unpublished studies and ad- existence and add any studies that we had not identified.
dress reporting bias, we present here a method for index- In step 6, we sent the assembled indexes to the corre-
ing the study programmes of the human papillomavirus sponding HPV vaccine manufacturers and requested them
Page 99 of 637
to verify the indexed studies’ existence and add any studies III studies (89/206, 43%). The remaining studies were ei-
that we had not identified. We gave the manufacturers a ther follow-ups (23 industry and three non-industry),
1-month deadline to respond and sent a reminder if the non-randomized (24 and 18), or of unclear design (two
manufacturer had not responded within 1 to 2 weeks. industry studies). Most randomized clinical trials had ei-
We indexed interventional prospective preventive (not ther another vaccine (for example, the hepatitis A vac-
therapeutic) comparative (with two or more intervention cine, Havrix) or the HPV vaccine aluminum adjuvants as
arms) HPV vaccine clinical studies (and their follow-up comparators (111/136, 82%); only 17 studies used a “pla-
studies) in humans. We classified studies by cross- cebo” (13%; note that if a study reported that it used a
verification as follows: “placebo” comparator, but the study did not describe the
“placebo”, i.e., as saline, we noted the study’s comparator
1) “Definitely exists” (cross-verification of a study’s as “placebo”). A third of the industry studies and no
existence from two or more sources) non-industry study used adjuvant comparisons (36/96
2) “Probably exists” (verification of a study’s existence vs. 0/40, P < 0.0001; Fisher’s exact test). More industry
from one source) studies were phase III (79/145 vs. 10/61, P < 0.0001)
3) “Probably does not exist” (no manufacturer or while more non-industry studies used a “placebo” com-
regulatory verification but with a passing reference parison (12/40 vs. 5/96, P = 0.0002) and were phase IV
identified from another source) studies (22/61 vs. 9/145, P < 0.0001) (see Table 1).
GlaxoSmithKline’s study programme primarily used
We indexed studies from the first two categories. No its Cervarix HPV vaccine (65/69, 94%) and included
language restriction was applied, and Google Translate more follow-up studies than the Merck Sharp &
was used for non-familiar languages. By comparing all Dohme’s study programme (18/69 vs. 5/66, P = 0.006)
gathered study IDs, we deleted duplicate entries. that mainly used its Gardasil or Gardasil 9 HPV vac-
One author (LJ) conducted steps 1 to 6 of the indexing cines (55/66, 83%, see Table 1).
and extracted data. The steps were conducted from Octo- Most industry studies were solely industry funded
ber 2016 to July 2017. A second author (TJ) checked the (128/145, 88%), but Merck Sharp and Dohme co-funded
indexing and data extraction. Any disagreements were more studies than GlaxoSmithKline (17/66 vs. 0/69, P <
solved by discussion or by consulting the third author 0.0001; ten Merck studies were co-funded with univer-
(PCG). One author (LJ) classified the studies according to sities, four with hospitals and three with governmental
the likelihood of their existence (e.g., “definitely exists”) healthcare institutions) (see Table 1).
and assessed the degree of manufacturer involvement (i.e., Most studies were completed (149/206, 72%). In com-
studies that were funded or partly funded by the manufac- parison to industry studies, non-industry studies were
turers were classified as industry studies). A second author on average of a longer duration (42.2 vs. 36.9 months),
(TJ) checked the classifications. while industry studies were more often multicentre (79/
For each study, one author (LJ) extracted the following 106 vs. 13/53, P < 0.0001) (see Table 1).
study information: type, phase (I-IV), intervention type, Most studies included only females (151/206, 73%)
completion status (completed or on-going), centre status and had participants younger than 18 years (105/206,
(single or multicentre), participant characteristics (age, 51%). More non-industry studies included both sexes
gender and number of participants), programme ID, manu- (19/61 vs. 22/145, P = 0.013), but Merck Sharp and
facturer ID, trial register ID, results availability, and modes Dohme funded more studies on both sexes than GlaxoS-
of identification. mithKline (16/66 vs. 4/69, P = 0.003). Industry studies
had on average twice as many participants enrolled com-
Results pared to non-industry studies (3602 vs. 1767). For multi-
We excluded 79 non-comparative prospective clinical centre studies, the mean number of participants was
studies and indexed 206 studies: 145 industry and 61 non- similar for industry and non-industry studies (2388 vs.
industry funded studies with a total of 623,005 participants. 2745), but non-industry studies enrolled twice as many
One of the four HPV vaccine manufacturers (GlaxoS- participants per study centre compared to industry stud-
mithKline) provided us with an index of 81 GlaxoSmithK- ies (128 vs. 66, see Table 1).
line studies and four MedImmune studies (MedImmune Most indexed studies were cross-verified from two
and GlaxoSmithKline collaborated in the early develop- or more sources (i.e., “definitely exists”, 160/206,
ment of GlaxoSmithKline’s Cervarix HPV vaccine) (see 78%). Merck Sharp and Dohme’s Gardasil and
Fig. 1, Table 1 and Additional files 1, 2, 3; and our study’s Gardasil 9 programme had more studies with a single
PRISMA statement, Additional file 4). verification than GlaxoSmithKline’s Cervarix programme
Two thirds of the indexed studies were randomized (i.e., “probably exists”, 16/66 vs. 0/69, P < 0.0001, see
clinical trials (136/206, 66%) and about half were phase Table 2).
Page 100 of 637

Fig. 1 Flowchart of the identification of the HPV vaccines industry study programmes and non-industry funded clinical studies
Most industry studies had a study programme spe- Discussion

cific ID (125/145, 86%), manufacturer specific ID (98/ Our index showed serious deficiencies and variability in
145, 68%) and the US national clinical trial (NCT) the availability of HPV vaccine studies and data. For ex-
ID, which was also used by most non-industry studies ample, only half of the completed studies listed on Clini-
(132/145 vs. 44/61, P = 0.0009). About two thirds of calTrials.gov posted their results. The clinical study
the industry studies used all three ID types (91/145, reports we obtained via our index process confirmed
63%, see Table 2). that the amount of information and data are vastly
Most of the included studies were listed in industry, greater than that in journal publications. For example,
public or regulatory registers or databases (195/206, the journal publication for one GlaxoSmithKline Cer-
95%)—in particular, on ClinicalTrials.gov (176/195, varix study (HPV-008) is 14 pages long [28] while its
90%). However, study results were only posted for publicly available corresponding clinical study report is
about half of the completed studies on ClinicalTrials.- more than 7000 pages long [29], even though it is a
gov (71/147, 48%), but more completed industry stud- shortened interim report.
ies posted results on ClinicalTrials.gov compared to Identification of some studies involved a considerable
non-industry studies (65/110 vs. 6/37, P < 0.0001, see amount of work and a fifth of the index could not be
Table 2). cross-verified. We indexed studies that we would not
We reconciled the index with journal publications, but have been able to index if we only relied on the journal
did not run all-inclusive journal publication database publication databases. For example, one Cervarix ran-
searches as recommended for systematic reviews, since domized clinical trial (HPV-002) was only listed on
clinical study reports (not journal publications) were our GlaxoSmithKline’s trial register. The index that GlaxoS-
focus. Journal publications were available for two thirds mithKline provided us with contained three Cervarix
of the completed studies (92/149, 62%, see Table 2). meta-analyses that were only identifiable by their GSK
Page 101 of 637

Table 1 Characteristics of the HPV vaccine industry study programmes and non-industry funded clinical studies
Study characteristics Industry HPV vaccine studies Non-industry funded HPV vaccine studies P valuek
j
Total: GSK Merck (Gardasil and Other : Total: N Cervarix: Gardasil and Other:
N = 145 (Cervarix): Gardasil 9): N = 66 N = 10 = 61 N=6 Gardasil 9: N=7
N = 69 N = 48
Type of study
Randomized clinical trial 96 (67%) 43 (62%) 45 (68%) 8 (80%) 40 (65%) 4 (67%) 33 (69%) 3 (43%) 1.00
a
- “Placebo” comparison 5 of 96 0 of 43 3 of 45 2 of 8 12 of 40 0 of 4 11 of 33 1 of 3 0.0002
(5%) (0%) (7%) (25%) (30%) (0%) (33%) (33%)
- Adjuvantb comparison 36 of 96 15 of 43 19 of 45 2 of 8 0 of 40 0 of 4 0 of 33 0 of 3 <0.0001
(39%) (35%) (42%) (25%) (0%) (0%) (0%) (0%)
- Vaccinec comparison 51 of 96 28 of 43 19 of 45 4 of 8 24 of 40 3 of 4 19 of 33 2 of 3 0.57
(51%) (65%) (42%) (50%) (60%) (75%) (58%) (67%)
- No intervention in control 0 of 96 0 of 43 0 of 45 0 of 8 3 of 40 1 of 4 2 of 33 0 of 3 0.027
arm (0%) (0%) (0%) (0%) (8%) (25%) (6%) (0%)
- Unclear 4 of 96 0 of 43 4 of 45 0 of 8 1 of 40 0 of 4 1 of 33 0 of 3 1.00
(5%) (0%) (9%) (0%) (2%) (0%) (3%) (0%)
Follow-up to randomized 23 (16%) 18 (26%) 5 (8%) 0 (0%) 3 (5%) 0 (0%) 0 (0%) 3 (43%) 0.037
clinical trial
Non-randomized 24 (16%) 8 (12%) 14 (21%) 2 (20%) 18 (30%) 2 (33%) 15 (31%) 1 (14%) 0.039
Unclear 2 (1%) 0 (0%) 2 (3%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1.00
Phase of studyd
I 8 (5%) 2 (3%) 4 (6%) 2 (20%) 6 (10%) 0 (0%) 2 (4%) 4 (57%) 0.36
II 32 (22%) 14 (20%) 14 (21%) 4 (40%) 7 (11%) 0 (0%) 6 (13%) 1 (14%) 0.081
III 79 (55%) 44 (64%) 31 (47%) 4 (40%) 10 (16%) 1 (17%) 9 (19%) 0 (0%) <0.0001
IV 9 (6%) 4 (6%) 5 (8%) 0 (0%) 22 (37%) 5 (83%) 17 (35%) 0 (0%) <0.0001
Unclear 17 (12%) 5 (7%) 12 (18%) 0 (0%) 16 (26%) 0 (0%) 14 (29%) 2 (29%) 0.013
Type of HPV vaccine used
Monovalent 5 (3%) 0 (0%) 5 (8%) 0 (0%) 4 (7%) 0 (0%) 0 (0%) 4 (57%) 0.45
Bivalent (e.g., Cervarix) 76 (52%) 65 (94%) 2 (3%) 9 (90%) 7 (11%) 6 (100%) 0 (0%) 1 (14%) <0.0001
Quadrivalent (e.g., Gardasil) 43 (30%) 4 (6%) 38 (57%) 1 (10%) 44 (72%) 0 (0%) 44 (92%) 0 (0%) <0.0001
Octavalent 3 (2%) 0 (0%) 3 (5%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0.56
Ninevalent (e.g., Gardasil 9) 17 (12%) 0 (0%) 17 (25%) 0 (0%) 4 (7%) 0 (0%) 4 (8%) 0 (0%) 0.45
Unclear 1 (1%) 0 (0%) 1 (2%) 0 (0%) 2 (3%) 0 (0%) 0 (0%) 2 (29%) 0.21
Funding
Industry funded study 128 69 49 10 0 0 0 0 <0.0001
(88%) (100%) (75%) (100%) (0%) (0%) (0%) (0%)
Industry co-funded study 17 (12%) 0 (0%) 17 (25%)i 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0.004
Non-industry funded study 0 (0%) 0 (0%) 0 (0%) 0 (0%) 61 (100%) 6 (100%) 48 (100%) 7 (100%) <0.0001
Study completion status
Completed 110 57 49 4 39 3 29 7 0.090
(76%) (83%) (75%) (40%) (64%) (50%) (60%) (100%)
- Mean study time in monthse 36.9 34.2 42.1 15.8 42.2 45 32.6 56 NAl
[2; 140] [2; 97] [8; 140] [8; 30] [5; 143] [12; 81] [6; 66] [5; 143]
On going 27 (19%) 7 (10%) 14 (20%) 6 (60%) 19 (31%) 2 (33%) 17 (36%) 0 (0%) 0.066
Terminated prematurely 5 (3%) 4 (6%) 1 (2%) 0 (0%) 2 (3%) 1 (17%) 1 (2%) 0 (0%) 1.00
Unclear 3 (2%) 1 (1%) 2 (3%) 0 (0%) 1 (2%) 0 (0%) 1 (2%) 0 (0%) 1.00
Study centre statusf
Single centre 27 of 106 9 of 60 11 of 37 7 of 9 40 of 53 6 of 6 28 of 41 6 of 6 <0.0001
(25%) (15%) (30%) (78%) (75%) (100%) (68%) (100%)
Page 102 of 637

Table 1 Characteristics of the HPV vaccine industry study programmes and non-industry funded clinical studies (Continued)
Study characteristics Industry HPV vaccine studies Non-industry funded HPV vaccine studies P valuek
j
Total: GSK Merck (Gardasil and Other : Total: N Cervarix: Gardasil and Other:
N = 145 (Cervarix): Gardasil 9): N = 66 N = 10 = 61 N=6 Gardasil 9: N=7
N = 69 N = 48
Multicentre 79 of 106 51 of 60 26 of 37 2 of 9 13 of 53 0 of 6 13 of 41 0 of 6 <0.0001
(75%) (85%) (70%) (22%) (25%) (0%) (32%) (0%)
- Mean centres per multicentre 36.5 35.3 39.9 6.0 21.5 NA 21.5 NA NA
study [2; 105] [2; 135] [2; 105] [2; 10] [2; 134] [2; 134]
Unclear 39 (27%) 9 (13%) 29 (44%) 1 (10%) 8 (13%) 0 (0%) 7 (15%) 1 (14%) 0.044
Participants
Both females and males 22 (16%) 4 (5%) 16 (24%) 2 (20%) 19 (31%) 0 (0%) 14 (29%) 5 (71%) 0.013
Only females 113 64 41 8 38 6 30 2 0.025
(78%) (93%) (62%) (80%) (62%) (100%) (63%) (29%)
Only males 5 (3%) 1 (2%) 4 (6%) 0 (0%) 4 (7%) 0 (0%) 4 (8%) 0 (0%) 0.45
Unclear 5 (3%) 0 (0%) 5 (8%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0.32
Total number of enrolled 522,298 122,323 376,643 23,332 100,707 42,801 47,452 10,454 <0.0001
participantsg
- Mean participants per studyg 3602 [2; 1773 [2; 6726 [24; 2592 [90; 1767 [12; 8560 [200; 1054 1493 [45; NA
189,629] 34,206] 189,629] 12,000] 24,000] 24,000] [12; 20,000] 10,000]
- Mean participants per 2388 [20; 2073 [20; 2692 6450 [900; 2745 [75; NA 2745 [75; NA NA
multicentre studyf,g 34,206] 34,206] [67; 14,840] 12,000] 20,000] 20,000]
- Mean participants per 66 59 67 1075 [450; 128 NA 128 NA NA
centre in multicentre studiesf,g [2; 1513] [2; 1513] [10; 240] 1200] [5; 2222] [5; 2222]
Studies with > 1000 51 (35%) 22 (32%) 26 (39%) 3 (30%) 10 (16%) 2 (33%) 7 (15%) 1 (14%) 0.007
participantsg
Studies with participants 77 (53%) 23 (33%) 49 (75%) 5 (50%) 28 (46%) 4 (67%) 23 (48%) 1 (14%) 0.36
under age 18h
a
If a study reported that it used a ‘placebo’ comparator, but the study did not describe the “placebo” (i.e., as saline), we noted the comparator as ‘placebo’
b
Adjuvant comparisons contained, for example, the aluminum adjuvants used in Cervarix and Gardasil/Gardasil 9, i.e., aluminum hydroxide (Al[OH]3) and
amorphous aluminum hydroxyphosphate sulfate (AAHS), respectively
c
Vaccine comparisons included: Adacel, Boostrix, Cervarix (compared with Gardasil or Gardasil 9), Dengvaxia, Engerix, Gardasil (compared with Cervarix or Gardasil
9), Gardasil 9 (compared with Cervarix or Gardasil), Havrix, Infanrix, Menactra, Priorix, Repevax, and Twinrix
d
If a study was both a phase I and a phase II study, we noted the study as the uppermost phase (i.e., phase II)
e
105 of the 110 completed industry studies and 33 of the 39 completed non-industry studies had information for mean study time
f
The exact number of study centers could be assessed in 106 of the 145 industry studies and 53 of the 61 non-industry studies
g
The number of participants could be assessed in 140 of 145 industry studies and 57 of 61 non-industry studies (studies that were terminated prematurely were
not included)
h
The number of participants under age 18 could be assessed in 140 of the 145 industry studies and 59 of the 61 non-industry studies
i
10 Merck studies were co-funded with universities, four with hospitals and three with governmental healthcare institutions
j
Other HPV vaccine manufacturers were Shanghai Zerun Biotechnology Co., Ltd. and Xiamen Innovax Biotech Co., Ltd
k
P values were calculated for total industry studies vs. total non-industry studies with Fisher’s exact test (http://www.langsrud.com/fisher.htm)
l
Not applicable
ID (i.e., 205206, 207644, and 205639), and the index did https://www.drks.de/; and India: http://ctri.nic.in/) (see
not include three randomized clinical trials (HPV-009 Additional files 2 and 3).
and HPV-016 and the prematurely terminated HPV-078) We also indexed more studies than those listed in the
that were listed on ClinicalTrials.gov. One Cervarix trial holdings of major regulators. For example, EMA con-
(HPV-049) and three Gardasil studies (V501–001, ducted a review (of the relation between HPV vaccination
V501–002, and V501–004) were only identified via and two syndromes: postural orthostatic tachycardia syn-
regulatory registers or correspondence. One Gardasil drome, POTS and complex regional pain syndrome, CRPS)
follow-up study (for V501–005) only had a journal pub- [30] that EMA believed was based on the manufacturers’
lication listed in PubMed without any manufacturer- complete HPV vaccine study programmes (“GSK [GlaxoS-
specific ID or registration. Four non-industry studies mithKline] has conducted a review of all available data
were published in journal publications but were not from clinical studies…with Cervarix”; and “The MAH
registered in any of the 45 trial registers. Three non- [Market Authorisation Holder, i.e., Merck Sharp and
industry studies were only listed on regional trial regis- Dohme] has reviewed data from all clinical studies of the
ters (Australia: https://www.anzctr.org.au/; Germany: qHPV vaccine [Gardasil]” (30,31)). However, when we
Page 103 of 637

Table 2 Study classification, identification and results availability in the HPV vaccine industry study programmes and non-industry
funded clinical studies
Study classification, identification Industry HPV vaccine studies Non-industry funded HPV vaccine studies P valueh
and results availability
Total: GSK Merck (Gardasil and Otherg: Total: Cervarix: Gardasil and Other:
N = 145 (Cervarix): Gardasil 9): N = 66 N = 10 N = 61 N=6 Gardasil 9: N=7
N = 69 N = 48
Study classificationa
‘Definitely exists’ 127 69 50 8 33 5 28 0 < 0.0001
(88%) (100%) (76%) (80%) (54%) (83%) (58%) (0%)
‘Probably exists’ 18 (12%) 0 (0%) 16 (24%) 2 (20%) 28 (46%) 1 (17%) 20 (42%) 7 (100%) < 0.0001
Study identification (ID)
Uses study programme 125 (86%) 66 (96%) 49 (74%) 10 (100%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) < 0.0001
specificb ID
Uses manufacturer specificc ID 98 (68%) 68 (99%) 30 (45%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) < 0.0001
Uses both study programme and 95 (66%) 65 (94%) 30 (45%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) < 0.0001
manufacturer specific IDs
Uses national clinical study (NCT) 132 (91%) 64 (93%) 58 (88%) 10 (100%) 44 (72%) 6 (100%) 33 (69%) 5 (71%) 0.0009
ID
Uses study programme, 91 (63%) 61 (88%) 30 (45%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) < 0.0001
manufacturer specific and
NCT IDs
Uses additional or other ID(s) 42 (29%) 26 (38%) 16 (24%) 0 (0%) 51 (84%) 5 (83%) 40 (83%) 6 (86%) < 0.0001
Study results availability
Listed in a register or databased 138 (95%) 69 (100%) 59 (89%) 10 (100%) 57 (93%) 6 (100%) 45 (94%) 6 (86%) 0.73
Listed on ClinicalTrials.gov 132 (90%) 64 (93%) 58 (88%) 10 (100%) 44 (72%) 6 (100%) 33 (69%) 5 (71%) 0.0009
- Results posted on 65 of 132 37 of 64 28 of 58 0 of 10 6 of 44 0 of 6 6 of 33 0 of 5 0.0002
ClinicalTrials.gov (44%) (58%) (48%) (0%) (14%) (0%) (18%) (0%)
- Results posted on 65 of 110 37 of 57 28 of 49 0 of 4 6 of 37 0 of 3 6 of 29 0 of 5 < 0.0001
ClinicalTrials.gov for completed (58%) (65%) (57%) (0%) (16%) (0%) (21%) (0%)
studies
Published in a biomedical journale 76 of 110 42 of 57 34 of 49 0 of 4 16 of 39 3 of 3 12 of 29 1 of 7 0.004
(69%) (74%) (69%) (0%) (41%) (100%) (41%) (14%)
Probably not published in a 34 of 110 15 of 57 15 of 49 4 of 4 23 of 39 0 of 3 17 of 29 6 of 7 0.004
biomedical journale,f (31%) (26%) (31%) (100%) (59%) (0%) (59%) (86%)
a
For “definitely exists” studies we demanded cross-verification of a studies existence from two or more sources. For ‘probably exists’ studies we demanded verifica-
tion of a studies existence from one source
b
The HPV vaccine manufacturers usually identified their HPV vaccine study programmes with specific identifiers, for example, “HPV-xxx” for Cervarix and “V50x-
xxx” Gardasil and Gardasil 9
c
The HPV vaccine manufacturers usually identified their HPV vaccine studies with manufacturer specific identifiers, for example, GlaxoSmithKline used a six-digit
identifier (e.g., 104,896) and Merck Sharp & Dohme used a seven-digit identifier (e.g., 2004_081)
d
See Additional file 1 for a complete list of the trial registers and databases that we searched
e
110 of the 145 industry studies and 39 of the 61 non-industry studies were completed and assessed for publication status
f
”Probably not published” studies were categorized as such if they were not identified as journal publications in the searches we performed (see Methods and
Additional files 1 and 3)
g
Other HPV vaccine manufacturers were Shanghai Zerun Biotechnology Co., Ltd. and Xiamen Innovax Biotech Co., Ltd
h
P values were calculated for total industry studies vs. total non-industry studies with Fisher’s exact test (http://www.langsrud.com/fisher.htm)
compared the manufacturers’ study programmes (submit- Approval Packages (Cervarix: 17/36, 47%; Gardasil: 6/11,
ted to EMA, see Additional file 1) with our index (see 54%; and Gardasil 9: 9/13, 69%). We find this very
Table 1 and Additional file 3), we found that only half (38/ disturbing.
79, 48%) of the manufacturers’ randomized clinical trials To our knowledge, our study is the first with the aim
and follow-ups of Cervarix and Gardasil completed before of indexing a complete study programme. We do not
the submission dates in July 2015 were included (EMA’s re- know if the considerable reporting bias we found is
view did not assess Gardasil 9) [30, 31]. Similarly, the FDA’s generalizable to all drugs and vaccines, but similar in-
Drug Approval Packages (DAPs) only mentioned half (32/ dustry examples exist for oseltamivir [15], rofecoxib
60, 53%) of the randomized clinical trials and follow-ups [32], and rosiglitazone where 83% (35/42) of the study
that were completed before the vaccines’ date of the Drug programme was unpublished [33, 34]. Indexing is
Page 104 of 637

important when there is high risk of reporting bias, incomplete clinical study reports more than 3 years after
which often is the case for industry funded drug trials. the initial request (as of 1 July 2017), which is only half of
Our approach should therefore be considered for sys- the clinical study reports included in the EMA review (18/
tematic reviews of drugs and vaccines. Steps 2 and 4 38) [30] and a fifth of our indexed randomized clinical in-
contributed quantitatively the most to the identification dustry trials (18/96).
and cross-verification of studies—in particular, searches
on ClinicalTrials.gov and the HPV vaccine manufac-
Conclusion
turers’ trial registers. Searches of regulatory registers and
Authors of systematic reviews may recognize and reduce
journal publication databases contributed to a lesser ex-
reporting bias if they adopt our index process of study
tent. Steps 1, 3, 5 and 6 contributed mainly to the verifi-
programmes. We believe we came close to indexing
cation of some studies (see Additional files 1 and 3).
complete HPV vaccine study programmes, but only one
Our six-step process is reproducible, the step sequence
of the four HPV vaccine manufacturers provided infor-
is interchangeable and most steps could be performed
mation for our index and a fifth of the index could not
simultaneously. For example, we started by correspond-
be cross-verified. However, we indexed larger study pro-
ing with EMA, since we are familiar with EMA’s hand-
grammes than those listed by major regulators (i.e., the
ling of study programmes and clinical study reports [27].
EMA and FDA that based their HPV vaccine approvals
This correspondence helped us get started (but EMA re-
on only half of the available trials). To reduce reporting
sponse times may prove very slow and EMA often de-
bias in systematic reviews, we advocate the registration
nies data requests (18)). The index took approximately
and publication of all studies and data in the public do-
3 months to assemble. Researchers may save time if they
main and that non-industry studies register and adhere
perform the steps simultaneously and focus on steps 2
to reporting guidelines similar to the ICH.
and 6. For example, researchers could start requesting
the drug manufacturers’ study programmes and subse-
quently make an independent index and compare the Additional files
two. However, correspondence with manufacturers may
prove challenging and slow. Only one of the four HPV Additional file 1: Index of the HPV vaccines clinical studies: Search
strategy for identifying the HPV vaccines industry study programmes and
vaccine manufacturers (GlaxoSmithKline) provided us non-industry funded clinical studies. (DOC 1294 kb)
with study programme information, which we received Additional file 2: Index of the HPV vaccines clinical studies:
9 months after our initial request. Merck Sharp and Correspondence with the HPV vaccine manufacturers for the assessment
Dohme responded to our enquiry, but did not provide of the accuracy of our indexed industry study programmes. (DOC 151 kb)
study programme information. Shanghai Zerun Biotech- Additional file 3: Index of the HPV vaccines clinical studies: Indexes
of the identified industry study programmes and non-industry funded
nology and Xiamen Innovax Biotech did not respond to clinical studies and a list of the identified corresponding journal pub-
our inquiries (Additional files 1, 2, 3, and 5). lications. (DOC 659 kb)
Compared to industry studies, non-industry funded Additional file 4: Prisma 2009 checklist. (DOCX 147 kb)
studies were registered less often (for example, on Clinical- Additional file 5: Data sharing agreement with GlaxoSmithKline.
Trials.gov) and posted less study results. Non-industry re- (PDF 301 kb)
searchers are not legally required to register their studies,
adhere to industry reporting guidelines (the International Abbreviations
Conference on Harmonization of Technical Requirements CTD: Common technical document; DAP: Drug approval package;
for Registration of Pharmaceuticals for Human Use, ICH: EMA: European medicines agency; EPAR: European public assessment report;
FDA: Food and Drug Administration; GSK: GlaxoSmithKline plc; HPV: Human
http://www.ich.org/) or produce clinical study reports un- papillomavirus; ICH: International conference on harmonization of technical
less their results are used to support a drug’s marketing requirements for registration of pharmaceuticals for human use;
authorization application. This involves a high risk of MAA: Marketing authorization application; MAH: Market authorization holder;
Merck: Merck Sharp & Dohme (or Merck & Co., Inc.); NCT: National clinical
reporting bias. Therefore, we recommend that non- trial
industry funders require researchers to register their stud-
ies and commit to reporting guidelines similar to the ICH. Acknowledgements
Finally, although study programme and clinical study re- We would like to thank EMA and GlaxoSmithKline for providing information
port access from the industry and regulators have im- for our index.
proved since 2010 [17], access is often slow and inefficient
[18]. In May 2014, one of us (TJ) requested the HPV vac- Funding
Our study received no funding or grant other than salary to the authors and
cine clinical study reports from EMA. The request was associated expenses provided by the Nordic Cochrane Centre.
initially declined by EMA because, “disclosure would
undermine the protection of commercial interests”. TJ Availability of data and materials
successfully appealed, but EMA has only released 18 All raw data underlying our study is available upon request.
Page 105 of 637

Authors’ contributions confidential and published data on the safety and effectiveness of rhBMP-2
LJ and TJ contributed to the conception of the study and its design, for spinal fusion. BMJ. 2013;346:f3981.
collection and assembly of data, analysis and interpretation of the data, 10. Hughes S, Cohen D, Jaggi R. Differences in reporting serious adverse events
drafting of the article, critical revision of the article for important intellectual in industry sponsored clinical trial registries and journal articles on
content, and final approval of the article. PCG contributed to the conception antidepressant and antipsychotic drugs: a cross-sectional study. BMJ Open.
of the study, critical revision of the article for important intellectual content, 2014;4(7):e005535.
and final approval of the article. All authors read and approved the final 11. Belknap SM, Aslam I, Kiguradze T, Temps WH, Yarnold PR, Cashy J, et al.
manuscript. Adverse event reporting in clinical trials of Finasteride for androgenic
alopecia: a meta-analysis. JAMA Dermatol. 2015;151(6):600–6.
Ethics approval and consent to participate 12. Hodkinson A, Gamble C, Smith CT. Reporting of harms outcomes: a
Not applicable. comparison of journal publications with unpublished clinical study reports
of orlistat trials. Trials. 2016;17(1):207.
Consent for publication 13. Schroll JB, Penninga EI, Gøtzsche PC. Assessment of adverse events in
Not applicable. protocols, clinical study reports, and published papers of trials of Orlistat: a
document analysis. PLoS Med [Internet]. 2016 [cited 14 Nov 2016];13(8).
Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4987052/
Competing interests
14. Golder S, Loke YK, Wright K, Norman G. Reporting of adverse events in
LJ and PCG have no conflicts of interest to declare. TJ is occasionally
published and unpublished studies of health care interventions: a
interviewed by market research companies about phase I or II
systematic review. PLoS Med. 2016 Sep;13(9):e1002127.
pharmaceutical products. In 2011–14, TJ acted as an expert witness in a
litigation case related to the antiviral oseltamivir, in two litigation cases on 15. Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJ, et al.
potential vaccine-related damage and in a labour case on influenza vaccines Neuraminidase inhibitors for preventing and treating influenza in adults and
in healthcare workers in Canada. He has acted as a consultant for Roche children. In: Cochrane database of systematic reviews [internet]. John Wiley
(1997–99), GlaxoSmithKline (2001–2), Sanofi-Synthelabo (2003) and IMS & Sons, Ltd; 2014 [cited 14 Nov 2016]. Available from: http://onlinelibrary.
Health (2013). In 2014–16, TJ was a member of three advisory boards for wiley.com/doi/10.1002/14651858.CD008965.pub4/abstract
Boehringer Ingelheim and is holder of a Cochrane Methods Innovations 16. Sharma T, Guski LS, Freund N, Gøtzsche PC. Suicidality and aggression
Fund grant to develop guidance on the use of regulatory data in Cochrane during antidepressant treatment: systematic review and meta-analyses
reviews. TJ was a member of an independent data monitoring committee based on clinical study reports. BMJ. 2016;352:i65.
for a Sanofi Pasteur clinical trial on an influenza vaccine. TJ was a co- 17. Gøtzsche PC, Jørgensen AW. Opening up data at the European medicines
signatory of the Nordic Cochrane Centre’s complaint to EMA over maladmin- agency. BMJ. 2011;342:d2686.
istration at EMA in relation to the investigation of alleged harms of HPV vac- 18. Doshi P, Jefferson T. Open data 5 years on: a case series of 12 freedom of
cines and consequent complaints to the European Ombudsman. TJ is also information requests for regulatory data to the European medicines agency.
sole author of a complaint to the European Ombudsman over slowness and Trials [Internet]. 2016 [cited 14 Nov 2016];17. Available from: http://www.
redactions of clinical study reports of HPV vaccine released by EMA. ncbi.nlm.nih.gov/pmc/articles/PMC4750208/
19. Jefferson T, Doshi P, Thompson M, Heneghan C. Ensuring safe and effective
drugs: who can do what it takes? BMJ. 2011;342:c7258.
Publisher’s Note 20. Gøtzsche PC. Why we need easy access to all data from all clinical trials and
Springer Nature remains neutral with regard to jurisdictional claims in how to accomplish it. Trials. 2011;12:249.
published maps and institutional affiliations. 21. van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for
unpublished trials in Cochrane reviews may not be worth the effort. J Clin
Received: 13 April 2017 Accepted: 8 January 2018 Epidemiol. 2009 Aug;62(8):838–844.e3.
22. Golder S, Loke YK, Wright K, Sterrantino C. Most systematic reviews of adverse
effects did not include unpublished data. J Clin Epidemiol. 2016;77:125–33.
References 23. Baudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of
1. McGauran N, Wieseler B, Kreis J, Schüler Y-B, Kölsch H, Kaiser T. Reporting searching clinical trial registries in systematic reviews of pharmaceutical
bias in medical research - a narrative review. Trials. 2010;11:37. treatments: methodological systematic review and reanalysis of meta-
2. Ross JS, Tse T, Zarin DA, Xu H, Zhou L, Krumholz HM. Publication of NIH analyses. BMJ. 2017;356:j448.
funded trials registered in ClinicalTrials.Gov: cross sectional analysis. BMJ. 24. Jones CW, Keil LG, Weaver MA, Platts-Mills TF. Clinical trials registries are
2012 Jan 3;344:d7292. under-utilized in the conduct of systematic reviews: a cross-sectional
3. Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and analysis. Syst Rev. 2014;3:126.
conclusions in randomized drug trials: a reflection of treatment effect or 25. Potthast R, Vervölgyi V, McGauran N, Kerekes MF, Wieseler B, Kaiser T.
adverse events? JAMA. 2003;290(7):921–8. Impact of inclusion of industry trial results registries as an information
4. Eyding D, Lelgemann M, Grouven U, Härter M, Kromp M, Kaiser T, et al. source for systematic reviews. PloS One. 2014;9(4):e92067.
Reboxetine for acute treatment of major depression: systematic review and 26. van Enst WA, Scholten RJ, Hooft L. Identification of additional trials in
meta-analysis of published and unpublished placebo and selective prospective trial registers for Cochrane systematic reviews. PLOS One. 2012;
serotonin reuptake inhibitor controlled trials. BMJ. 2010;341:c4737. 7(8):e42812.
5. Lundh A, Lexchin J, Mintzes B, Schroll JB, Bero L. Industry sponsorship and 27. Jørgensen L, Gøtzsche PC, Jefferson T. PROSPERO: Benefits and harms
research outcome. In: Cochrane database of systematic reviews [internet]. of the human papillomavirus vaccines: systematic review of industry
John Wiley & Sons, Ltd; 2017 [cited 3 Apr 2017]. Available from: http:// and nonindustry study reports [Internet]. Available from: Web: http://
onlinelibrary.wiley.com/doi/10.1002/14651858.MR000033.pub3/abstract www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42017056093.
6. Wieseler B, Kerekes MF, Vervoelgyi V, McGauran N, Kaiser T. Impact of document PDF: https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_
type on reporting quality of clinical drug trials: a comparison of registry reports, 20170030.pdf
clinical study reports, and journal publications. BMJ. 2012;344:d8141. 28. Paavonen J, Naud P, Salmerón J, Wheeler CM, Chow S-N, Apter D, et al.
7. Maund E, Tendal B, Hróbjartsson A, Jørgensen KJ, Lundh A, Schroll J, et al. Efficacy of human papillomavirus (HPV)-16/18 AS04-adjuvanted vaccine
Benefits and harms in clinical trials of duloxetine for treatment of major against cervical infection and precancer caused by oncogenic HPV types
depressive disorder: comparison of clinical study reports, trial registries, and (PATRICIA): final analysis of a double-blind, randomized study in young
publications. BMJ. 2014;348:g3510. women. Lancet. 2009;374(9686):301–314.
8. Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. Selective 29. Interim Clinical Study Report for Study 580299/008 (HPV-008) [Internet].
reporting bias of harm outcomes within studies: findings from a cohort of Available from: https://www.gsk-clinicalstudyregister.com/files2/gsk-580299-
systematic reviews. BMJ. 2014;349:g6501. 008-clinical-study-report-redact.pdf
9. Rodgers MA, Brown JVE, Heirs MK, Higgins JPT, Mannion RJ, Simmonds MC, 30. European Medicines Agency. Assessment report review under article 20 of
et al. Reporting of industry funded study outcome data: comparison of regulation (EC) no 726/2004 human papillomavirus (HPV) vaccines [internet].
Page 106 of 637

[cited 14 Nov 2016 ]. Available from: http://www.ema.europa.eu/docs/en_

GB/document_library/Referrals_document/HPV_vaccines_20/Opinion_
provided_by_Committee_for_Medicinal_Products_for_Human_Use/
WC500197129.pdf
31. Jefferson T, Jørgensen L. Human papillomavirus vaccines, complex regional
pain syndrome, postural orthostatic tachycardia syndrome, and autonomic
dysfunction – a review of the regulatory evidence from the European
Medicines Agency. Indian J Med Ethics. [S.l.], v. 2, n. 1 (NS), p. 30, nov. 2016.
ISSN 0975-5691. Available at: Accessed 13 Jan 2018.
32. Lenzer JFDA. Is incapable of protecting US “against another Vioxx”. BMJ.
2004;329(7477):1253.
33. Gøtzsche PC. Deadly medicines and organised crime: how big pharma has
corrupted health care. London: Radcliffe Publishing; 2013.
34. Cohen D. Rosiglitazone: what went wrong? BMJ. 2010;341:c4848.
Submit your next manuscript to BioMed Central

and we will help you at every step:
• We accept pre-submission inquiries
• Our selector tool helps you to find the most relevant journal
• We provide round the clock customer support
• Convenient online submission
• Thorough peer review
• Inclusion in PubMed and all major indexing services
• Maximum visibility for your research
Submit your manuscript at

www.biomedcentral.com/submit
Page 107 of 637

Appendix 1: Index of the HPV vaccines clinical studies: Search strategy for identifying the HPV vaccine
industry study programmes and non-industry funded clinical studies
The search strategy for the HPV vaccines studies was developed iteratively in six steps. Steps 2 and 4
contributed quantitatively the most to the identification of studies—in particular, searches of
ClinicalTrials.gov and the HPV vaccine manufacturers trial registers. Searches of regulatory registers and
journal publication databases contributed to a lesser extent. Steps 1, 3, 5 and 6 contributed mainly in the
verification of some studies (see Appendix 3).
- Summary of identified studies/hits with each of the six search steps:
Step Source Number of studies/hits

1 Correspondence with the EMA and inclusion of the HPV vaccines Market Authorisation Applications 20
2 Searches of industry, public and regulatory trial registers 3,401
3 Inclusion of Drug Approval Packages and European Public Assessment Reports from the FDA and the EMA 34
4 Searches on CENTRAL, Google Scholar, MEDLINE and WikiLeaks 1,646
5 Inclusion of HPV vaccine studies listed on recent regulatory reviews 80
Total 5,181
6 The HPV vaccine manufacturers assessment of the study indexes accuracy: 85
N= studies in the Cervarix index that GlaxoSmithKline provided us with
Step 1: Correspondence with the EMA and inclusion of the HPV vaccines’ Market Authorisation Applications
We included studies listed in correspondence with the EMA and studies listed on the HPV vaccines Market
Authorisation Applications.
- Summary of identified studies/hits in Step 1:

1a Correspondence with the EMA 13
1b The HPV vaccines Market Authorisation Applications 7
Total hits 20
Step 1a: Correspondence with EMA

From May 2014 we corresponded with and obtained HPV vaccine clinical study reports from the EMA. For
example, we requested the EMA for a list of the completed HPV vaccine studies listed on the EMA’s
pharmacovigilance plan. The following is an email from the EMA dated 18 October 2016 (some of the
studies that EMA listed had very opaque titles, e.g., ‘Protocol using the PGRx’ and ‘System Pregnancy
Registry’):
“Dear Dr Tom Jefferson, The list of completed studies from the pharmacovigilance plan with the final report submitted for regulatory review is as follows: Cervarix:
Study HPV-008: A phase III, double-blind, randomized, controlled, multicentre study to evaluate the efficacy of GlaxoSmithKline Biologicals’ HPV-16/18 VLP/AS04 vaccine
compared to hepatitis A vaccine as control in prevention of persistent HPV-16 or HPV-18 cervical infection and cervical neoplasia, administered intramuscularly
according to a 0, 1, 6 month schedule in healthy females 15-25 years of age.
Study HPV-009: A double-blind, controlled, randomized, phase III study of the efficacy of an HPV-16/18 VLP vaccine in the prevention of advanced cervical intraepithelial
neoplasia (CIN2, CIN3, adenocarcinoma in situ [AIS] and invasive cervical cancer) associated with HPV-16 or HPV-18 cervical infection in healthy young adult women in
Costa Rica
Study HPV-020: A phase I/II, partially-blind, randomized, controlled study to assess the safety and immunogenicity of GlaxoSmithKline Biologicals’ HPV-16/18 L1 VLP
AS04 vaccine administered intramuscularly according to a three-dose schedule (0, 1, 6-month) in human immunodeficiency virus (HIV)-infected female subjects aged 18-
25 years (South Africa)
EPI-HPV-020: An observational cohort study assessing the risk of spontaneous abortions during the first 23 weeks of gestation in women aged 15 to 25 years exposed to
Cervarix, residing in the United Kingdom and reporting their last menstrual period between 30 days before and 45 days after any dose of Cervarix
EPI-HPV-018 (PASS): An observational cohort study assessing the risk of spontaneous abortions during the first 23 weeks of gestation in women aged 15 to 25 years
exposed to Cervarix, residing in the United Kingdom and reporting their last menstrual period between 30 days before and 45 days after any dose of Cervarix
EPI-HPV-015 : A post-marketing observational safety study of autoimmune diseases following GlaxoSmithKline (GSK) Biologicals’ HPV-16/18 L1 VLP AS04 vaccine
(Cervarix®) vaccination in females aged 9-25 years enrolled in United States health plans
EPI-HPV-040 (PASS study): (replacing study HPV-015). An observational cohort study to assess the risk of autoimmune diseases in adolescent and young adult women
aged 9 to 25 years exposed to Cervarix® in the United Kingdom (using the CPRD GOLD data source in the UK)
Study HPV-015: A phase III, double-blind, randomized, controlled study to evaluate the safety, immunogenicity and efficacy of GlaxoSmithKline Biologicals’ HPV-16/18
L1/AS04 vaccine administered intramuscularly according to a three-dose schedule (0, 1, 6 month) in healthy adult female subjects aged 26 years and above
Study HPV-040: A phase III/IV, community-randomized, controlled study to evaluate the effectiveness of two vaccination strategies using GlaxoSmithKline Biologicals’
HPV-16/18 L1 VLP AS04 vaccine in reducing the prevalence of HPV-16/18 infection when administered intramuscularly according to a 0, 1, 6-month schedule in healthy
female and male study participants aged 12 – 15 years.
Gardasil
Long term follow-up study in adolescents (P018-11)
Post-Licensure Safety study in Females (P031)
Protocol using the PGRx System
Pregnancy Registry
Vaccination Impact in Population Study (P033)
Gardasil 9
No studies from the pharmacovigilance plan have been completed yet. No interim results from studied in the pharmacovigilance plan have been submitted for
regulatory review."
Page 108 of 637

Step 1b: Inclusion of the HPV vaccines Market Authorisation Applications
We requested the EMA for the Market Authorisation Applications (MAAs) for the HPV vaccines. We
obtained the Market Authorisation Applications for Gardasil 9, but we did not obtain the Market
Authorisation Applications for Cervarix or Gardasil from EMA. Module 2.5 of the Common Technical
Document (CTD) of Gardasil 9 (EMA reference: GARDASIL 9, EMA ASK 21232, first batch release) was sent
to us by the EMA on 10 November 2016. The following screenshot is a list from Module 2.5 with the studies
that contributed to the safety dataset of Gardasil 9 studies:
Step 2: Searches on industry, public and regulatory trial registers

We searched industry, public and regulatory trial registers for HPV vaccine studies.

2a Industry trial registers 404
2b Public trial registers 1,672
2c Regulatory registers 1,325
Total hits 3,401
Step 2a: Searches on industry trial registers

We searched an industry trial register if the manufacturer had been involved (or possibly involved) in one
or more HPV vaccine studies:
Industry register searched Search phrase or term used Date of search Number of hits
CSDR: https://clinicalstudydatarequest.com/ “Human Papillomavirus Types 16 and 18 Vaccine” [this 7 July 2017 57
was the only eligible search option]
GlaxoSmithKline: https://gsk-clinicalstudyregister.com/ "Human papillomavirus vaccine", “HPV vaccine”, 7 July 2017 227
“Cervarix”, “Gardasil”
Inovio Pharmaceuticals: http://www.inovio.com/ "Human papillomavirus vaccine", “HPV vaccine” 7 July 2017 56
Merck Sharp & Dohme: http://www.merck.com/clinical-trials/ "Human papillomavirus vaccine", “HPV vaccine”, 7 July 2017 64
Sanofi Pasteur: http://www.sanofipasteur.com/ "Human papillomavirus vaccine", “HPV vaccine”, 7 July 2017 0
Serum Institute of India Pvt. Ltd: Not applicable [no search option on this website. We did 7 July 2017 Unknown
http://www.seruminstitute.com/ not contact the manufacturer as no identified HPV vaccine
study was funded by the manufacturer]
Shanghai Zerun Biotechnology Co., Ltd: Not applicable [no search option on this website. We 7 July 2017 Unknown
http://www.zerunbio.com/ contacted this manufacturer as the manufacturer had
funded several HPV vaccine studies]
Xiamen Innovax Biotech Co., Ltd: http://www.innovax.cn/ "Human papillomavirus vaccine", “HPV vaccine” 7 July 2017 0
Total 404
Page 109 of 637

Step 2b: Searches on public trial registers
We searched the following 32 international and regional trial registers for HPV vaccine studies chosen
according to their level of impact (e.g., https://clinicaltrials.gov and http://apps.who.int/trialsearch/ are
used internationally and were considered high impact) and where one or more HPV vaccine studies had
been conducted (for example, we searched the Chinese Clinical Trial Registry:
http://www.chictr.org.cn/index.aspx, since several HPV vaccine studies had been conducted in China):
International trial registers

Trial register Search phrase or term used Date of search Number of hits
CenterWatch: http://www.centerwatch.com/ Human Papilloma Virus (HPV) Clinical Trials [this was the 9 July 2017 5
only eligible search option]
Clinical Trial Facts: http://www.clinicaltrialfacts.com/ "Human papillomavirus vaccine", “HPV vaccine”, 9 July 2017 87
Clinical Trials Feeds: http://clinicaltrialsfeeds.org/ Not applicable [no eligible search option] 9 July 2017 0
ClinicalTrials.gov: https://clinicaltrials.gov/home “Bivalent HPV OR quadrivalent HPV OR HPV vaccine OR 9 July 2017 421
human papillomavirus vaccine OR Cervarix OR Gardasil”
NHS Choices: http://www.nhs.uk/Conditions/Clinical- "HPV vaccination" [only eligible search option] 9 July 2017 102
trials/Pages/clinical-trial.aspx
Patients Like Me: "Condition: Papillomavirus [intervention type: biologic]” 9 July 2017 118
https://www.patientslikeme.com/clinical_trials
The ISRCTN Registry: http://www.isrctn.com/ "Human papillomavirus vaccine", “HPV vaccine”, 9 July 2017 18
Trials Central: http://www.trialscentral.org/ Not applicable [no search option on this website] 9 July 2017 0
VirtualTrials.com: Not applicable [no eligible search option] 9 July 2017 0
https://virtualtrials.com/clinical_trials_finder.cfm
WHO International Clinical Trials Registry Platform (ICTRP): Advanced search: Intervention: “Human papillomavirus 9 July 2017 531
http://apps.who.int/trialsearch/ vaccine OR HPV vaccine OR Cervarix OR Gardasil”
African trial registers
Pan African Clinical Trials Registry: http://www.pactr.org/ "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 6
South African National Clinical Trial Register: Not applicable [no eligible search options] 11 July 2017 0
http://www.sanctr.gov.za/
Asian trial registers
Chinese Clinical Trial Registry: Not applicable [this website did not respond to any 11 July 2017 Unknown
http://www.chictr.org.cn/index.aspx searches]
India’s Clinical Trials Registry: “Cervarix” OR “Gardasil” OR “HPV vaccine” AND 11 July 2017 8
http://ctri.nic.in/Clinicaltrials/login.php "Intervention and comparator agent"
Japanese Clinical Trial Registry: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 7
http://rctportal.niph.go.jp/en/ “Cervarix”, “Gardasil”
Korean National Clinical Research Coordination Centre: Not applicable [this website is not accessible from a 11 July 2017 Unknown
http://ncc.re.kr/ non-Korean server]
Sri Lankan Clinical Trials Registry: http://www.slctr.lk/ "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 0
European and Middle Eastern trial registers
European Clinical Trials Register: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 148
https://www.clinicaltrialsregister.eu/index.html “Cervarix”, “Gardasil”
German Clinical Trials Register: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 169
https://www.drks.de/drks_web/ “Cervarix”, “Gardasil”
Iranian Registry of Clinical Trials: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 0
http://www.irct.ir/searchen.php “Cervarix”, “Gardasil”
Nederlands Trial Register: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 0
http://www.trialregister.nl/trialreg/index.asp “Cervarix”, “Gardasil”
United Kingdom Cancer Research: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 0
http://www.cancerresearchuk.org/about-cancer/find-a- “Cervarix”, “Gardasil”
clinical-trial
United Kingdom’s Clinical Trials Gateway: "The system will search for clinical trials from two 11 July 2017 Not applicable
https://www.ukctg.nihr.ac.uk/ sources: the ClinicalTrials.gov register; and the ISRCTN."
[ClinicalTrials and ISRCTN were already included]
North American trial registers
ClinicalTrials.com: http://www.clinicaltrials.com/ "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 0
Health Canada: https://www.canada.ca/en/health- "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 3
canada/ “Cervarix”, “Gardasil”
Trial X: http://trialx.com/ "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 13
Page 110 of 637

Oceania trial registers
Australian Clinical Trials: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 4
https://www.australianclinicaltrials.gov.au/ “Cervarix”, “Gardasil”
Australian and New Zealand Clinical Trials Registry: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 32
http://www.anzctr.org.au/default.aspx “Cervarix”, “Gardasil”
New Zealand Clinical Trials Portal: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 0
http://clinicaltrials.health.nz/ “Cervarix”, “Gardasil”
South American trial registers
Brazilian Clinical Trials Registry: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 0
http://www.ensaiosclinicos.gov.br/ “Cervarix”, “Gardasil”
Cuban Public Registry of Clinical Trials: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 0
http://registroclinico.sld.cu/en/home “Cervarix”, “Gardasil”
South American (SciELO) Public Health Trial Register: "Human papillomavirus vaccine", “HPV vaccine”, 11 July 2017 0
http://www.scielosp.org/scielo.php?lng=en “Cervarix”, “Gardasil”
Total hits 1,672
Step 2c: Searches on regulatory registers

We searched a regulatory register if the regulator had been involved (or possibly involved) in the
assessment or approval of one or more of the HPV vaccines:
Register searched Search phrase or term used Date of search Number of hits
Chinese Food and Drug Administration: http://eng.sfda.gov.cn/WS03/CL0755/ "Human papillomavirus vaccine", 13 July 2017 52
“HPV vaccine”, “Cervarix”, “Gardasil”
European Medicines Agency: http://www.ema.europa.eu/ema/ “HPV vaccine OR Cervarix OR 13 July 2017 408
Gardasil AND study OR trial"
India’s Central Drugs Control Organization: "Human papillomavirus vaccine", 13 July 2017 1
http://cdsco.nic.in/forms/Default.aspx “HPV vaccine”, “Cervarix”, “Gardasil”
United Kingdom Medicines & Healthcare products Regulatory Agency: "Human papillomavirus vaccine OR 13 July 2017 274
https://www.gov.uk/government/organisations/medicines-and-healthcare- HPV vaccine OR Cervarix OR
products-regulatory-agency Gardasil”
United States Food and Drug Administration: http://www.fda.gov/ "Human papillomavirus vaccine", 13 July 2017 590
“HPV vaccine”, “Cervarix”, “Gardasil”
Total 1,325
Step 3: Drug Approval Packages and European Public Assessment Reports from the FDA and the EMA
We searched the HPV vaccines’ Drug Approval Packages and European Public Assessment Reports from the
FDA and EMA for HPV vaccine studies:
The FDA’s Drug Approval Packages

HPV vaccine Source Date of search Number of studies
Cervarix https://www.fda.gov/BiologicsBloodVaccines/Vaccines/ApprovedProducts/ucm186957.htm 14 July 2017 17
Gardasil https://www.fda.gov/biologicsbloodvaccines/vaccines/approvedproducts/ucm094042 14 July 2017 7
Gardasil 9 https://www.fda.gov/biologicsbloodvaccines/vaccines/approvedproducts/ucm426445.htm 14 July 2017 10
The EMA’s European Public Assessment Reports
HPV vaccine Source Date of search Number of studies
Cervarix http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_- 14 July 2017 Unknown*
_Summary_for_the_public/human/000721/WC500024634.pdf
Gardasil http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_- 14 July 2017 Unknown*
Gardasil 9 http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_- 14 July 2017 Unknown*
Total studies 34
*On the EMA’s website (http://www.ema.europa.eu/ema/) there was only access to “public” European Public Assessment Report summaries,
although EMA states that, "The full EPAR [European Public Assessment Report] and risk management plan summary...can be found on the agency’s
[EMA’s] website."
(http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_Summary_for_the_public/human/003852/WC500189114.pdf). We have
requested the “full” EPARs from the EMA (reference ASK-32128), but we have not received any “full” EPARs yet.
Step 4: Searches on CENTRAL, Google Scholar, MEDLINE and WikiLeaks
Page 111 of 637

We searched the following databases to identify HPV vaccine studies not listed on industry, public and
regulatory trial registers:
Database searched Search phrase or term used Date of search Number of hits
CENTRAL: http://onlinelibrary.wiley.com/cochranelibrary/ "Papillomavirus Vaccines"[Mesh] AND “Trial”[Mesh] 15 July 2017 241
Google Scholar: https://scholar.google.com “Cervarix” OR “Gardasil” OR “HPV vaccine” AND 15 July 2017 1,000* [7,310]
“Clinical Trial”
MEDLINE: https://www.ncbi.nlm.nih.gov/pubmed "Papillomavirus Vaccines"[Mesh] AND "Clinical Trial" 15 July 2017 302
[Publication Type]
WikiLeaks: https://wikileaks.org/ "Human papillomavirus vaccine OR HPV vaccine OR 15 July 2017 103
Cervarix OR Gardasil” [We accepted: "Include
external sources: Associated Twitter accounts,
Snowden + Hammond Documents, Cryptome
Documents, ICWatch, This Day in WikiLeaks Blog and
WikiLeaks Press, WL Central"]
Total hits 1,646
*We used the “sorted by relevance” function in Google Scholar and assessed 1,000 of 7,310 hits. We stopped assessing hits after N= 1000, since
there had been no relevant entries for approximately 300 hits.
Step 5: Inclusion of studies listed on recent regulatory reviews and crosschecking of study IDs
We included HPV vaccine studies from two recent regulatory reviews and summarised Steps 1 to 5 by
crosschecking the identified study programme IDs.
Step Source Number of studies

5a EMAs review* of postural orthostatic tachycardia syndrome (POTS) and complex regional pain 39
syndrome (CRPS)
5b Independent review** with studies that the EMA used in their review of postural orthostatic 41
tachycardia syndrome (POTS) and complex regional pain syndrome (CRPS)
5c Crosschecking of study programme IDs Not applicable
Total studies 80
*“Review under Article 20 of Regulation (EC) No 726/2004 Human papillomavirus (HPV) vaccines.” Available from:
http://www.ema.europa.eu/docs/en_GB/document_library/Referrals_document/HPV_vaccines_20/Opinion_provided_by_Committee_for_Medici
nal_Products_for_Human_Use/WC500197129.pdf
**Jefferson T, Jørgensen L. Human papillomavirus vaccines, complex regional pain syndrome, postural orthostatic tachycardia syndrome, and
autonomic dysfunction - a review of the regulatory evidence from the European Medicines Agency. Indian J Med Ethics Publ Online Oct 17 2016.
Available from: http://ijme.in/articles/human-papillomavirus-vaccines-complex-regional-pain-syndrome-postural-orthostatic-tachycardia-
syndrome-and-autonomic-dysfunction-a-review-of-the-regulatory-evidence-from-the-european-medi/?galley=html
Step 5a:
In 2015, GlaxoSmithKline provided the EMA with a study list of Cervarix studies to be included in the EMA
review of postural orthostatic tachycardia syndrome (POTS) and complex regional pain syndrome (CRPS):
Page 112 of 637

In 2015, Merck Sharp & Dohme provided the EMA with a study list of Gardasil studies to be included in the
EMA review of postural orthostatic tachycardia syndrome (POTS) and complex regional pain syndrome
(CRPS):
Step 5b: Recent reviews on The HPV vaccines

We identified studies listed in recent reviews of the HPV vaccines. In 2016, an independent review included
a list of the Cervarix studies that the EMA and GlaxoSmithKline used in their review of postural orthostatic
tachycardia syndrome (POTS) and complex regional pain syndrome (CRPS):
Page 113 of 637

In 2016, an independent review included a list of the Gardasil studies that the EMA and Merck Sharp &
Dohme used in their review of postural orthostatic tachycardia syndrome (POTS) and complex regional pain
syndrome (CRPS):
Step 5c: Crosschecking of study identifiers

To summarize Steps 1 to 5 we crosschecked the identified industry study programme IDs. We reasoned
that these IDs would be allocated in a progressive chronological way, since the main industry HPV vaccine
studies are indexed as: “311-HPV-xxxx,” “HPV-xxx,” “HPV-PRO-xxx” and “V50x-xxx.” Thus, we searched
https://www.google.com for the following IDs:
1. GlaxoSmithKline: “HPV-001,” “HPV-002,” […], and “HPV-200.”
2. Merck Sharp & Dohme: “V501-001,” “V501-002,” […], and “V501-200.”
7. Shanghai Zerun Biotechnology Co., Ltd.: “311-HPV-1001,” “311-HPV-1002,” […] “311-HPV-1050.”
8. Xiamen Innovax Biotech Co., Ltd.: “HPV-PRO-001,” “HPV-PRO-002,” […] “HPV-PRO-050.”
We did not note the number of hits in this step, since this was merely a checking of the studies identified
and the studies that could not be identified in a chronologic order, for example, we identified study “HPV-
005” and study “HPV-007,” but we did not identify study “HPV-006.”
Step 6: The HPV vaccine manufacturers assessment of the study indexes accuracy
We sent the indexes of the HPV vaccines studies (gathered in the search Steps 1 to 5) to their respective
manufacturers for their assessment of the indexes accuracy and requested the manufacturers to confirm
studies that we indexed and add any missing studies (see Appendix 2):
Manufacturer HPV vaccine Number of studies/hits Number of indexed studies Number of studies
sent to us by the confirmed by the added to the index by
manufacturers manufacturers the manufacturers
GlaxoSmithKline Cervarix (bivalent) 0 66 of 69 studies 0
Merck Sharp & Dohme Gardasil (mono-, quadri-, octa- 0 0 of 66 studies 0
and ninevalent)
Shanghai Zerun Biotechnology HPV vaccine (bi- and 0 0 of 4 studies 0
quadrivalent)
Xiamen Innovax Biotech HPV vaccine (bivalent) 0 0 of 6 studies 0
Total 0 66 of 145 studies 0
Page 114 of 637

Appendix 2: Index of the HPV vaccines clinical studies: Correspondence with the HPV vaccine
manufacturers for the assessment of the accuracy of our indexed industry study programmes
The GlaxoSmithKline* Cervarix study programme

*All GlaxoSmithKline employees’ names have been changed to ‘GSK’ and all employee email addresses have been deleted or changed to
GSK@gsk.com from the correspondence.
GSK <GSK@gsk.com> 9 November 2016 at 13:44

To: "lj@cochrane.dk" <lj@cochrane.dk>
Dear Dr. Jørgensen,
I have received your request below and thank you for your interest in our studies. However, in order to best address your question could
you please provide some more clarity what you expect from us, especially for which parts in the Excel sheet that you shared you require our
input?
Information about the GSK trials is published on https://www.gsk-clinicalstudyregister.com and https://clinicaltrials.gov/. The information
on these websites is updated regularly and should be the main source for your information and quality checks.
Would you be able to provide some more detail about the scope of the exploratory review of regulatory documents that you mention in
your mail?
Thanks in advance.
Best regards,
GSK
GSK
Global Medical Affairs HPV
Vaccine Value & Health Science
GSK Vaccines
Expéditeur: <lj@cochrane.dk>
Date: 8 novembre 2016 à 11:51:55 UTC+1
Destinataire: <GSK@gsk.com>
Cc: <jefferson.tom@gmail.com>
Objet: Cervarix trial programme
Dear GSK,
We are writing to you to ask for your help in creating an accurate up-to-
date list of the clinical trials (and their follow-ups) for
GlaxoSmithKlines' human papillomavirus (HPV) vaccine: Cervarix™.
We are at present working on a protocol for an exploratory review of

regulatory documents (mainly clinical study reports) of HPV vaccines.
As a preliminary to the review and to size the task, we are making a list
of all known HPV vaccine trials.
We kindly ask you to assess the attached list of currently identified

Cervarix™ trials performed by GlaxoSmithKline and suggest any corrections
and/or edits. (The trials marked in yellow are conjectural, for which we
could find little information.)
Your help will be highly appreciated and acknowledged in any publication.
We would be grateful to receive our assessments before 9 December 2016.
Thank you for your time,

We look forward to your answer.
Best wishes,
PhD student Lars Jørgensen, MD
Honorary research fellow Tom Jefferson, MD
Centre director Peter C. Gøtzsche, Professor
PS: If you are not the right person to deal with this request we would
appreciate if you would pass the request on to a colleague appropriate for
the task.
Cervarix trial programme.xlsx
103K
Page 115 of 637

Lars Jørgensen <larsjorgensens@gmail.com> 9 November 2016 at 17:58
To: GSK <GSK@gsk.com>
Cc: lj@cochrane.dk
Dear GSK,
Thank you for your quick response and enquiry.
We have searched both https://www.gsk-clinicalstudyregister.com and https://clinicaltrials.gov/ for available Cervarix trials and follow up
studies.
To clarify, we hope that you-:

1) can confirm that all the trials (and follow up studies) in our list exist.
2) can add any trials to the list that we have missed in our inclusion.
3) can add further details to listed trials (e.g. if stated NA=not available/applicable in the list) and to any missing trials.
The scope of the exploratory review of regulatory documents is to outline the resources required to carry out a systematic review of
comparative evidence of the effects of HPV vaccines.
We look forward to hear from you again.
Best wishes,
Lars Jørgensen
Tom Jefferson
Peter C. Gøtzsche

To: Lars Jørgensen <larsjorgensens@gmail.com>
Cc: "lj@cochrane.dk" <lj@cochrane.dk>
Thanks for the clarification.

As your questions concerns information that is not in the public domain, you may need to submit an official clinical study data request.
However, I need to check with the responsible person how to qualify your request and I will get back to you early next week.
Best regards,
GSK
From: Lars Jørgensen [mailto:larsjorgensens@gmail.com]

Sent: mercredi 9 novembre 2016 17:58
To: GSK
Cc: lj@cochrane.dk
Subject: Re: Your request to GSK

Dear GSK,
Much obliged.
Looking forward to hear from you.
Best wishes,
Lars

Dear GSK,
We wonder whether you have had time to consider our request for fact checking on the list you received the 9th of November.
With best wishes,

Tom and Lars
GSK<GSK@gsk.com> 18 November 2016 at 14:15
Page 116 of 637

Thank you again for your interest in our data.
GSK is committed to sharing research data with independent researchers and organisations through our SHARE initiative. As part of this
initiative, we require that all requests from independent researchers for information on GSK trials be submitted through the SHARE website
(https://clinicalstudydatarequest.com/).
Your request will be reviewed by an Independent review panel. The Independent Review Panel comprises external experts. It accepts or
rejects proposals based on the scientific rationale and relevance to medical science or patient care. Following approval of your proposal and
subsequent signing of a Data sharing agreement, we can share information on our research with you including the availability of data from
studies that are not listed on https://clinicaltrials.gov/ or https://www.gsk-clinicalstudyregister.com. Furthermore, you can have access to
patient level data as well, if needed in the course of the research project.
Additional details on the process are available on https://clinicalstudydatarequest.com/Step-By-Step.aspx and

on https://clinicalstudydatarequest.com/Study-Sponsors-GSK-Details.aspx.
May I propose that you submit your request through the above process? Please, do not hesitate to come back to us with any questions
concerning the submission.

Dear GSK.
Thank you for your response.
Your help is highly appreciated.
Our primary request is for you to simply confirm that all GSKs Cervarix studies are represented by a row in our spreadsheet (the list is mainly
constructed via clinicaltrials.gov and gsk.com).
We do not request any publications or data. But we would like to know if the SHARE agreement includes restrictions on publications.
Best wishes,
Lars

As I mentioned in my mail there are studies that are not listed on the websites. Information about these studies (including study design
elements, sample size etc….) would be considered information that needs to be requested via SHARE.
You can just submit your enquiry according to the process below. This ensures that your request is correctly channelled.
The SHARE agreement does not include any restriction on publications. On the contrary, upon completion researchers are expected to
publish their research in a peer-reviewed journal. GSK requests a copy of the publication after it has been submitted to a scientific congress
or journal to assess whether there are any patent implications.
Let me know if you have further questions.
Best regards,
GSK
Sent: mardi 22 novembre 2016 10:31
To: GSK

Dear GSK,
We have read the agreement (attached) and are not 100% comfortable with it.
We understand that the agreement requirement is due to point 2) of our previous request:
2) can add any trials to the list that we have missed in our inclusion.
Page 117 of 637

We therefore kindly ask you only to assess point 1) and 3) for the attached list of currently identified Cervarix™ trials and suggest any
corrections and/or edits:
We hope this is acceptable and look forward to your response.
Best wishes,
Lars
DATA-SHARING-AGREEMENT.pdf
302K

The SHARE platform is the central entry point for requests concerning the clinical trials. As I mentioned you can submit any request
according to the process that I have laid out and it does not matter if it is your question 1, 2 or 3. This will guarantee a correct review by our
experts and the necessity to set up an agreement will be determined based on the review.
If you have any further questions, we can also offer to set up a teleconference to discuss any concerns related to the process or agreement.
Best regards,
GSK

GSK Vaccines

Sent: lundi 28 novembre 2016 14:05
Lars Jørgensen <larsjorgensens@gmail.com> 17 July 2017 at 16:52

To: GSK@gsk.com, GSK <GSK@gsk.com>
Cc: Tom Jefferson <jefferson.tom@gmail.com>
Dear GSK,
We are writing to you to ask for your help in creating an accurate up-to-date list of the clinical trials (and their follow-ups) for
GlaxoSmithKline's human papillomavirus (HPV) vaccine: Cervarix.
We are at present working on a protocol for an exploratory review of regulatory documents (mainly clinical study reports) of HPV vaccines.
As a preliminary to the review and to size the task, we are making a list of all known interventional HPV vaccine trials that are prospective,
preventive and comparative.
We wrote to GlaxoSmithKline in November 2016 with a similar request, but we did not receive any conclusive or explicit feedback on our
index of Cervarix studies.
We kindly ask you to reconsider to assess the attached list of currently identified Cervarix trials and suggest any corrections and/or edits.
Best wishes,
HPV vaccine trials - GlaxoSmithKline - 17 July.xlsx

88K
GSK <GSK@gsk.com> 20 July 2017 at 10:51

Page 118 of 637

Thank you very much for your interest in our HPV vaccine clinical trial program.
We are now reviewing your new request in the context of the responses that we are preparing to the previous set of questions that have
been sent by the Nordic Cochrane Center to us in June.
We will come back to you shortly on both requests.
Best regards,
GSK
GSK <GSK@gsk.com> 28 July 2017 at 16:51

I refer to your request from July 17 to review a list of clinical trials that evaluated Cervarix.
You have just been copied in our communication to Dr. Jefferson to provide responses to the six questions that were sent by Dr. Jefferson in
June. We have included an overview of all GSK sponsored and Medimmune initiated clinical trials that have ever evaluated Cervarix (see
again attached). This list provides you with a tool to validate the completeness of the clinical trial list that you have sent to us on July 17.
Information for all trials is available on our clinical trial register website (see also our answer to question 2 in our response to Dr. Jefferson
for more details). We recommend to use the GSK ID provided in the first column to search for individual trials and to navigate on our
website.
In case you cannot retrieve required information for specific trials from our website, we invite you to send us a list of the trials concerned
and to indicate for each trial what type of information is missing. It may require a request via
the https://www.clinicalstudydatarequest.com/ website to obtain information that is not in the public domain (eg. patient level data).
Best regards,
GSK
Cervarix sponsored studies - Disclosure Status July 2017.xlsx

1502K
Page 119 of 637

The Merck Sharp & Dohme Gardasil and Gardasil 9 study programme
*All Merck Sharp & Dohme employees’ names have been changed to ‘Merck’ and all employee email addresses have been changed to
Merck@merck.com or deleted from the correspondence.
lj@cochrane.dk <lj@cochrane.dk> 8 November 2016 at 11:06

To: Merck@merck.com
Cc: larsjorgensens@gmail.com
Dear Merck,
We are writing to you to ask for your help in creating an accurate up-to-date list of the clinical trials (and their follow-ups) for Mercks'
human papillomavirus (HPV) vaccine: Gardasil™.
As a preliminary to the review and to size the task, we are making a list of all known HPV vaccine trials.
We kindly ask you to assess the attached list of currently identified Gardasil™ trials performed by or in collaboration with Merck and suggest
any corrections and/or edits. (The trials marked in blue are performed in collaboration with Merck, and trials marked in yellow are
conjectural, for which we could find little information.)
We would be grateful to receive our assessments before 8 December 2016.

Best wishes,
PS: If you are not the right person to deal with this request we would appreciate if you would pass the request on to a colleague appropriate
for the task.
Gardasil trial programme.xlsx

44K
Merck <Merck@merck.com> 8 November 2016 at 12:59

Cc: "lj@cochrane.dk" <lj@cochrane.dk>, "larsjorgensens@gmail.com" <larsjorgensens@gmail.com>
Dear Merck,
I received the following query for MRL from colleagues at the Nordic Cochrane Center.
I have attached the referenced document for your review.
Best regards,
Merck
2 attachments
Gardasil trial programme.xlsx

44K
ATT00001.htm
1K

Cc: "jefferson.tom@gmail.com" <jefferson.tom@gmail.com>
Hello, Dr. Jørgensen. You've reached the right place.
We will be happy to support. We are scoping out how much work is involved and will come back to you shortly of an estimate of when the
requested information may be available.
Cheers, Merck
Page 120 of 637

Merck
Global Health & Medical Affairs
Merck Vaccines
Tom Jefferson <jefferson.tom@gmail.com> 9 November 2016 at 08:14

To: Merck <Merck@merck.com>
Cc: "lj@cochrane.dk" <lj@cochrane.dk>
Dear Merck, thank you so much for your quick and helpful answer.
We are grateful for GSK’s [Merck’s] support in our quest for factual accuracy of our work. It will be acknowledged.
Best wishes,
Tom.
Dr Tom Jefferson
Honorary Research Fellow
Centre for Evidence Based Medicine
Oxford OX2 6GG

Dear Merck,
Have you had the time to consider our request for fact checking on the list we sent you on the 8th of November?
With best wishes,
Lars Jørgensen
On 8 November 2016 at 21:08, Merck <Merck@merck.com> wrote:

Lars, can you help us understand better what your request encompasses? That will help us scope out how much work in involved.
Regarding timing, our staff is dispersing as people go out on our US harvest festival, Thanksgiving – not as large as the Chinese New Year
traveling in China, nor as long perhaps as August in France, but distracting nonetheless.
Are you asking us simply to confirm that all our studies are represented by a row in your spreadsheet? [presumably you have already
consulted ClinicalTrials.gov, which we have methodically populated]
If you are asking us to fact-check each cell, this would add to the time required.
I am copying my colleague Merck, who leads our Medical Affairs efforts globally for Gardasil and Gardasil 9.
Merck

Dear Merck,
Thank you for your response and inviting Merck to the conversation.
Your help is highly appreciated.
"Are you asking us simply to confirm that all our studies are represented by a row in your spread-sheet?"
Yes, that is our primary request. We would appreciate if you would prioritise this. The list is mainly constructed
via clinicaltrials.gov and merck.com. (If you find the time later on to confirm/assess other aspects of the spread-sheet, we would appreciate
this as well.)
Hope to hear from you soon.
Best wishes,
Lars

To: Merck@merck.com
Page 121 of 637

Dear Merck,
Unfortunately our group soon has to move on in the process we are in.
We therefore hope that you in the near future will have the time to confirm
that all Merck's Gardasil studies are represented by a row in the spread-sheet
we sent on the 8th of November, 2016.
We hope you can assess this by the 8th of December 2016.
We wish to reiterate that your help will be highly appreciated and acknowledged in any publication.
Best wishes,
Lars Jørgensen

Hello Lars,
Thanks for your note. As Merck mentioned, some, but not all of us, have just returned from the Thanksgiving holiday.
We will do our best to get this information to you by Dec 8, but need to have the input of some members of our team who are still out on
vacation this week.
Can you please tell me how these data will be used? I am not sure I understand what is meant by a “protocol for an exploratory review of
regulatory documents” mentioned in your earlier note.
Thanks for your patience,
Merck
Merck < Merck@merck.com> 1 December 2016 at 18:16

Lars,
One more question for you, please. Are you asking us to verify all the data in the entire document? If so, that will take considerably more
time.
Merck
From: Merck
Sent: Tuesday, November 29, 2016 5:08 PM
To: 'Lars Jørgensen'
Subject: RE: Gardasil trial programme
Lars Jørgensen <larsjorgensens@gmail.com> 1 December 2016 at 18:47

To: Merck < Merck@merck.com>
Dear Merck,
Thank you for your email.
We are primarily requesting a verification of the existence of the studies that we included in the spread-sheet.
We would also like to know if we have missed any studies and what they are called (i.e. V50X-XXX).
We do not request you to verify all the data in the document, but we would appreciate a notice if you find some obvious mistakes present in
the spread-sheet.
We are grateful for your help,
Best wishes,
Lars
Merck <Merck@merck.com> 5 December 2016 at 06:17

Thanks Lars. Can you please tell me how the data will be used (as requested in my email below of Nov 29)?
Merck

Page 122 of 637

Dear Merck,
Sorry, I did not see this email: "Can you please tell me how these data will be used? I am not sure I understand what is meant by a “protocol
for an exploratory review of regulatory documents” mentioned in your earlier note."
We want to:
1) Reconstruct the complete study program of Gardasil.
2) Make a protocol for a review of the study program.
3) Analyse the data that the regulators have of the study program of Gardasil.
Best wishes,
Lars

Dear Merck,
Is there any way we may assist you in the assessment of the study list?
Sincerely,
Lars

Dear Merck,
Have you had the time to assess the study list?
Best wishes,
Lars
Lars Jørgensen <larsjorgensens@gmail.com> 17 July 2017 at 16:52

To: Merck@merck.com
Dear Merck,
We are writing to you to ask for your help in creating an accurate up-to-date list of the clinical trials (and their follow-ups) for Merck Sharp &
Dohme's human papillomavirus (HPV) vaccines: Gardasil and Gardasil 9.
As a preliminary to the review and to size the task, we are making a list of all known interventional HPV vaccine trials that are prospective,
preventive and comparative.
We wrote to Merck Sharp & Dohme in November 2016 with a similar request, but we did not receive any conclusive or explicit feedback on our
index of the Gardasil and Gardasil 9 studies.
We kindly ask you to reconsider to assess the attached list of currently identified Gardasil and Gardasil 9 trials and suggest any corrections and/or
edits.

Best wishes,
HPV vaccine trials - Merck Sharp & Dohme - 17 July.xlsx

82K
Page 123 of 637

The Shanghai Zerun Biotechnology HPV vaccine study programme
lj@cochrane.dk <lj@cochrane.dk> 17 July 2017 at 10:30
To: zerunbd@walvax.com
Dear Shanghai Zerun Biotechnology Co., Ltd.,
We are writing to you to ask for your help in creating an accurate up-to- date list of the clinical trials (and their follow-ups) for Shanghai
Zerun Biotechnology Co., Ltd's human papillomavirus (HPV) vaccines.
We kindly ask you to assess the attached list of currently identified Shanghai Zerun Biotechnology Co., Ltd. HPV vaccine trials and suggest
any corrections and/or edits.
We would appreciate your response before 17 August.

Best wishes,
The Nordic Cochrane Centre

Rigshospitalet dept. 7811
Blegdamsvej 9
2100 Copenhagen
Denmark

To: zerunbd@walvax.com
Dear Shanghai Zerun Biotechnology Co., Ltd.,
We hope that you have received our previous email about your HPV vaccine studies.
If you have any questions, please do not hesitate to contact us.

Best wishes,
Lars Jørgensen, MD
Tom Jefferson, MD
Peter C. Gøtzsche, Professor
Page 124 of 637

The Xiamen Innovax Biotech HPV vaccine study programme
To: 400-001-8508@innovax.cn
Cc: larsjorgensens@gmail.com, jefferson.tom@gmail.com
Dear Xiamen Innovax Biotech Co., Ltd.,
We are writing to you to ask for your help in creating an accurate up-to-date list of the clinical trials (and their follow-ups) for Xiamen
Innovax Biotech Co., Ltd's bivalent human papillomavirus (HPV) vaccine.
We kindly ask you to assess the attached list of currently identified Xiamen Innovax Biotech Co., Ltd HPV vaccine trials and suggest any
corrections and/or edits.
We would appreciate your response before 15 August.

Best wishes,
The Nordic Cochrane Centre

Rigshospitalet dept. 7811
Blegdamsvej 9
2100 Copenhagen
Denmark

To: 400-001-8508@innovax.cn
Cc: larsjorgensens@gmail.com, jefferson.tom@gmail.com
Dear Xiamen Innovax Biotech Co., Ltd.,
We hope that you have received our previous email about your HPV vaccine studies.
If you have any questions, please do not hesitate to contact us.

Best wishes,
Lars Jørgensen, MD
Tom Jefferson, MD
Peter C. Gøtzsche, Professor
18
Page 125 of 637
Appendix 3: Index of the HPV vaccines clinical studies: Indexes of the identified industry study
programmes and non-industry clinical studies and a list of the identified corresponding publications
Index of identified HPV vaccine industry study programmes

The following indexes are the manufacturers study programmes listed in a chronological fashion according
to their study programme IDs:
Index of the GlaxoSmithKline Cervarix study programme:

No. Study Study programme Manufacturer National Clinical Validation from 1st source Validation from 2nd source
classification specific ID specific ID Trial (NCT) ID
1 Definitely exist HPV-001 580299/001 NCT00689741 https://clinicaltrials.gov/show/NCT00689741 https://www.gsk-
clinicalstudyregister.com/study/580299/001
2 Definitely exist HPV-001-NG-PRI 109836 NCT00478621 https://clinicaltrials.gov/show/NCT00478621 https://www.gsk-clinicalstudyregister.com/study/109836
3 Definitely exist HPV-002 580299/002 Not identified https://www.gsk- GlaxoSmithKline’s study list
4 Definitely exist HPV-003 580299/003 Not identified https://www.fda.gov/downloads/BiologicsBlo http://www.tandfonline.com/doi/abs/10.4161/hv.5.5.72
odVaccines/Vaccines/ApprovedProducts/UCM 11
237976.pdf
(HPV-001 follow-up) clinicalstudyregister.com/study/580299/007
9 Definitely exist HPV-009 580299/009 NCT00128661 https://clinicaltrials.gov/show/NCT00128661 http://www.sciencedirect.com/science/article/pii/S00029
37816003094
10 Definitely exist HPV-010 108933 NCT00423046 https://clinicaltrials.gov/show/NCT00423046 https://www.gsk-clinicalstudyregister.com/study/108933

12 Definitely exist HPV-011-EPI 112677 NCT01498627 https://clinicaltrials.gov/show/NCT01498627 https://www.gsk-clinicalstudyregister.com/study/112677
VS FR PMS
107476 NCT00337818 clinicalstudyregister.com/study/580299/012
107477
107479
107481
104904 clinicalstudyregister.com/study/580299/013
15 Definitely exist HPV-013 follow-up 104802 NCT00316706 https://clinicaltrials.gov/show/NCT00316706 https://www.gsk-clinicalstudyregister.com/study/104896
104904
104918
104996
104902
104904
104918

VS US DB

VS UK DB
22 Definitely exist HPV-019-PRI 109823 NCT01031069 https://clinicaltrials.gov/show/NCT01031069 https://www.gsk-clinicalstudyregister.com/study/109823
23 Definitely exist HPV-020-EPI 114176 NCT01290393 https://clinicaltrials.gov/show/NCT01290393 http://www.gsk-clinicalstudyregister.com/files2/114176

VS US

(HPV-001 follow-up) 109624
109625
(HPV-001 follow-up)
(HPV-001 follow-up)
29 Definitely exist HPV-025 111375 NCT00877877 https://clinicaltrials.gov/show/NCT00877877 http://gsk-clinicalstudyregister.com/files2/baee5139-
(HPV-013 follow-up) abc9-4887-be66-eb8e2ad444af
30 Definitely exist HPV-026-PRI 111567 NCT00637195 https://clinicaltrials.gov/show/NCT00637195 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3165228
/
31 Definitely exist HPV-027 115006 NCT01393470 https://clinicaltrials.gov/show/NCT01393470 http://www.clinicaltrialfacts.com/Uterine-Cervical-
(HPV-008 follow-up) Cancer/Evaluation-of-Long-term-HPV-Vaccine-Effi-110268
(HPV-015 follow-up)
35 Definitely exist HPV-031 104479 NCT00344032 https://clinicaltrials.gov/show/NCT00344032 http://www.gsk-

clinicalstudyregister.com/files2/20003.pdf
19
Page 126 of 637
42 Definitely exist HPV-040-EPI VS UK 116239 NCT01953822 https://clinicaltrials.gov/show/NCT01953822 https://www.gsk-clinicalstudyregister.com/study/116239

(HPV-015 follow-up)
43 Definitely exist HPV-040-PRI 106636 NCT00534638 https://clinicaltrials.gov/show/NCT00534638 http://www.ema.europa.eu/docs/en_GB/document_libra
ry/EPAR_-_Assessment_Report_-
_Variation/human/000721/WC500212077.pdf
47 Definitely exist HPV-049 Not identified Not identified http://www.ema.europa.eu/docs/en_GB/doc EMA email from 18 October 2016
ument_library/EPAR_-_Assessment_Report_-
_Variation/human/000721/WC500121597.pdf
48 Definitely exist HPV-051-TETRA 102115 NCT00231413 https://clinicaltrials.gov/show/NCT00231413 http://www.sciencedirect.com/science/article/pii/S02644
10X14004095
49 Definitely exist HPV-051 follow-up 107918 NCT00359619 https://clinicaltrials.gov/show/NCT00359619 https://www.gsk-clinicalstudyregister.com/study/108052
107919
107921
108052
(HPV-008 follow-up)
(HPV-008 follow-up)
52 Definitely exist HPV-056 111712 NCT00811798 https://clinicaltrials.gov/show/NCT00811798 https://clinicalstudydatarequest.com/Posting.aspx?Postin
(HPV-035 follow-up) gID=212
(HPV-008 follow-up)
(HPV-069 co-study)
(HPV-013 follow-up)
(HPV-015 follow-up)
(HPV-032 follow-up)
(HPV-015 follow-up)
(HPV-015 follow-up)
(HPV-023 follow-up)
(HPV-058 co-study)
66 Definitely exist HPV-081 200255 Not identified http://apps.who.int/trialsearch/Trial2.aspx?Tr https://www.clinicaltrialsregister.eu/ctr-

(HPV-048 follow-up) ialID=EUCTR2014-000060-17-DE search/trial/2014-000060-17/DE
67 Definitely exist HPV-111103-EPI 111103 Not identified http://apps.who.int/trialsearch/Trial2.aspx?Tr https://www.clinicaltrialsregister.eu/ctr-
ialID=EUCTR2007-006651-39-GB search/trial/2007-006651-39/GB
68 Definitely exist MENACWY-TT-054 113823 NCT01755689 https://clinicaltrials.gov/show/NCT01755689 https://www.gsk-clinicalstudyregister.com/study/113823
69 Definitely exist Not identified 113763 NCT01551537 https://clinicaltrials.gov/show/NCT01551537 https://www.gsk-clinicalstudyregister.com/study/113763
20
Page 127 of 637
Index of the Merck Sharp & Dohme Gardasil and Gardasil 9 study programmes:
1 Probably exist V501-001 Not identified Not identified http://www.fda.gov/ohrms/dockets/ac/06/sli Not identified
des/2006-4222S-2_files/frame.htm
4 Definitely exist V501-005 2006_515 NCT00365378 https://clinicaltrials.gov/show/NCT00365378 http://www.merck.com/clinical-
trials/study.html?id=V501-005&kw=gardasil
5 Probably exist V501-005 follow-up Not identified Not identified https://www.ncbi.nlm.nih.gov/pmc/articles/P Not identified
MC2749988/#R10
7 Definitely exist V501-011 2007_576 NCT00517309 https://clinicaltrials.gov/show/NCT00517309 https://www.ncbi.nlm.nih.gov/pubmed/18164106?dopt=
Abstract
8 Definitely exist V501-012 2004_080 NCT00092482 https://clinicaltrials.gov/show/NCT00092482 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1951095
/
Abstract
16 Definitely exist V501-024 2005_093 NCT00337428 https://clinicaltrials.gov/show/NCT00337428 http://www.clinicaltrialfacts.com/Neoplasms-Glandular-
and-Epithelial/Concomitant-Use-of-Gardasil-V501-
Human-29955
17 Definitely exist V501-025 2005_092 NCT00325130 https://clinicaltrials.gov/show/NCT00325130 http://www.clinicaltrialfacts.com/Neoplasms-Glandular-
and-Epithelial/Concomitant-Use-of-Gardasil-V501-Huma-
29044
21 Definitely exist V501-030 follow-up Not identified NCT01427777 https://clinicaltrials.gov/show/NCT01427777 http://adisinsight.springer.com/trials/700025686

25 Definitely exist V501-046 Not identified NCT01245764 https://clinicaltrials.gov/show/NCT01245764 http://www.merck.com/clinical-
30 Definitely exist V502-003 2006_503 NCT00365443 https://clinicaltrials.gov/show/NCT00365443 https://www.clinicaltrialsregister.eu/ctr-
search/trial/2006-004933-14/DK
32 Definitely exist V503-001 follow-up Not identified Not identified http://www.ins.gob.pe/ensayosclinicos/rpec/r http://apps.who.int/trialsearch/Trial2.aspx?TrialID=PER-
ecuperarECPBNuevoEN.asp?numec=090-13 090-13
33 Definitely exist V503-002 Not identified NCT00943722 https://clinicaltrials.gov/show/NCT00943722 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4514432
(immunobridging /
study)
34 Definitely exist V503-002 (lot 2009_611 NCT00943722 https://clinicaltrials.gov/show/NCT00943722 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4514432
consistency study) /
35 Definitely exist V503-002 follow-up Not identified Not identified http://www.ins.gob.pe/ensayosclinicos/rpec/r http://apps.who.int/trialsearch/Trial2.aspx?TrialID=PER-
ecuperarECPBNuevoEN.asp?numec=120-12 120-12
36 Definitely exist V503-003 Not identified NCT01651949 https://clinicaltrials.gov/show/NCT01651949 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=NCT0
1651949
trials/study.html?id=V503-004&kw=Gardasil
38 Definitely exist V503-005 Not identified NCT00988884 https://clinicaltrials.gov/show/NCT00988884 http://pediatrics.aappublications.org/content/136/3/e56
3.long
Abstract
(V503-002 follow- trials/study.html?id=V503-
up) 008&kw=vaccines&tab=eligibility
42 Definitely exist V503-009 GDS01C NCT01304498 https://clinicaltrials.gov/show/NCT01304498 http://www.merck.com/product/usa/pi_circulars/g/gard
asil_9/gardasil_9_pi.pdf
43 Definitely exist V503-010 Not identified NCT01984697 https://clinicaltrials.gov/show/NCT01984697 https://www.fda.gov/downloads/BiologicsBloodVaccines/
Vaccines/ApprovedProducts/UCM524629.pdf
44 Probably exist V503-018 Not identified Not identified http://www.fda.gov/downloads/BiologicsBloo Not identified
dVaccines/Vaccines/ApprovedProducts/UCM1
90977.pdf
45 Probably exist V503-019 Not identified Not identified http://www.fda.gov/downloads/BiologicsBloo Not identified
dVaccines/Vaccines/ApprovedProducts/UCM2
51763.pdf
Page 128 of 637

46 Definitely exist V503-020 GDS07C NCT02114385 https://clinicaltrials.gov/show/NCT02114385 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=EUCT
R2013-003399-10-DE
47 Probably exist V503-021 Not identified NCT02653118 https://clinicaltrials.gov/show/NCT02653118 Not identified
(V503-001 follow-
up)
50 Definitely exist Not identified Not identified NCT00572832 https://clinicaltrials.gov/show/NCT00572832 https://www.ncbi.nlm.nih.gov/pubmed/20629576?dopt=
Abstract
51 Definitely exist Not identified Not identified NCT00573651 https://clinicaltrials.gov/show/NCT00573651 http://www.clinicaltrialfacts.com/Arthritis/Safety-and-
Efficacy-of-Gardasil-in-Femal-47769
52 Definitely exist Not identified Not identified NCT00727636 https://clinicaltrials.gov/show/NCT00727636 http://www.clinicaltrialfacts.com/Inflammatory-Bowel-
Disease/Immunogenicity-to-Human-Papillomavirus-V-
59478
53 Definitely exist Not identified Not identified NCT01489527 https://clinicaltrials.gov/show/NCT01489527 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4378717
/
54 Definitely exist Not identified Not identified NCT01924754 https://clinicaltrials.gov/show/NCT01924754 http://www.clinicaltrialfacts.com/HPV-
Seroconversion/Gardasil-Immunogenicity-With-Needle-
Free-141969
55 Definitely exist Not identified Not identified NCT02199691 https://clinicaltrials.gov/show/NCT02199691 https://www.clinicaltrialsregister.eu/ctr-
search/trial/2016-001963-35/3rd
56 Definitely exist Not identified Not identified NCT02968420 https://clinicaltrials.gov/show/NCT02968420 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=NCT0
2968420
57 Definitely exist Not identified Not identified Not identified http://apps.who.int/trialsearch/Trial2.aspx?Tr http://cris.nih.go.kr/cris/en/search/search_result_st01.js
ialID=KCT0000604 p?seq=2624
58 Probably exist Not identified Not identified NCT00767897 https://clinicaltrials.gov/show/NCT00767897 Not identified
Page 129 of 637

Index of ‘other’ industry HPV vaccine study programmes:
Shanghai Zerun Biotechnology Co., Ltd.
1 Definitely exist 311-HPV-1001 Not identified NCT03085381 https://clinicaltrials.gov/show/NCT03085381 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=NCT0
3085381
2740790
2733068
2740777
Xiamen Innovax Biotech Co., Ltd.
5 Definitely exist HPV-PRO-002 Not identified NCT01356823 https://clinicaltrials.gov/show/NCT01356823 https://www.ncbi.nlm.nih.gov/pubmed/26100924?dopt=
Abstract
6 Definitely exist HPV-PRO-003 Not identified NCT01735006 https://clinicaltrials.gov/show/NCT01735006 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=NCT0
1735006
7 Probably exist HPV-PRO-004 Not identified NCT02405520 https://clinicaltrials.gov/show/NCT02405520 Not identified

2710851
9 Probably exist HPV-PRO-006 Not identified NCT02562508 https://clinicaltrials.gov/show/NCT02562508 Not identified

follow-up 3206255
Page 130 of 637

Index of identified HPV vaccine non-industry funded studies
The following is an index of the identified non-industry funded studies:
No. Study classification Study ID National Clinical Validation from 1st source Validation from 2nd source
Trial (NCT) ID
Non-industry funded Cervarix studies

1 Definitely exist NL26.113.000.08 NCT00815282 https://clinicaltrials.gov/show/NCT00815282 http://ard.bmj.com/content/73/8/1500.long
2 Probably exist LIS144 NCT01082861 https://clinicaltrials.gov/show/NCT01082861 Not identified
3 Definitely exist Not identified NCT02276521 https://clinicaltrials.gov/show/NCT02276521 https://academic.oup.com/cid/article/doi/10.1093/cid/ciw

865/2747464/Sustained-Antibody-Responses-6-Years-
Following-1-2
4 Definitely exist 342/2009 NCT02296255 https://clinicaltrials.gov/show/NCT02296255 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=NCT02
296255
5 Definitely exist FASTER-Tialpan Study NCT03105856 https://clinicaltrials.gov/show/NCT03105856 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=NCT03

105856
6 Definitely exist ESCUDDO NCT03180034 https://clinicaltrials.gov/show/NCT03180034 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=NCT03
180034
Non-industry funded Gardasil and Gardasil 9 studies
7 Definitely exist P1047 NCT00339040 https://clinicaltrials.gov/show/NCT00339040 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3033215/
8 Definitely exist H07-00928 NCT00501137 https://clinicaltrials.gov/show/NCT00501137 http://jamanetwork.com/journals/jama/fullarticle/168293

9
9 Definitely exist DDEAMC-07-43X NCT00501189 https://clinicaltrials.gov/show/NCT00501189 http://www.clinicaltrialfacts.com/Papillomavirus-

Infections/Gardasil-Vaccination-as-Therapy-in-Low-G-
42300
10 Definitely exist HPV01 NCT00524745 https://clinicaltrials.gov/show/NCT00524745 http://jid.oxfordjournals.org/content/208/8/1325.long
11 Definitely exist A5240 NCT00604175 https://clinicaltrials.gov/show/NCT00604175 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4305143/
12 Probably exist ATN064 NCT00710593 https://clinicaltrials.gov/show/NCT00710593 Not identified
13 Probably exist 090024 NCT00798265 https://clinicaltrials.gov/show/NCT00798265 Not identified
14 Definitely exist Pro00014388 NCT00862810 https://clinicaltrials.gov/show/NCT00862810 http://jamanetwork.com/journals/jama/fullarticle/168293

9
15 Definitely exist Not identified NCT00923702 https://clinicaltrials.gov/show/NCT00923702 https://www.isrctn.com/ISRCTN98283094
16 Definitely exist HRPO-07-0648 NCT00941889 https://clinicaltrials.gov/show/NCT00941889 http://www.clinicaltrialfacts.com/HIV-Positive/The-Effect-

of-HPV-Vaccination-on-Recurre-75760
17 Definitely exist SG09-EN01 NCT00949572 https://clinicaltrials.gov/show/NCT00949572 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3306286/
18 Definitely exist HPV CSP01 NCT00956553 https://clinicaltrials.gov/show/NCT00956553 http://www.clinicaltrialfacts.com/HPV-

Infections/Reactogenicity-Study-of-Cervarix-and-Gar-
76885
19 Probably exist 08-0012-N01AI80006C NCT01030562 https://clinicaltrials.gov/show/NCT01030562 Not identified
20 Definitely exist MITU-001 NCT01173900 https://clinicaltrials.gov/show/NCT01173900 https://www.ncbi.nlm.nih.gov/pubmed/22711908
21 Definitely exist LTN0001 NCT01386164 https://clinicaltrials.gov/show/NCT01386164 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4896591/
22 Definitely exist 9427-L1802/1-21C NCT01456715 https://clinicaltrials.gov/show/NCT01456715 http://www.clinicaltrialfacts.com/Human-

Papillomavirus/Immunogenicity-of-Gardasil-and-Twinrix-a-
115114
23 Definitely exist A5298 NCT01461096 https://clinicaltrials.gov/show/NCT01461096 http://www.clinicaltrialfacts.com/HIV-
Infections/Evaluating-the-Effectiveness-of-the-Quad-
115451
24 Probably exist IRB00046117 NCT01505049 https://clinicaltrials.gov/show/NCT01505049 Not identified
25 Definitely exist HLS04/2011 NCT01512784 https://clinicaltrials.gov/show/NCT01512784 http://www.clinicaltrialfacts.com/HPV/Long-Term-

Immunogenicity-of-Quadrivalent-119444
26 Definitely exist 883 NCT01717118 https://clinicaltrials.gov/show/NCT01717118 http://www.tandfonline.com/doi/full/10.1080/21645515.

2015.1058458?scroll=top&needAccess=true
27 Probably exist CIHR-MOP-125949 NCT01824537 https://clinicaltrials.gov/show/NCT01824537 Not identified
28 Probably exist HPV girls 5 year follow up NCT01896986 https://clinicaltrials.gov/show/NCT01896986 Not identified
29 Probably exist 2013/422 NCT01914367 https://clinicaltrials.gov/show/NCT01914367 Not identified
30 Probably exist ICI-VPH-1 NCT02009800 https://clinicaltrials.gov/show/NCT02009800 Not identified
31 Definitely exist NL45200.018.13 NCT02087384 https://clinicaltrials.gov/show/NCT02087384 http://www.clinicaltrialfacts.com/AIN/HPV-Human-

Papilloma-Virus-Vaccination--147051
32 Probably exist Pro00014388_1 NCT02280642 https://clinicaltrials.gov/show/NCT02280642 Not identified
33 Probably exist GINI Study NCT02363660 https://clinicaltrials.gov/show/NCT02363660 Not identified
34 Probably exist NTWC/CREC/15035 NCT02477254 https://clinicaltrials.gov/show/NCT02477254 Not identified
Page 131 of 637

35 Probably exist HPV 2355 NCT02567955 https://clinicaltrials.gov/show/NCT02567955 Not identified
36 Definitely exist MISP-53183 NCT02750202 https://clinicaltrials.gov/show/NCT02750202 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=NCT02

750202
37 Definitely exist MITU-002 NCT02834637 https://clinicaltrials.gov/show/NCT02834637 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=NCT02

834637
39 Probably exist cycdc2016-4 NCT02888418 https://clinicaltrials.gov/show/NCT02888418 Not identified
40 Definitely exist Not identified Not identified https://www.clinicaltrialsregister.eu/ctr- http://apps.who.int/trialsearch/Trial2.aspx?TrialID=EUCTR

search/trial/2016-002083-13/GB 2016-002083-13-GB
41 Definitely exist ACTRN12608000339358 Not identified http://apps.who.int/trialsearch/Trial2.aspx?Tri https://www.anzctr.org.au/Trial/Registration/TrialReview.

alID=ACTRN12608000339358 aspx?ACTRN=12608000339358
42 Probably exist ACTRN12613001207707 Not identified https://www.anzctr.org.au/Trial/Registration/T Not identified

rialReview.aspx?id=364244&isReview=true
43 Probably exist CTRI/2013/11/004140 Not identified http://ctri.nic.in/Clinicaltrials/pmaindet2.php?t Not identified
rialid=5969&EncHid=&userName=papillomavir
us
44 Definitely exist IPAR0001 Not identified http://apps.who.int/trialsearch/Trial2.aspx?Tri https://www.clinicaltrialsregister.eu/ctr-search/trial/2012-
alID=EUCTR2012-000445-12-DK 000445-12/DK
45 Probably exist Not identified Not identified https://www.isrctn.com/ISRCTN14732216 Not identified
46 Definitely exist Not identified Not identified http://www.isrctn.com/ISRCTN14732216 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=ISRCTN

14732216
47 Definitely exist Not identified Not identified https://www.isrctn.com/ISRCTN32729817 http://apps.who.int/trialsearch/Trial2.aspx?TrialID=ISRCTN

32729817
48 Definitely exist LTN0001 Not identified http://apps.who.int/trialsearch/Trial2.aspx?Tri https://www.clinicaltrialsregister.eu/ctr-search/trial/2011-
alID=EUCTR2011-001871-37-DK 001871-37/DK
49 Definitely exist 2010-1090 Not identified http://apps.who.int/trialsearch/Trial2.aspx?Tri https://www.clinicaltrialsregister.eu/ctr-search/trial/2012-
alID=EUCTR2012-004007-13-DE 004007-13/DE
50 Definitely exist P150957 Not identified https://www.clinicaltrialsregister.eu/ctr- http://apps.who.int/trialsearch/Trial2.aspx?TrialID=EUCTR

search/trial/2016-002455-20/FR 2016-002455-20-FR
51 Definitely exist VACCAIN-P Not identified http://apps.who.int/trialsearch/Trial2.aspx?Tri https://www.clinicaltrialsregister.eu/ctr-search/trial/2013-

alID=EUCTR2013-002009-70-NL 002009-70/NL
52 Probably exist Not identified Not identified http://www.tandfonline.com/doi/full/10.1080/ Not identified

21645515.2016.1277846?scroll=top&needAcce
ss=true
53 Probably exist Not identified Not identified http://www.sciencedirect.com/science/article/ Not identified
pii/S0264410X13008128
54 Probably exist Not identified Not identified http://online.liebertpub.com/doi/abs/10.1089/ Not identified
jwh.2009.1753
Other non-industry funded HPV vaccine studies

56 Probably exist HP-00041372 NCT00257738 https://clinicaltrials.gov/show/NCT00257738 Not identified
57 Probably exist HP-41372 NCT00704041 https://clinicaltrials.gov/show/NCT00704041 Not identified
58 Probably exist CDR0000383786 NCT00091130 https://clinicaltrials.gov/show/NCT00091130 Not identified
59 Probably exist PE1201 NCT02564237 https://clinicaltrials.gov/show/NCT02564237 Not identified
60 Probably exist DRKS00005278 Not identified https://www.drks.de/drks_web/navigate.do?n Not identified

avigationId=trial.HTML&TRIAL_ID=DRKS000052
78
61 Probably exist Not identified Not identified http://jnci.oxfordjournals.org/content/93/4/28 Not identified
4.long
Page 132 of 637

Index of identified industry and non-industry funded publications in the HPV vaccine study programmes
The following is a list of identified publications of industry and non-industry funded studies:
Identified biomedical journal publications of industry funded studies

No. Study programme Manufacturer National Clinical Publication source
specific ID specific ID Trial (NCT) ID
1 HPV-001 580299/001 NCT00689741 http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(04)17398-4/abstract
2 HPV-001-NG-PRI 109836 NCT00478621 http://www.sciencedirect.com/science/article/pii/S0264410X14004095
3 HPV-004 580299/004 NCT00693615 http://www.sciencedirect.com/science/article/pii/S0264410X06007092
4 HPV-005 580299/005 NCT00693966 http://www.sciencedirect.com/science/article/pii/S0264410X06007092

(HPV-001 follow-up)
7 HPV-009 580299/009 NCT00128661 http://www.sciencedirect.com/science/article/pii/S0002937816003094
8 HPV-010 108933 NCT00423046 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3338932/
9 HPV-011 580299/011 NCT00309166 http://www.jahonline.org/article/S1054-139X(08)00434-5/pdf
10 HPV-012 107479 NCT00337818 http://onlinelibrary.wiley.com/doi/10.1002/ijc.25887/full
11 HPV-013 104904 NCT00196924 http://www.jahonline.org/article/S1054-139X(10)00102-3/abstract
12 HPV-013 follow-up 104896 NCT00316706 http://www.jahonline.org/article/S1054-139X(11)00615-X/abstract
13 HPV-014 103514 NCT00196937 http://www.sciencedirect.com/science/article/pii/S0264410X08015004
14 HPV-015 104820 NCT00294047 http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(14)60920-X/abstract
15 HPV-018 107682 NCT00369824 https://www.ncbi.nlm.nih.gov/pubmed/21817954

(HPV-001 follow-up)
19 HPV-025 (HPV-013 111375 NCT00877877 http://adisinsight.springer.com/trials/700229791
follow-up)
21 HPV-029 110886 NCT00578227 http://www.jahonline.org/article/S1054-139X(11)00353-3/pdf
23 HPV-031 104479 NCT00344032 http://onlinelibrary.wiley.com/doi/10.1111/j.1447-0756.2009.01167.x/abstract
24 HPV-032 104798 NCT00316693 https://www.ncbi.nlm.nih.gov/pubmed/20606533
25 HPV-033 104951 NCT00290277 http://jkms.org/DOIx.php?id=10.3346/jkms.2010.25.8.1197
26 HPV-035 106001 NCT00306241 http://www.hkmj.org/abstracts/v16n3/171.htm
27 HPV-036 105926 NCT00345878 http://www.e-mjm.org/2014/v69n1/cervical-cancer-vaccine.pdf
30 HPV-040 106636 NCT00534638 http://www.sciencedirect.com/science/article/pii/S0264410X1401648X
31 HPV-042 108464 NCT00426361 http://www.jahonline.org/article/S1054-139X(09)00629-6/abstract
32 HPV-044 109179 NCT00552279 https://www.ncbi.nlm.nih.gov/pubmed/21273939?dopt=Abstract
35 HPV-058 (HPV-069 112022 NCT00996125 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4186032/

co-study)

follow-up)
Page 133 of 637

follow-up)
38 HPV-068 (HPV-023 114379 NCT01418937 http://adisinsight.springer.com/trials/700228771
follow-up)

(HPV-058 co-study)
40 HPV-070 114700 NCT01381575 https://academic.oup.com/jid/article/215/11/1711/3862470/Sustained-Immunogenicity-of-2-dose-Human
42 HPV-PRO-002 Not identified NCT01356823 https://www.ncbi.nlm.nih.gov/pubmed/26100924?dopt=Abstract
43 V501-005 2006_515 NCT00365378 http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(07)60852-6/abstract
44 V501-005 follow-up Not identified Not identified https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2749988/#R10
45 V501-007 2006_516 NCT00365716 https://www.ncbi.nlm.nih.gov/pubmed/15863374?dopt=Abstract
47 V501-012 2004_080 NCT00092482 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1951095/
48 V501-013 2004_081 NCT00092521 http://www.nejm.org/doi/full/10.1056/NEJMoa061760#t=article
50 V501-018 2004_084 NCT00092547 http://pediatrics.aappublications.org/content/134/3/e657.long
52 V501-020 2004_103 NCT00090285 http://www.nejm.org/doi/full/10.1056/NEJMoa0909537
56 V501-027 2006_032 NCT00378560 http://onlinelibrary.wiley.com/doi/10.1111/cas.12106/abstract
57 V501-030 2007_021 NCT00496626 https://www.ncbi.nlm.nih.gov/pubmed/22433961
58 V501-031 2010_019 NCT01078220 http://jamanetwork.com/journals/jamapediatrics/fullarticle/1363509
59 V501-033 2010_018 NCT01077856 http://jnci.oxfordjournals.org/content/106/3/djt460.long
60 V501-046 Not identified NCT01245764 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4514396/
62 V502-002 2009_552 NCT00851643 http://adisinsight.springer.com/trials/700042113
63 V503-001 2007_538 NCT00543543 http://www.nejm.org/doi/full/10.1056/NEJMoa1405044
64 V503-002 Not identified NCT00943722 http://pediatrics.aappublications.org/content/136/1/e28

(immunobridging
study)
(lot consistency
study)
66 V503-003 2012-002758- NCT01651949 https://www.ncbi.nlm.nih.gov/pubmed/26144901?dopt=Abstract
22
67 V503-005 Not identified NCT00988884 http://pediatrics.aappublications.org/content/136/3/e563.long
68 V503-006 2010_504 NCT01047345 http://www.sciencedirect.com/science/article/pii/S0264410X15011871
70 V503-008 Not identified NCT01254643 http://198.61.244.207/Upload/85_Applied%20Redaction%20V503%20P008%20CSR%20Synopsis.pdf

(V503-002 follow-up)
71 V503-009 GDS01C NCT01304498 https://www.ncbi.nlm.nih.gov/pubmed/26090572
72 V503-010 2013-001314- NCT01984697 http://jamanetwork.com/journals/jama/article-abstract/2620075

15
73 V503-020 GDS07C NCT02114385 http://www.sciencedirect.com/science/article/pii/S0264410X16304807
75 Not identified Not identified NCT00572832 https://www.ncbi.nlm.nih.gov/pubmed/20629576?dopt=Abstract
Page 134 of 637

76 Not identified Not identified NCT01489527 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4378717/
Identified biomedical journal publications of non-industry funded studies
No. Study programme Manufacturer National Clinical Publication source

specific ID specific ID Trial (NCT) ID
77 Not applicable Not NCT00524745 http://jid.oxfordjournals.org/content/208/8/1325.long
applicable
78 Not applicable Not NCT00604175 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4305143/

applicable
79 Not applicable Not NCT00815282 http://ard.bmj.com/content/73/8/1500.long

applicable
80 Not applicable Not NCT00862810 http://jamanetwork.com/journals/jama/fullarticle/1682939
applicable
81 Not applicable Not NCT00923702 http://www.sciencedirect.com/science/article/pii/S1470204515004143

applicable

applicable

applicable
84 Not applicable Not NCT01173900 https://www.ncbi.nlm.nih.gov/pubmed/22711908

applicable
applicable
86 Not applicable Not NCT01717118 http://www.tandfonline.com/doi/full/10.1080/21645515.2015.1058458?scroll=top&needAccess=true

applicable
87 Not applicable Not NCT02276521 https://academic.oup.com/cid/article/doi/10.1093/cid/ciw865/2747464/Sustained-Antibody-Responses-6-Years-Following-1-2

applicable
88 Not applicable Not NCT02296255 http://www.tandfonline.com/doi/full/10.4161/hv.24337

applicable
89 Not applicable Not Not identified http://jnci.oxfordjournals.org/content/93/4/284.long
applicable
90 Not applicable Not Not identified http://online.liebertpub.com/doi/abs/10.1089/jwh.2009.1753
applicable
91 Not applicable Not Not identified http://www.tandfonline.com/doi/full/10.1080/21645515.2016.1277846?scroll=top&needAccess=true

applicable
92 Not applicable Not Not identified http://www.sciencedirect.com/science/article/pii/S0264410X13008128

applicable
Page 135 of 637

Section/topic # Checklist item Reported on page #
TITLE
Title 1 Identify the report as a systematic review, meta-analysis, Not applicable
or both.
ABSTRACT
Structured summary 2 Provide a structured summary including, as applicable: 2
background; objectives; data sources; study eligibility
criteria, participants, and interventions; study appraisal
and synthesis methods; results; limitations; conclusions
and implications of key findings; systematic review
registration number.
INTRODUCTION
Rationale 3 Describe the rationale for the review in the context of 3
what is already known.
Objectives 4 Provide an explicit statement of questions being Not applicable
addressed with reference to participants, interventions,
comparisons, outcomes, and study design (PICOS).
METHODS
Protocol and 5 Indicate if a review protocol exists, if and where it can be No protocol exist for this study;
registration accessed (e.g., Web address), and, if available, provide however, a protocol exists for our
registration information including registration number. planned systematic review.
Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of 3 and 4
follow-up) and report characteristics (e.g., years
considered, language, publication status) used as criteria
for eligibility, giving rationale.
Information sources 7 Describe all information sources (e.g., databases with Appendices 1 and 2
dates of coverage, contact with study authors to identify
additional studies) in the search and date last searched.
Search 8 Present full electronic search strategy for at least one Appendix 1
database, including any limits used, such that it could be
repeated.
Study selection 9 State the process for selecting studies (i.e., screening, 3 and 4
eligibility, included in systematic review, and, if
applicable, included in the meta-analysis).
Data collection 10 Describe method of data extraction from reports (e.g., 3 and 4
process piloted forms, independently, in duplicate) and any
processes for obtaining and confirming data from
investigators.
Data items 11 List and define all variables for which data were sought 3 and 4
(e.g., PICOS, funding sources) and any assumptions and
simplifications made.
Page 136 of 637
Risk of bias in 12 Describe methods used for assessing risk of bias of Not applicable
individual studies individual studies (including specification of whether this
was done at the study or outcome level), and how this
information is to be used in any data synthesis.
Summary measures 13 State the principal summary measures (e.g., risk ratio, 4
difference in means).
Synthesis of results 14 Describe the methods of handling data and combining 4
results of studies, if done, including measures of
consistency (e.g., I2) for each meta-analysis.
Page 1 of 2
Reported on
page #
Risk of bias across 15 Specify any assessment of risk of bias that may affect the cumulative evidence Not applicable
studies (e.g., publication bias, selective reporting within studies).
Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup Not applicable
analyses, meta-regression), if done, indicating which were pre-specified.
RESULTS
Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the 3 and 4
review, with reasons for exclusions at each stage, ideally with a flow diagram.
Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., Not applicable
Risk of bias within 19 Present data on risk of bias of each study and, if available, any outcome level Not applicable
studies assessment (see item 12).
Results of individual 20 For all outcomes considered (benefits or harms), present, for each study: (a) Not applicable
studies simple summary data for each intervention group (b) effect estimates and
confidence intervals, ideally with a forest plot.
Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals and Not applicable
measures of consistency.
Risk of bias across 22 Present results of any assessment of risk of bias across studies (see Item 15). Not applicable
studies
Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup Not applicable
DISCUSSION
Summary of 24 Summarize the main findings including the strength of evidence for each main 5-7
evidence outcome; consider their relevance to key groups (e.g., healthcare providers,
users, and policy makers).
Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at 5-7
review-level (e.g., incomplete retrieval of identified research, reporting bias).
Conclusions 26 Provide a general interpretation of the results in the context of other 5-7
FUNDING
Funding 27 Describe sources of funding for the systematic review and other support (e.g., 8
Page 137 of 637
supply of data); role of funders for the systematic review.
From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews
and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed1000097
For more information, visit: www.prisma-statement.org.
Page 2 of 2
Page 138 of 637

Paper 3
Challenges of independent HPV vaccine assessment
Page 139 of 637

U N lVbRyilY U ^ COF bMH AU bn
^ r if6r m a t (bn i^ n <Rh )0.^ yiden t ^ :j^ ;^
N a m e of P h D st u den t La r sJ 0r gen sen
E -m a il lj@coch r a n e.dk ; jcl649@su n d.k u .dk ; la r sjor gen sen s@gm a il.com
Da t e of bir t h 4 F ebr u a r y 1987
Wor k pla ce Nor dic Coch r a n e Cen t r e
P r in cipa l su per visor P et er C. G0t zsch e
iTttleO^ PhD^ hesl^
Ben efit s a n d h a r m s of t h e h u m a n pa pillom a vir u s (H P V) va ccin es
;Th is;decla r a t ion lc()n c^ r n sAh ^ fb!lpw!n g a r t icle^ !
J 0r gen sen L/ Dosh i P , G0t zsch e P C a n d J effer son T. Ch a llen ges of in depen den t a ssessm en t of pot en t ia l h a r m s of H P V
va ccin es. BMJ . 2018Sep;362:k 369. h t t ps://doi.or g/l0.1136/bm j.k 3694.
Th eP h Ds t u d en ^ s con r t r ibu t ibr it ot h &a r t ict e:.^ ^ . ;^ :

::(p ^ e a ^ M s ^ t h 6 /s ca le :(A^ j^ C }^ e h w a 5 b e n ch m Q r ?^ -: -• '• ;': ,;'.^ -:'"';'-.'.' '.< i(^ ?
1. F or m u la t ion /iden t ifica t ion of t h e scien t ific pr oblem t h a t fr om t h eor et ica l qu est ion s n eed t o be B
cla r ified. Th is in clu des a con den sa t ion of t h e pr oblem t o specific scien t ific qu est ion s t h a t is J u dged
t o be a n swer a ble by exper im en t s
2. P la n n in g of t h e exper im en t s a n d m et h odology design , in clu din g select ion of m et h ods a n d m et h od c
developm en t
3. In volvem en t in t h e exper im en t a l wor k c
4. P r esen t a t ion , in t er pr et a t ion a n d discu ssion in a jou r n a l a r t icle for m a t of obt a in ed da t a B
*Ben ch m a r k sca le of t h e P h D st u den t 's con t r ibu t ion t o t h e a r t icle

A. r efer s t o: H a s con t r ibu t ed t o t h e co-oper a t ion 0-33%
B. r efer s t o: H a s con t r ibu t ed con sider a bly t o t h e co-oper a t ion 34-66 %
C. r efer s t o: H a s pr edom in a n t ly execu t ed t h e wor k in depen den t ly 67-100 %
l.:S ign ^ f^ r e \ ot t b e '(:6 ^ u t h or s ^ ;::\ :^ ^ ;• .;";^ ^ ^ ^ :^ '"\ ^ '-':^ -^ •

Da t e: N a m e: Tit le: Sign a t u r e:,
^^-
15 P et er Dosh i P h D, a ssist a n t
Oct ober pr ofessor ,
2018 Un iver sit y of
Ma r yla n d Sch ool
of P h a r m a cy
15 P et er C. G0t zsch e Dr MedSd/
Oct ober pr ofessor
Page 140 of 637
2018
15 Tom J effer son MD
Oct ober
2018
-:| Sigriatur^ Eof.th^ RKD^ pu(ienf?and| ^ he-pri^ ^ ^

Da t e: 15 Oct ober 2018 Da t e: 15 Oct ober 2018
P h D st u den t : P r in cipa l su per visor :
Page 141 of 637

BMJ 2018;362:k3694 doi: 10.1136/bmj.k3694 (Published 24 September 2018) Page 1 of 5
Analysis
BMJ: first published as 10.1136/bmj.k3694 on 24 September 2018. Downloaded from http://www.bmj.com/ on 24 September 2018 by guest. Protected by copyright.
ANALYSIS
Challenges of independent assessment of potential

harms of HPV vaccines
After three years of trying to access trial data for HPV vaccines, Lars Jørgensen and colleagues
find current transparency policies unfit for their purpose
1 2 1
Lars Jørgensen researcher, , Peter Doshi assistant professor , Peter Gøtzsche professor , Tom
1
Jefferson researcher
1
Nordic Cochrane Centre, Rigshospitalet, Copenhagen, Denmark; 2University of Maryland School of Pharmacy, Baltimore, Maryland, USA;
Correspondence to: T Jefferson tj@cochrane.dk
completed HPV vaccine studies were not published and only

Key messages about half of completed studies posted results on
Public confidence in interventions such as vaccines relies on ClinicalTrials.gov.16
comprehensive, independent, and accurate assessments
Although regulators can request the underlying data (such as
Clinical study reports contain more information than journal publications
but are harder to access
clinical study reports with case report forms and serious adverse
Only half of potentially eligible reports for a systematic review of HPV
event narratives) for all a manufacturer’s trials, regulators usually
vaccines had been obtained after three years, and these were incomplete conduct in-depth analyses of only some of the trials and do not
and contained redactions use systematic review methods. For EMA’s investigation into
Regulators did not have the full data and the manufacturers place a potential link between HPV vaccines and POTS and CRPS,
restrictions on the dissemination of data
the manufacturers included only half of the studies likely to
The process for releasing clinical study reports should be improved to
make it faster and more complete
contribute data.16 EMA concluded that the “benefits of HPV
vaccines continue to outweigh their risks,”14 even though many
cases of POTS and CRPS were unrecognised or under-reported
Since the registration of the first human papillomavirus (HPV) by the vaccine manufacturers.17 18
vaccine (Gardasil) in 2006, HPV vaccination has been rolled
out across the globe.1 There are currently three registered HPV Best way to assess potential harms
vaccines: GlaxoSmithKline’s Cervarix and Merck’s Gardasil Using journal publications as the source of data for systematic
and Gardasil 9. The vaccines are mostly given to healthy reviews often has serious limitations, especially for harms. For
adolescents for the prevention of HPV related diseases, such as example, a systematic review that compared journal publications
cervical cancer, making them an important public health with unpublished reports—across 10 drug interventions—found
intervention.1 that between 43% and 100% (median 64%) of the harms were
In the late 2000s, reports of potential harms associated with missing from the journal publications compared with the
HPV vaccines, including postural orthostatic tachycardia unpublished reports.15 One of the included studies of psychiatric
syndrome (POTS) and complex regional pain syndrome (CRPS), drugs, which compared journal publications with clinical trial
began appearing in the media (see supplementary data on register entries, found that about half of the deaths and suicides
bmj.com),2 3 followed by reports in scientific journals.4-6 Both were missing in the journal publications.19
syndromes are conglomerates of signs and symptoms and their Drug companies describe their trials extensively in clinical study
diagnoses are complex. reports, which are included in licensing applications to
In response to concerns about potential harms, the European regulators. Clinical study reports are structured according to
Medicines Agency carried out an investigation in 2015 and international guidelines20 and provide much more detail than
concluded there was no evidence of a relation between HPV journal publications. For example, the publication for one HPV
vaccination and the two syndromes.7 Independent research8-10 vaccine trial (NCT00122681) is 14 pages long,21 whereas its
and systematic reviews11-13 drew the same conclusion. However, corresponding clinical study report is more than 7000 pages.22
we believe there is reason to be cautious. EMA did not do the However, systematic reviews rarely use clinical study reports.
assessment itself but relied on the vaccine manufacturers’ A systematic review of randomised trials can identify even
analyses of the underlying harm data.14 Furthermore, the relatively rare harms, provided the trials are large enough and
systematic reviews were based exclusively or predominantly have adequate follow-up. We thought that the HPV vaccine
on journal publications, which often are limited by publication trials were sufficiently large and with long enough follow-up
and other reporting biases.15 We have previously shown reporting to potentially resolve whether POTS and CRPS could be
bias of the HPV vaccine studies; in our analysis one third of the causally
Page 142 of 637 linked to the HPV vaccines if all the data could be
For personal use only: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe
ANALYSIS
obtained.23 We therefore decided to conduct a systematic review events in the reports we downloaded were heavily redacted (fig
using the clinical study reports and their patient level data with 2).
serious harms narratives and case report forms, which provide From our incomplete data we identified several areas that needed
the most detailed information on harms.23 However, most of the clarification, especially the choice of comparator and the clinical
data are not publicly available and must be requested. evidence regarding the effects of the adjuvant used in the
vaccines. We put our questions to the manufacturers and
Accessing clinical study reports regulators. GlaxoSmithKline answered most of our questions,
but Merck answered just one of eight satisfactorily. For example,
In May 2014, we requested the HPV vaccine clinical study
we asked why it did not use an inert placebo injection in any of
reports from EMA. Since 2010 it has had a policy that, “access
the Gardasil 4 trials. An inert placebo most closely replicates
to documents or parts thereof may be granted whenever an
the real life choice people must make on vaccination (that is, to
over-riding public interest in disclosure can be identified by the
have vaccination or not), and use of an adjuvant in the control
Agency.”24 However, EMA initially denied our request on the
group could have made it harder to detect any harms from the
grounds that it “would undermine the protection of commercial
vaccine. Merck responded with a four page description of the
interests.” We appealed, arguing that the public interest
properties of aluminium salts but provided no explanation for
argument for a global public health intervention like the HPV
its choice of a non-placebo comparator. After 14 months, EMA
vaccines was overwhelming. Subsequently, EMA approved our
has not responded to our inquiries.
request and began to release the clinical study reports in
September 2014. Ultimately, we ended up including clinical study reports for 24
of the 48 potentially eligible studies using 12 reports obtained
Through our searches as part of our ongoing systematic review, 23
from EMA and 16 obtained from GlaxoSmithKline; for four
we had identified 206 comparative studies, 48 of which we
trials we obtained reports both from EMA and GlaxoSmithKline.
judged to be industry studies likely to have clinical study reports
The 24 studies represented about 80% of the participants. The
potentially eligible for our systematic review.16 However, EMA
reports’ incompleteness and redactions meant that our systematic
informed us that it held clinical study reports for only 29
review will be limited by reporting bias, which we had hoped
industry studies, which meant that, even in a best case scenario,
our systematic review could reduce.23
it would be unable to provide us with clinical study reports for
all the studies we wished to assess. After three years, we had
obtained just 18 clinical study reports (62% of EMA’s 29 Better process
reports), of which 12 were eligible for our review. There are signs of hope for more transparency and sharing of
EMA released over 35 000 pages for the 18 clinical study reports trial data: the European Union court recently ruled that whole
(table 1) in 61 batches. Unfortunately, the reports still lacked clinical study reports cannot be considered commercially
important sections, such as protocols and serious harms confidential27; the Food and Drug Administration (FDA)
narratives, and most reports contained redactions of allocation launched a pilot programme to release clinical study reports28;
numbers, vaccine batch numbers, and study centre and and Health Canada has said it will begin sharing trial data.29 But
participant ID numbers (fig 1). Only three reports included much work remains to increase the validity of systematic
completed case report forms. reviews that use clinical study reports and other regulatory trial
EMA’s release of the documents in so many batches made it data to reduce reporting bias.
difficult to keep track of the data, with clinical study reports In our view, independent researchers ought to be able to obtain
often divided across several files and across batches. For complete and unredacted clinical study reports within a
example, one study report (HPV-008) of 4263 pages was reasonable time frame without too many constraints or
released in 17 files across seven batches over 12 months. limitations—especially when potential serious harms are
In January 2017, we were approaching our data lock date of 1 reported after regulatory approval. In October 2016 we
July 2017 for our systematic review23 and sent EMA a prioritised complained to the European ombudsman about the problems
list of 11 clinical study reports from the largest and longest HPV with getting reports for our research, but the ombudsman judged
vaccine trials that would most benefit our systematic review. in early 2018 that EMA’s actions were “reasonable” and did not
We also asked EMA to speed up the release of the reports. EMA constitute maladministration (see supplementary data on
subsequently informed us that the slow release was due to the bmj.com).
complexity of our request, a high volume of data requests they The slow release and high demand reported by EMA may
were handling—in 2017, industry submitted 379 of 865 warrant extra staff dedicated to releasing clinical study reports.
requests25—and too few staff, and that the clinical study reports However, because of EMA’s workload and staff loss during
were, “released [to us] as submitted by the company except for Brexit, it recently scaled back its data sharing policies, limiting
the redactions that might have been applied,” confirming that one to EU citizens and temporarily suspending publishing new
for some studies, the clinical study reports that industry provides data packages under the other policy.30 Nevertheless, since public
to EMA are incomplete (eg, missing appendices). interests ought to trump commercial interests, we believe that
As the release was slow we considered obtaining clinical study independent researchers should be granted priority for access
reports directly from the vaccine manufacturers. However, to clinical study reports over industry.
Merck requires that researchers do not disclose data to third Redaction policies also need to be reconsidered so that benefits
parties,26 and GlaxoSmithKline grants access to complete trial and harms can be fairly assessed. US survey data indicate that
data only through a portal that prohibits the download and public most people are willing to have their data shared with
distribution of data. These policies conflicted with our aim to independent researchers.31 While it is important that participants
make the underlying data for our systematic review publicly remain anonymous and efforts to ensure an acceptably low risk
available,23 and we decided not to pursue this route. Although of identification should be maintained, liability for
GlaxoSmithKline publishes versions of its clinical study reports re-identification could rest with those who assess the clinical
on its trial register, the reports often lack serious adverse event study reports and be punishable by law.
narratives and case report forms, and the data on serious adverse
Page 143 of 637
ANALYSIS
We notified the ombudsman of some of our recommendations 4 Chandler RE, Juhlin K, Fransson J, Caster O, Edwards IR, Norén GN. Current safety
concerns with human papillomavirus vaccine: a cluster analysis of reports in VigiBase.
(such as replacing patient ID numbers with novel ID numbers),
Drug Saf 2017;40:81-90. 10.1007/s40264-016-0456-3 27638661
but the ombudsman supported EMA’s argument—that replacing 5 Brinth LS, Pors K, Theibel AC, Mehlsen J. Orthostatic intolerance and postural tachycardia
syndrome as suspected adverse effects of vaccination against human papilloma virus.
patient ID numbers would not sufficiently reduce the risk of Vaccine 2015;33:2602-5. 10.1016/j.vaccine.2015.03.098 25882168
patient identification—and judged EMA’s allocation of staff 6 Kinoshita T, Abe RT, Hineno A, Tsunekawa K, Nakane S, Ikeda S. Peripheral sympathetic
nerve dysfunction in adolescent Japanese girls following immunization with the human
(12.5 full-time equivalents handling requests) as “reasonable.” papillomavirus vaccine. Intern Med 2014;53:2185-200.
10.2169/internalmedicine.53.3133 25274229
We encourage regulators and industry to enhance their release
7 European Medicines Agency. Human papillomavirus vaccines. EMA/788882/2015. 2015.
of clinical study reports to independent researchers. In particular, http://www.ema.europa.eu/ema/index.jsp?curl=pages/medicines/human/referrals/Human_
they should respond to future requests by providing a detailed papillomavirus_vaccines/human_referral_prac_000053.jsp&mid=WC0b01ac05805c516f
8 Feiring B, Laake I, Bakken IJ, etal . HPV vaccination and risk of chronic fatigue
list of the clinical study reports they hold (including which parts syndrome/myalgic encephalomyelitis: a nationwide register-based study from Norway.
of each report are available) and an estimation of when the Vaccine 2017;35:4203-12. 10.1016/j.vaccine.2017.06.031 28648542
9 Miranda S, Chaignot C, Collin C, Dray-Spira R, Weill A, Zureik M. Human papillomavirus
reports will start to be released and how long it will take. Most vaccination and risk of autoimmune diseases: a large cohort study of over 2 million young
importantly, regulators should release complete and coherent girls in France. Vaccine 2017;35:4761-8. 10.1016/j.vaccine.2017.06.030 28750853
10 Scheller NM, Pasternak B, Mølgaard-Nielsen D, Svanström H, Hviid A. Quadrivalent HPV
clinical study reports. vaccination and the risk of adverse pregnancy outcomes. N Engl J Med 2017;376:1223-33.
In her decision, the ombudsman concluded that EU rules on 10.1056/NEJMoa1612296 28355499
11 Meggiolaro A, Migliara G, La Torre G. Association between human papilloma virus (HPV)
access to documents “are ill-suited to the purpose of making vaccination and risk of multiple sclerosis: a systematic review. Hum Vaccin Immunother
(large amounts of) scientific data available to researchers.”32 2018;14:1266-74. 10.1080/21645515.2017.1423155 29333935
12 Harder T, Wichmann O, Klug SJ, van der Sande MAB, Wiese-Posselt M. Efficacy,
The rules clearly need amending. Urgent changes are essential effectiveness and safety of vaccination against human papillomavirus in males: a
for open and transparent assessment of the harms and benefits systematic review. BMC Med 2018;16:110. 10.1186/s12916-018-1098-3 30016957
13 Arbyn M, Xu L, Simoens C, Martin-Hirsch PPL. Prophylactic vaccination against human
of interventions. papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database Syst
We thank EMA and GlaxoSmithKline for making clinical study reports available. Rev 2018;5:CD009069. 10.1002/14651858.CD009069.pub3. 29740819
14 Jefferson T, Jørgensen L. Human papillomavirus vaccines, complex regional pain
We thank GlaxoSmithKline for providing answers to our questions. syndrome, postural orthostatic tachycardia syndrome, and autonomic dysfunction—a
review of the regulatory evidence from the European Medicines Agency. Indian J Med
Ethics 2017;2:30-7. 10.20529/IJME.2017.0062017;1 27867145
Contributors and sources: PG and TJ are experienced systematic reviewers. TJ 15 Golder S, Loke YK, Wright K, Norman G. Reporting of adverse events in published and
and PD were coauthors of the first Cochrane review based on clinical study reports. unpublished studies of health care interventions: a systematic review. PLoS Med
2016;13:e1002127. . 10.1371/journal.pmed.1002127 27649528
LJ is an evidence based medicine critic at the Nordic Cochrane Centre and is
16 Jørgensen L, Gøtzsche PC, Jefferson T. Index of the human papillomavirus (HPV) vaccine
coauthor of several articles on HPV vaccine assessment. This article originated industry clinical study programmes and non-industry funded studies: a necessary basis
to address reporting bias in a systematic review. Syst Rev 2018;7:8.
from three years of preparatory work we carried out as part of a systematic review
10.1186/s13643-018-0675-z 29347995
of HPV vaccines. All authors contributed to the conception, drafting, and revision 17 Gøtzsche PC, Jørgensen KJ, Jefferson T, Auken M, Brinth L. Our comment on the decision
of the article and approved the final submission. TJ is the guarantor. by the European Ombudsman about our complaint over maladministration at the European
Medicines Agency related to safety of the HPV vaccines. 2 Nov 2017. https://nordic.
Competing interests: We have read and understood BMJ policy on declaration of cochrane.org/sites/nordic.cochrane.org/files/public/uploads/nordic_cochrane_views_on_
the_ombudsmans_decision_2_nov_2017.pdf.
interests and declare the following interests. PD and TJ are recipients of a grant
18 Rapporteurs’ day 150 joint response assessment report of Gardasil 9. 25 Nov 2014.
from the Laura and John Arnold Foundation to run a RIAT Support Center and EMEA/H/C/3852.
were corecipients of a UK National Institute for Health Research grant 19 Hughes S, Cohen D, Jaggi R. Differences in reporting serious adverse events in industry
sponsored clinical trial registries and journal articles on antidepressant and antipsychotic
(HTA–10/80/01 Update and amalgamation of two Cochrane reviews: neuraminidase drugs: a cross-sectional study. BMJ Open 2014;4:e005535.
inhibitors for preventing and treating influenza in healthy adults and children). They 10.1136/bmjopen-2014-00553510.1136/bmjopen-2014-005535 25009136
20 International Council for Harmonisation of Technical Requirements for Pharmaceuticals
are also in receipt of a Cochrane Methods Innovations Fund grant to develop for Human Use. ICH guidelines. 2005. http://www.ich.org/products/guidelines.html
guidance on the use of regulatory data in Cochrane reviews. PD sued the attorney 21 Paavonen J, Naud P, Salmerón J, etal. HPV PATRICIA Study Group. Efficacy of human
papillomavirus (HPV)-16/18 AS04-adjuvanted vaccine against cervical infection and
general of Canada to get access to data on HPV vaccines. He has received funding
precancer caused by oncogenic HPV types (PATRICIA): final analysis of a double-blind,
from the American Association of Colleges of Pharmacy for a study to analyse randomised study in young women. Lancet 2009;374:301-14.
10.1016/S0140-6736(09)61248-4 19586656
written medical information on the possible harms of statins. PD is also an associate
22 GSK. Interim clinical study report for study 580299/008 (HPV-008). https://www.gsk-
editor of The BMJ and an unpaid member of the IMEDS steering committee at the clinicalstudyregister.com/files2/gsk-580299-008-clinical-study-report-redact.pdf
Reagan-Udall Foundation for the FDA, which focuses on drug safety research. TJ 23 Jørgensen L, Gøtzsche PC, Jefferson T. Benefits and harms of the human papillomavirus
vaccines: systematic review of industry and non-industry study reports. PROSPERO
is occasionally interviewed by market research companies about phase I or II 2017: CRD42017056093. https://www.crd.york.ac.uk/PROSPEROFILES/56093_
drugs. He has acted as an expert witness in litigation cases related to the antiviral PROTOCOL_20170030.pdf
24 European Medicines Agency. Policy on access to documents: policy/0043. 2010. http://
oseltamivir and influenza vaccines and been a scientific adviser to a legal team
www.ema.europa.eu/docs/en_GB/document_library/Other/2010/11/WC500099473.pdf
acting on oseltamivir. TJ was a member of three advisory boards for Boerhinger 25 European Medicines Agency.Annexes to the annual report of the European Medicines
Agency 2017. 2018. (p129). http://www.ema.europa.eu/docs/en_GB/document_library/
Ingelheim and a member of an independent data monitoring committee for a Sanofi
Annual_report/2018/05/WC500248491.pdf
Pasteur clinical trial of an influenza vaccine. From 1994 to 2013, TJ was the 26 Merck procedure on access to clinical trial data. https://www.merck.com/clinical-trials/pdf/
coordinator of the Cochrane Vaccines Field. He is coholder of a Jean Monnet Merck%20Procedure%20on%20Clinical%20Trial%20Data%20Access%20Final_Updated%
20July_9_2014.pdf
Network Grant. He is an unpaid collaborator to the project Beyond Transparency 27 EMA. General court confirms EMA approach to transparency. 6 Feb 2018. http://www.
in Pharmaceutical Research and Regulation led by Dalhousie University and funded ema.europa.eu/docs/en_GB/document_library/Press_release/2018/02/WC500243216.
pdf
by the Canadian Institutes of Health Research. LJ and PCG have no competing 28 Doshi P. FDA to begin releasing clinical study reports in pilot programme. BMJ
interests to declare. 2018;360:k294. 10.1136/bmj.k294 29363507
29 Iacobucci G. Canadian government ordered to release unpublished Tamiflu data in
Provenance and peer review: Not commissioned; externally peer reviewed. landmark ruling. BMJ 2018;362:k3112. 10.1136/bmj.k3112. 30006357
30 Doshi P. EMA scales back transparency initiatives because of workload. BMJ
2018;362:k3513. 10.1136/bmj.k3513 30108101
1 WHO. Human papillomavirus (HPV) vaccines: WHO position paper. Wkly Epidemiol Rec
31 Mello MM, Lieou V, Goodman SN. Clinical trial participants’ views of the risks and benefits
2017;92:241-68. http://www.who.int/immunization/policy/position_papers/hpv/en/.28530369
of data sharing. N Engl J Med 2018;378:2202-11. 10.1056/NEJMsa1713258 29874542
2 28 Women miscarry after receiving HPV vaccine Gardasil; FDA says no reason to
32 European Ombudsman. Decision in case 1602/2016/JAS on the European Medicines
re-examine approval. Fox News 2007 Dec 6. http://www.foxnews.com/story/2007/12/06/
Agency’s handling of an access to documents request related to clinical study reports.
28-women-miscarry-after-receiving-hpv-vaccine-gardasil-fda-says-no-reason-to-re.html
2018. https://www.ombudsman.europa.eu/en/decision/en/89507
3 Rennie R. Schoolgirls ill after cervical cancer jabs. Sydney Morning Herald 2007 May 23.
http://www.smh.com.au/news/health/schoolgirls-ill-after-cervical-cancer-jabs/2007/05/22/ Published by the BMJ Publishing Group Limited. For permission to use (where not already
1179601385261.html granted under a licence) please go to http://group.bmj.com/group/rights-licensing/
permissions
Page 144 of 637

ANALYSIS
Table
Table 1| Details of EMA’s release of 18 clinical study reports from September 2014 to July 2017*
Content
Study Protocol Main body Serious harms narratives Case report forms for serious harms No of pages released
Cervarix
HPV-001 Yes Yes Yes Yes—forms filled in 5813
HPV-008 No Yes Yes Yes—forms filled in 4263
HPV-012 Yes Yes Yes Yes—blank forms 3153
HPV-013 No Yes No No 382
HPV-014 No Yes Yes No 238
HPV-015 No Yes No No 543
HPV-040 No Yes No Yes—blank forms 128
HPV-070 No Yes No Yes—blank forms 353
Gardasil
V501-005 No Yes Yes No 357
V501-012 Yes Yes Yes No 397
V501-013 Yes Yes No No 1797
V501-015 Yes Yes No No 713
V501-019 Yes Yes Yes Yes—blank forms 2645
Gardasil 9
V503-001 Yes Yes Yes Yes—forms filled in 9523
Total released 9 18 12 3 filled in +4 blank 35 253
* Items marked No were not released by our data lock of 1 July 2017 because of lack of clarity in our request or for unclear reasons.
Page 145 of 637

ANALYSIS
Figures
Fig 1 Example of a redaction applied by EMA to study centre numbers of a clinical study report (Cervarix study HPV-014)
Fig 2 Example of a redaction applied by GlaxoSmithKline to serious harms in a clinical study report (Cervarix study HPV-015)
Page 146 of 637

Supplementary material provided by the authors
Media reports of HPV vaccine related harms 2007-17
Year Press report
Fox News: “28 Women Miscarry After Receiving HPV Vaccine Gardasil; FDA Says No Reason
2007 to Re-Examine Approval.” December 6, 2007.
Sydney Morning Herald: “Schoolgirls Ill after Cervical Cancer Jabs.”
NBC News: “Some Girls Fainting after Receiving HPV Vaccine.”

http://www.nbcnews.com/id/22492557/ns/health-childrens_health/t/some-girls-fainting-
2008 after-receiving-hpv-vaccine/
US News and World Report: “Is HPV Vaccine to Blame for a Teen’s Paralysis?”
http://health.usnews.com/health-news/blogs/on-women/2008/07/02/is-hpv-vaccine-to-
blame-for-a-teens-paralysis
Telegraph: “Two Thousand Schoolgirls Suffer Suspected Ill-Effects from Cervical Cancer
Vaccine.” http://www.telegraph.co.uk/news/health/news/6178045/Two-thousand-
2009 schoolgirls-suffer-suspected-ill-effects-from-cervical-cancer-vaccine.html
ABC News: “Is the HPV Shot Safe for Teenage Girls?”
http://abcnews.go.com/Health/CancerPreventionAndTreatment/gardasil-hpv-vaccine-faces-
safety-questions/story?id=8356717
NBC 5: “Parents, Teens Question Safety of HPV Vaccine.”

http://www.nbcdfw.com/news/health/Parents-Teens-Question-Safety-of-HPV-Vaccine-
2010 94745174.html
Washington Post: “Can HPV Vaccine Cause Mental Retardation?”

http://www.washingtonpost.com/national/health-science/can-hpv-vaccine-cause-mental-
2011 retardation/2011/09/14/gIQAUpH4SK_video.html
NZ Herald: “Mother Blames Cancer Vaccine for Teen’s Death.”

2012 http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10825493
LifeSiteNews: “Parents Sue after Quebec Teen Dies Following Gardasil Vaccination.”
https://www.lifesitenews.com/news/parents-sue-after-quebec-teen-dies-following-gardasil-
vaccination
The Globe and Mail: “Why Some Parents Still Won’t Give Daughters the HPV Vaccine.”
https://www.theglobeandmail.com/life/health-and-fitness/health/why-some-parents-still-
wont-give-daughters-the-hpv-vaccine/article4616841/
2013 TV2: “13-Årige Sara Syg I Fem Måneder Efter HPV-Vaccine - TV 2.”
Page 147 of 637

http://nyheder.tv2.dk/article.php/id-71216717:13%C3%A5rige-sara-syg-i-fem-
m%C3%A5neder-efter-hpvvaccine.html
TV2: “19-Årige Astrid Sidder I Kørestol: Tror Det Skyldes HPV-Vaccine.”

http://nyheder.tv2.dk/article.php/id-71228321:19%C3%A5rige-astrid-sidder-i-
k%C3%B8restol-tror-det-skyldes-hpvvaccine.html
KMBC: “Teen Suffers Health Problems after Receiving HPV Vaccine.”

http://www.kmbc.com/article/teen-suffers-health-problems-after-receiving-hpv-vaccine-
1/3676021
Washington Times: “U.S. Court Pays $6 Million to Gardasil Victims.”

http://www.washingtontimes.com/news/2014/dec/31/us-court-pays-6-million-gardasil-
victims/
Medical Daily: “16-Year-Old Girl Becomes Infertile After Gardisal HPV Vaccine: Loses All
Ovarian Function, Goes Into Menopause.” http://www.medicaldaily.com/16-year-old-girl-
becomes-infertile-after-gardisal-hpv-vaccine-loses-all-ovarian-function-goes-250241
The Washington Times: “HPV Vaccine Cited in Infertility Case.”

www.washingtontimes.com/news/2013/nov/11/hpv-vaccine-cited-in-infertility-case/
CBS News: “Side Effect Fears Stop Parents from Getting HPV Vaccine for Daughters.”
https://www.cbsnews.com/news/side-effect-fears-stop-parents-from-getting-hpv-vaccine-
for-daughters/
Fox News: “’Scariest Thing in My Entire Life!’ Mother Says Her Daughter Rushed to the ER
after Receiving HPV Vaccine.” http://fox6now.com/2014/08/08/scariest-thing-in-my-entire-
2014 life-mother-says-her-daughter-rushed-to-the-er-after-receiving-hpv-vaccine/
La Depeche: “Le Gardasil a-T-Il Tué La Jeune Adriana, 17 Ans?”

http://www.ladepeche.fr/article/2014/04/08/1858904-pour-nous-notre-fille-est-morte-a-
cause-du-gardasil.html
The Sun: “Side-Effects of Cancer Jab the HPV Vaccine Almost Killed Us.”
https://www.thesun.co.uk/archives/health/743747/side-effects-of-cancer-jab-almost-killed-
us/
News 9: “Moore Woman Blames Gardasil For Disabling Her.”

http://www.news9.com/story/26177426/moore-woman-blames-gardasil-for-disabling-her
SRF: “Verdacht Auf Schwere Nebenwirkung Bei Krebsprävention Für Mädchen.”

https://www.srf.ch/news/schweiz/verdacht-auf-schwere-nebenwirkung-bei-
krebspraevention-fuer-maedchen
Page 148 of 637

Independent: “Another 18 Girls Claim Adverse Reactions to the HPV Vaccine.”
http://www.independent.ie/irish-news/health/another-18-girls-claim-adverse-reactions-to-
2015 the-hpv-vaccine-34282433.html
LifeSiteNews: “Gardasil Linked to Deaths and Disabilities after Young Girls Vaccinated:
Toronto Star Investigation.” https://www.lifesitenews.com/news/gardasil-linked-to-deaths-
and-disabilities-after-young-girls-vaccinated-tor
Coach Nine: “Mother Blames HPV Vaccine for Her Daughter’s Constant, Excruciating Pain.”
http://coach.nine.com.au/2015/07/24/11/15/mother-blames-hpv-vaccine-for-her-
daughters-constant-excruciating-pain
Independent: “Teenagers Enduring Illnesses after Routine Cancer Vaccination.”

http://www.independent.co.uk/life-style/health-and-families/thousands-of-teenage-girls-
report-feeling-seriously-ill-after-routine-school-cancer-vaccination-10286876.html
The Daily Beast: “She Says the HPV Vaccine ‘Disabled’ Her Daughter.”
http://www.thedailybeast.com/articles/2015/11/06/she-says-the-hpv-vaccine-disabled-her-
daughter
Romford Recorder: “Hornchurch Mother Tells of Daughter’s Debilitating Illness as Recorder

Investigates HPV Vaccine.” http://www.romfordrecorder.co.uk/news/health/hornchurch-
mother-tells-of-daughter-s-debilitating-illness-as-recorder-investigates-hpv-vaccine-1-
4237783
TV2: “De Vaccinerede Piger - Syge Og Svigtede.”

http://play.tv2.dk/programmer/dokumentar/serier/de-vaccinerede-piger/syge-og-svigtede-
97048/
Los Angeles Times: “How a Major Newspaper Bungled a Vaccine Story, Then Smeared Its
Critics.” http://www.latimes.com/business/hiltzik/la-fi-mh-how-a-major-newspaper-
20150213-column.html
Daily Mail: “Just How Safe Is the Cervical Cancer Jab?”

http://www.dailymail.co.uk/health/article-3106372/Just-safe-cervical-cancer-jab-families-
say-daughters-suffered-devastating-effects-HPV-vaccine-experts-worried-too.html
Express: “Tens of Thousands of Teen Girls Suffer Serious Illnesses after HPV Cervical Cancer
Jab.”
http://www.express.co.uk/life-style/health/581393/Tens-of-thousands-teen-girls-suffer-
serious-illnesses-HPV-cervical-cancer-health-concerns
Kent News: “Did HPV Cervical Cancer Vaccine Cause Life-Changing Illnesses in Kent
Teenagers?”
http://www.kentnews.co.uk/news/did-hpv-cervical-cancer-vaccine-cause-life-changing-
illnesses-in-kent-teenagers-1-4241748
Page 149 of 637

Kobe News: “63 Patients with Cervical Cancer Vaccine Sued by Country.”
2016 https://www.kobe-np.co.jp/news/iryou/201608/0009328447.shtml
The Sun: “Mum Claims Cervical Cancer Jab Left Her Daughter, 13, Wheelchair Bound and
Suffering from Narcolepsy.” https://www.thesun.co.uk/news/1360639/mum-claims-
cervical-cancer-jab-left-her-13-year-old-daughter-wheelchair-bound-and-suffering-from-
narcolepsy/
The Sun: “Teen Girl Gets HOSPITALISED Every Time She Has Her Period - and Mum Blames
HPV Vaccine.” https://www.thesun.co.uk/living/1389913/teenage-girl-gets-such-severe-
bouts-of-vomiting-during-her-periods-that-she-gets-hospitalised-and-her-mum-thinks-the-
hpv-vaccine-is-the-cause/
Corriere: “Vaccino Anti-Papilloma Virus: Ancora Polemiche Su Possibili Effetti Avversi.”

http://www.corriere.it/salute/sportello_cancro/16_gennaio_04/vaccino-anti-papilloma-
virus-ancora-polemiche-possibili-effetti-avversi-95f286c6-b2db-11e5-8f58-
73f8cf689159.shtml
Telegraph: “Teenage Girl Dies Five Days after Receiving HPV Vaccine Jab.”
http://www.telegraph.co.uk/news/2016/05/10/teenage-girl-dies-five-days-after-receiving-
hpv-vaccine-jab/
IOL News: “Cervical Cancer Vaccine Gave My Child Brain Disease.”

https://www.iol.co.za/news/south-africa/western-cape/cervical-cancer-vaccine-gave-my-
2017 child-brain-disease-8791732
STV: “HPV: Chloe’s Warning to Other Girls about Side-Effects of Jab.”

https://stv.tv/news/features/1386706-hpv-chloe-s-warning-to-other-girls-about-side-
effects-of-jab/
Corriere: “I Vaccini Contro Il Papilloma Virus E Gli Effetti Indesiderati.”

http://www.corriere.it/inchieste/reportime/salute/i-vaccini-contro-papilloma-virus-effetti-
indesiderati/023243d2-221e-11e7-807d-a69c30112ddd.shtml
Wired: “Le Domande Sui Dubbi Di Report Sul Vaccino Anti-Hpv.”

https://www.wired.it/scienza/medicina/2017/04/18/domande-report-vaccino-anti-hpv/
The Sun: “Mum Claims HPV Vaccine Has Left Her Football-Loving Teenage Daughter
Wheelchair-Bound.” https://www.thesun.co.uk/living/3507121/mum-claims-hpv-vaccine-
has-left-her-football-loving-teenage-daughter-wheelchair-bound-like-an-old-lady-in-a-
teenagers-body/
Il Post: “Vaccini Contro Il Papilloma Virus.”

http://www.ilpost.it/2017/04/18/report-vaccini-papilloma-virus-hpv/
Page 150 of 637

Japan Times: “Suit Opens in Tokyo Court over Cervical Cancer Vaccine Side Effects.”
https://www.japantimes.co.jp/news/2017/02/13/national/crime-legal/suit-opens-tokyo-
court-cervical-cancer-vaccine-side-effects/
Metro: “Teenager Paralysed after Having the HPV Cervical Cancer Vaccine.”
http://metro.co.uk/2017/03/03/teenager-paralysed-after-having-the-hpv-cervical-cancer-
vaccine-6485548/
Daily Star: “Teen Dancer Left Paralysed after HPV Cervical Cancer Vaccine.”
http://www.dailystar.co.uk/news/latest-news/593546/hpv-cervical-cancer-vaccine-
teenager-dancer-paralysed-jab
Slate: “What the Gardasil Testing May Have Missed.”

https://slate.com/health-and-science/2017/12/flaws-in-the-clinical-trials-for-gardasil-made-
it-harder-to-properly-assess-safety.html
Irish Times: “Almost 650 girls needed medical intervention after HPV vaccine.”
https://www.irishtimes.com/news/health/almost-650-girls-needed-medical-intervention-
after-hpv-vaccine-1.3217346
Page 151 of 637

Additional file
Challenges of independent assessment of potential harms of HPV vaccines
Table of contents
1. Complaint to the Ombudsman (27 October 2016) ............................. pp. 2-5

2. The Ombudsman's response (8 February 2018) .................................. p. 6
3. Questions to Merck (14 March 2017).................................................. pp. 7-10
4. Response from Merck (10 April 2017)................................................. pp. 11-57
5. Follow-up questions to Merck (24 April 2017) .................................... pp. 58-60
6. Follow-up response from Merck (23 May 2017) ................................. pp. 61-94
7. Additional follow-up questions to Merck (7 June 2017) ....................... pp. 95-96
8. Questions to GlaxoSmithKline (16 June 2017)..................................... pp. 97-98
9. Response from GlaxoSmithKline (27 July 2017) .................................. pp. 99-108
10. Questions to EMA (4 July 2017) ........................................................... pp. 109-111
11. EMA ( 2018) ................p. 112
Page 152 of 637

Complaint to the Ombudsman – Dr Tom Jefferson
POINT 1: Rate of release of data from EMA under policy 00043
I requested Human Papilloma Virus vaccines Clinical Study Reports (Cervarix, Gardasil) and the anti-HCV
th
molecule Sofosbivir (Sovaldi) on May 29 2014.
My request was refused under CCI exception on the 23 June 2014
I appealed on 1 July 2014
The result of the appeal was never communicated to me but on 17 July 2014 I was sent the list of EMA
holdings containing 8 Gardasil 4 CSRs and 20 Cervarix CSRs.
I was asked to prioritise their release, which I did. The first release took place on 12 September 2014.
To date (END OF Setpember 2016) there have been 80 transmissions of as many file batches: 8 Gardasil 4
CSRs (total 10216 pages) and 5 Cervarix CSRs (12009 pages). Average batch size is 300-400 pages.
Main consequences:
12
Very slow release (increasingly so, as documented )
Batching of CSRs files (300-400 pages at a time) makes it very difficult to keep track of what is going on and
needs software to reconstruct a single CSR from multiple batches of files
POINT 2: Excessive unnecessary anonymization and failure to assign fake ID to line listing
This concerns individual participant data tables, in which each participant’s relevant data are reported, I,e,
medications [also batch numbers of ampoules of biologics - see below].
Below is a reproduction from one the listings Cervarix Trial HPV-008 – CSR main body pdf page 269/641:
As can be seen both the Case number and the Subject Number have been redacted, without assigning the
participant a fake ID.
Main consequences: the impossibility of following an individual through the CSRs narrative makes
interpretation impossible or difficult especially when this person is a subject of a Serious Adverse Event
report narrative. In other words, this type of redaction defeats the object of transparency. Of note is that GSK
appear to have used progressively more redactions, as when GSK released 30 complete CSRs directly to us
in 2013, only the participant ID had been redacted.
Page 153 of 637

I have complained and appealed on the basis of the public health importance of HPV vaccines and my lack
of interest in both identification of participants and of reconstructing the original ID allocated. I received the
following answer on 28 September 2016:
“As a preliminary note, we would like to highlight that we cannot consider your email to constitute a “formal
appeal”. Regulation (EC) No 1049/2001 (the “ATD Regulation”) does not establish a second route of internal
appeal after a confirmatory application has been submitted.
The fourth paragraph of Article 73 of Regulation (EC) No 726/2004, making explicit reference to the legal
regime established by the ATD Regulation, sets out the remedies available to you should you disagree with
th
the Agency’s confirmatory decision. These remedies were highlighted in our decision letter of 10 August
2016 regarding batch 28 of your confirmatory application:
“should you wish to avail yourself of the remedies available under Union law against this decision,
please be informed that you can bring a complaint before the European Ombudsman, pursuant to
Article 228 of the Treaty on the Functioning of the European Union (TFEU). You can also institute
legal proceedings before the General Court of the European Union in accordance with Article 263 of
the TFEU.”
The Agency would also like to highlight that the ATD Regulation does not permit us to modify the content of
documents released in response to requests for access to documents. Article 10(3) of this Regulation clearly
states that “Documents shall be supplied in an existing version and format”.
The General Court ruled on a number of occasions (for example in Case Dufour v ECB, T-436/09, ECR,
EU:T:2011:634, paragraph 149; and in Case Typke v Commission, T-214/13, ECR, EU:T:2015:448) that if
an application for access to documents requires the creation of a new document, even if this new requested
document would be based on information already appearing in existing documents held by the institution,
such application does not come within the parameters of Regulation No 1049/2001. Therefore, the Agency is
not permitted to modify an existing document but only to apply redactions to it in order to protect information
covered by the exceptions set out in Article 4 of the ATD Regulation. This is the reason why the Agency
cannot apply anonymisation (i.e. replacing information with another) to the documents that are subject to
your request for access to documents but can only redact certain information contained therein.
We completely understand that you may encounter difficulties in your work as a consequence of the
redaction of patient identification numbers or case numbers. The Agency is also fully aware of the existing
public attention regarding HPV vaccines.
However, in accordance with Article 4(1)(b) of the ATD Regulation and the European Union legislation
regarding the protection of personal data read in conjunction with Regulation (EC) No 45/2001, the Agency
has to redact all protected personal data in order to avoid that the disclosure of the document would
undermine the privacy and integrity of any individual.
In accordance with applicable EU data protection legislation, information is considered personal data in so
far as it refers to “an identified or identifiable individual person” and that an “identifiable person” is one that
can be identified, directly or indirectly, “in particular by reference to an identification number or to one or
more factors specific to his or her physical, […] identify” (Article 2(a) of Regulation (EC) No 45/2001). The
position of the Agency is that coding of an individual trial participant (i.e. allocation of a random number to
each patient) would not eliminate the risk of a possible re-identification of the concerned individual and
therefore the public disclosure of this information could not be justified on the basis of current data protection
legislation.
In this regard, we would like to highlight that the replacement of patient direct identifiers (such as names)
with patient IDs is a known pseudonymisation technique. This technique is explained in the Opinion 05/2014
on anonymisation techniques of the Article 29 Data Protection Working Party (please see page 20). It is
considered a pseudonymisation technique because the unique number that is allocated to the research
subject (unique for the data set) allows for the linking of this number to an individual patient. In other words
the link between the research subject and the number does exist and is not undermined by its
pseudonymisation. Therefore from this perspective, the Agency considers that the data sets that are
submitted to the EMA for the regulatory review are not fully anonymised and are therefore not adequate for
Page 154 of 637

release to third parties in accordance with the ATD Regulation or other forms of public release. The risk of
releasing pseudonymised data sets is clearly highlighted in the Opinion 05/2014 on anonymisation
techniques of the Article 29 Data Protection Working Party which reads:
“A specific pitfall is to consider pseudonymised data to be equivalent to anonymised data. The Technical
Analysis section will explain that pseudonymised data cannot be equated to anonymised information as they
continue to allow an individual data subject to be singled out and linkable across different data sets.
Pseudonymity is likely to allow for identifiability, and therefore stays inside the scope of the legal regime of
data protection. This is especially relevant in the context of scientific, statistical or historical research.”
As you may know the patient IDs that are allocated to subject enrolled in clinical trials are not just random
codes. The patient ID numbers are typically made up in a sequence of numbers and letters that are
associated with three elements, the study, the study site and the patient. Therefore, it is understood that by
releasing the patient numbers that were allocated at the time of randomisation one would release information
about the geographical location of the patient. It is widely recognised that in addition to the direct identifiers,
the geographical location is one of the data elements that poses high risk of re-identification.
In addition these unique numbers are considered to constitute direct identifiers. The uniqueness should be
understood as the possibility to associate the number/code with only one research subject in a given data
set. This is documented in the literature, for example we can refer you to the decision rule for classifying
identifiers (figure B-3) which is available in the Appendix B of the IOM report: Sharing Clinical Trial Data:
Maximizing benefits, minimizing risks.
For all the reasons mentioned above, the EMA is redacting the patient ID numbers from the documents that
are released to the public in accordance with the ATD Regulation. The Agency conducts this exercise in light
of the sensitive nature of the information at stake, the applicable rule on processing of health data laid down
in Article 10 of Regulation (EC) 45/2001 and regulatory guidelines including for example the provisions of the
Recommendations on the handling of requests for access to Periodic Safety Update Reports (PSURs) “the
minimum personal data to be deleted to ensure anonymisation of the information would require the deletion
of information on 1) Date of birth; 2) (Reporting) country; 3) Patient identification code”. Furthermore, the
Recommendations provide that “it should never be possible to identify a natural person from the information
disclosed”.
The position adopted by EMA takes into account the advice of the European Data Protection Supervisor
(EDPS) regarding the extent to which information received for the purpose of pharmacovigilance, including
adverse reactions should be considered personal data in accordance with Article 2(a) of Regulation (EC)
45/2001.
We hope that you can understand the EMA’s position on this sensitive matter”.
As I’ve pointed out to EMA redactions of this type make the release of data a cosmetic exercise. The
situation gets worse when EMA redact batch numbers from vials of biologics such as vaccines making it
impossible to assess whether a particular effect observed (for example a suspected adverse event is due to
a spefici batch of that vaccine.
POINT 3: Lack of availability of a list of CSRs and other important material held by EMA
3
My group and I have complained in private and in public several times about the lack of visibility of what is
available under policy EMA 0043. EMA have failed to respond in any way.
Main consequences: sizeable increase in work load trying to identify availability or blanket (blind) requests
for data and unnecessary burden on requestor and EMA.
References
1. Doshi P, Jefferson T. Open data 5 years on: a case series of 12 freedom of information requests for
regulatory data to the European Medicines Agency. Trials (2016) 17:78 DOI 10.1186/s13063-016-1194-7.
http://trialsjournal.biomedcentral.com/articles/10.1186/s13063-016-1194-7
Page 155 of 637

2. Peter Doshi & Tom Jefferson. Access to clinical trial data from the European Medicines Agency.
http://blogs.biomedcentral.com/on-medicine/2016/02/11/access-clinical-trial-data-european-medicines-
agency/
3. Tom Jefferson and Peter Doshi: Thanksgiving special—menus needed at the EMA’s restaurant.
BMJ Blogs
Rome, 27 October 2016
Dr Tom Jefferson MD MSc FFPHM MRCGP
Page 156 of 637

The Ombudsman concluded “no maladministration” in response to Tom Jefferson’s case with the
ID 1602/2016/JAS.
The Ombudsman’s decision (dated 8 February 2018) in case 1602/2016/JAS can be found
online: http://www.ombudsman.europa.eu/en/decision/en/89507
Page 157 of 637

Alfred J. Saah, MD, MPH
Executive Director
Merck Research Laboratories
351 N. Sumneytown Pike (UG-3CDS003)
North Wales, PA 19454-2505
USA
alfred_saah@merck.com
Dear Dr Saah,
We are researchers carrying out a systematic review titled “Benefits and harms of human
papillomavirus vaccines: systematic review of industry clinical study reports and non-industry
published and unpublished reports.”
We have recently registered the protocol in PROSPERO1 and are in the process of gathering clinical
study reports to carry out the review, which covers all the HPV vaccines: Cervarix, Gardasil 4,
Gardasil 9, and experimental HPV vaccines.
We are contacting you with some inquiries based on our preliminary research and hope you might
help us in answering them.
Questions we have come up with thus far:
1. Why did Merck not use an inert placebo vaccine in any of the Gardasil 4 clinical trials?
Patients, doctors, and other decision makers are considering Gardasil against the option of no
Gardasil. As such, we were surprised to find that none of the randomized Gardasil 4 trials used
an inert placebo control. All trials used a control group that either received a solution containing
the vaccine adjuvant amorphous hydroxyphosphate sulfate (AAHS), another vaccine (for
example, Repevax) or other substances. To our knowledge, only protocol V503-006 from the
Gardasil 9 trial program used pure saline as a comparator.
2. Was AAHS ever approved by regulators before being used as an adjuvant in vaccines (not
necessarily limited to Gardasil) on the market?
We are unable to identify the transition from “aluminium salts” to “AAHS” used in Gardasil and
we are not sure whether regulators ever approved AAHS, and if so, based upon what data.
3. What are the clinical effects of the AAHS-containing control solution versus inert placebo?
We understand from correspondence with certain regulators (i.e., the European Medicines
Agency, EMA, Australia’s Therapeutic Goods Administration, TGA, Health Canada and New
Zealand’s MedSafe) that there is no separate registration for adjuvants (including AAHS). EMA
states that, “Experimental studies to demonstrate absorption, distribution, metabolism, and
1
Lars Jørgensen, Peter Christian Gøtzsche, Tom Jefferson. Benefits and harms of human
published and unpublished reports. PROSPERO 2017:CRD42017056093 Available from
http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42017056093
Page 158 of 637

excretion of the active ingredients in Gardasil have not been performed for any of the
component viruses. This is in line with Note for guidance on preclinical pharmacological and
toxicological testing of vaccines (CPMP/SWP/465/95).”2 The Merck Aluminium Adjuvant
(aluminium hydroxyphosphate sulphate adjuvant) is used in other vaccines, which are approved
in Europe, and it is agreed that no further studies on the adjuvant are required.3 The lack of
clinical evidence of AAHS-containing control solution versus inert placebo seems especially
important because according to Luxembourg et al4, AAHS has a higher concentration (i.e., a
factor of 1.88) than virus-like particles (VLPs) in the vaccines. Thus, we wonder what safety
studies Merck has undertaken of AAHS?
4. Are the clinical effects of AAHS consistent and predictable?

A 2013 Merck patent application for AAHS describes “intra-batch” and “batch-to-batch”
variability of the concentration of AAHS in its manufacturing process.5 From the patent text it
appears that AAHS may be chemically inconsistent in form. Can you confirm your understanding
of this point and its relevance to the issue of whether the clinical effects of AAHS are consistent
and predictable?
5. When did Merck become aware that the adjuvant in its adjuvanted vaccines was in fact AAHS?
We note that a number of vaccine descriptions refer to AAHS as: “Amorphous Aluminum
Hydroxyphosphate sulfate (previously called aluminum hydroxide).”6 However, the two are not
synonymous.
6. Why does Merck describe the comparator in trial HPV-018 as “normal saline” when it contains
more than water and sodium chloride?
The control in trial HPV 018 is described as “normal saline” however the control contained a
carrier solution (i.e., a control dose of 0.5 milliliters contained 9.56 milligrams of sodium
chloride, 0.78 milligrams of L-histidine, 50 micrograms of polysorbate 80, 35 micrograms of
sodium borate, <7 micrograms of yeast protein, and water). In clinical medicine, normal saline
consists of only sterile water with 0.9% sodium chloride to provide an isotonic solution.
7. Does Gardasil contain DNA/RNA and, if so, what is the clinical significance of this?
According to a document by Japan’s Pharmaceuticals and Medical Devices Agency (PMDA)
regulator there are viral DNA and RNA strands in unknown concentrations in Gardasil 4.7 The
2
Gardasil EPARs.
http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_-_Scientific_Discussion/human/0007
03/WC500021140.pdf
3
http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_-_Scientific_Discussion/human/0007
03/WC500021140.pdf
4
Luxembourg A, Brown D, Bouchard C, Giuliano AR, Iversen O-E, Joura EA, et al. Phase II studies to
select the formulation of a multivalent HPV L1 virus-like particle (VLP) vaccine. Human Vaccines &
Immunotherapeutics. 2015 Jun 3;11(6):1313–22.
http://www.tandfonline.com/doi/full/10.1080/21645515.2015.1012010
5
Method for preparation of aluminum hydroxyphosphate adjuvant (WO 2013078102 A1).
https://www.google.com/patents/WO2013078102A1?cl=en
6
Merck & co, Inc. Liquid PedvaxHIB data sheet 1998.
7
http://www.pmda.go.jp/drugs/2011/P201100122/170050000_22300AMX00600000_A100_2.pdf
Page 159 of 637

concentrations were redacted so we do not know what these concentrations were and we do
not know if this finding has any clinical significance. However, we know that RNA/DNA alone
injected intramuscularly into BALB-C mice activates the immune response.8
8. Are there any long-term Gardasil studies with cancer (i.e., cervical, anal, oropharyngeal, and
penile cancer) as a specific outcome and, if so, what are the studies identification numbers?
We have been unable to locate any study where actual cancer (i.e., not a surrogate for actual
cancer, such as, cervical/anal/penile intraepithelial neoplasia [CIN/AIN/PIN] etc.) is reported as
the specific outcome. We are aware that, in 2004, two years before the regulatory approval of
Gardasil, the World Health Organization (WHO) approved a surrogate outcome for cervical
cancer (i.e., cervical intraepithelial neoplasia grade two or more, CIN2+: CIN2, CIN3,
adenocarcinoma in situ and cervical cancer) for regulatory approval.9
We would be grateful for your response by 31 March 2017.
With best wishes,
8
Lee SH. Detection of human papillomavirus (HPV) L1 gene DNA possibly bound to particulate
aluminum adjuvant in the HPV vaccine Gardasil. J Inorg Biochem. 2012 Dec;117:85-92. doi:
10.1016/j.jinorgbio.2012.08.015.
9
Pagliusi SR, Teresa Aguado M. Efficacy and other milestones for human papillomavirus vaccine
introduction. Vaccine. 2004 Dec 16;23(5):569–78.
Page 160 of 637

Tom Jefferson, MD, MSc, FFPHM, MRCGP
Senior Associate Tutor
Oxford OX2 6GG
United Kingdom
Peter C Gøtzsche, DrMedSci, MSc, Director

Nordic Cochrane Centre
Rigshospitalet, Dept. 7811
Blegdamsvej 9
2100 Copenhagen Ø, Denmark
Lars Jørgensen, MD, PhD Student

Blegdamsvej 9
Peter Doshi, PhD, Assistant Professor

University of Maryland School of Pharmacy
Baltimore, Maryland, 21201 U.S.A.
14 March 2017
Page 161 of 637

Merck& Co.,Inc.
U.S.Human Health
P.O.Box1000
NorthWales,PA19454-1099
Tel:800672 6372
Fax:8006372568
April 10,2017
Peter Doshi, PhD.

Assistant Professor
0MERCK
20 North Pine Street
Baltimore MD 21201
Dear Dr. Doshi:
This is in response to your recent inquiry regarding GARDASIL (human papillomavirus quadrivalent (types 6, 11,
16, and 18) vaccine, recombinant). Your inquiry concerned specific information regarding the use of placebo in the
clinical trial program for GARDASIL:
• Why Merck did not use an inert placebo vaccine in any of the clinical trials for GARDASIL, and
• Why Merck describes the comparator in Protocol O18 as "normal saline" when it contains more than water and
sodium chloride.
Your inquiry also concerned specific information about the adjuvant in GARD AS IL:
• Whether AAHS was approved by regulators before being used as an adjuvant in vaccines on the market,
• The clinical effects of the AAHS-containing control solution versus inert placebo,
• Whether the clinical effects of AAHS are consistent and predictable, and
• When Merck became aware that the adjuvant in the adjuvanted vaccines was in fact AAHS.
In addition, you inquired whether GARDASIL contains DNA or RNA and the clinical significance of its presence.
The following information is provided as a professional service in response to your unsolicited inquiry. It is
intended to provide you with a review of the available scientific literature and/or data that you requested. Merck
does not recommend the use of its products in any manner other than as described in the prescribing information.
Enclosed for your convenience is prescribing information for GARDASIL.
If you require further information or need to report suspected adverse reactions, please contact the Merck National
Service Center at 1-800-672-6372.
Merck does not control nor assume any responsibility for the content of other Internet websites to which we provide
links.
Sincerely,
Richard Gersh, MD
Global Medical Information Physician
Enclosures:
GARDASIL Prescribing Information
Page 162 of 637

APRIL 10, 2017 PETER DOSHI, PHD. 2
PLACEBO IN CLINICAL TRIALS WITH GARDASIL
Information regarding the clinical study with GARDASIL that utilized placebo as a comparator is provided in the
Prescribing Information. Please see Section 6.1 Clinical Trials Experience on pages 5 through 11 for the safety
results of patients who received saline placebo. Please see section 14. Clinical Studies on page 13 for a description
of the design of the efficacy study and section 14.8 Immunogenicity on page 21 for the design of the
immunogenicity study.
ADJUV ANT IN GARDASIL
Provided below, for your review, is the most current information regarding the mechanism of action and safety of
the adjuvant in GARDASIL.
The following is a summary of the information pertinent to your inquiry. For detailed information, please refer to the
appropriate sections outlined below.
Summary
• The adjuvant used in GARDASIL is Amorphous Aluminum Hydroxyphosphate Sulfate (AAHS). Each
0.5-mL dose of the vaccine contains approximately 225 mcg of aluminum (provided as AAHS). (1)
• Aluminum has been used as an adjuvant in many vaccines licensed by the US Food and Drug
Administration (FDA) (2). Merck's proprietary Amorphous Aluminum Hydroxyphosphate Sulfate
adjuvant is used in other vaccines manufactured by Merck at quantities ranging from approximately
0.225 mg (225 mcg) to 0.5 mg (500 mcg) per dose (3). For the list of vaccines containing AAHS, refer to
'Aluminum Aqjuvant Used in Vaccines Manufactured by Merck'.
• Adjuvants are non-specific stimulators of the immune response (4). Preclinical studies in animals have
shown that although HPV virus-like proteins (VLPs) induced antibodies when delivered without an
adjuvant, formulations with an aluminum adjuvant generated significantly higher titers ofHPV-neutralizing
antibodies (3).
• There are three components of the mechanism of action which explain how adjuvants work.( 5-7) Refer to
'Mechanism of Action' for information regarding these potential mechanisms.
• Refer to 'Safety Profile' for information on common adverse events, serious local reactions, and allergic
responses to aluminum.(4;8)
• The US Agency for Toxic Substances and Disease Registry (ASTDR) has determined that the minimum
risk level for human exposure to aluminum from all sources is 2 mg/kg/day and it has been established that
the levels of aluminum that infants are exposed to through food and vaccines are less than this minimal risk
level.( 4;8). Refer to 'Safety Profile' below for additional information.
• The Global Advisory Committee on Vaccine Safety (GACVS), the Centers for Disease Control and
Prevention (CDC), and the American Academy of Pediatrics (AAP) provide statements on aluminum-
containing vaccines. Refer to 'General Recommendations' below.
Aluminum Adjuvant Used in Vaccines Manufactured by Merck
The proprietary aluminum-containing adjuvant used in vaccines manufactured by Merck is Amorphous Aluminum
Hydroxyphosphate Sulfate (AAHS) at quantities ranging from approximately 0.225 mg (225 mcg) to 0.5 mg
(500 mcg) per dose (3). This adjuvant is currently used in these Merck vaccines:
• GARDASIL [Human Papillomavirus Quadrivalent (Types 6, 11, 16, and 18) Vaccine, Recombinant]
• GARDASIL 9 [Human Papillomavirus 9-valent Vaccine, Recombinant]
Page 163 of 637

• PedvaxHIB [Haemophilus b Conjugate Vaccine (Meningococcal Protein Conjugate)]

• RECOMBIV AX HB [Hepatitis B Vaccine (Recombinant)]
• VAQTA (hepatitis A vaccine, Inactivated)
Background Information
Aluminum salts have historically been the most commonly used adjuvants in vaccines licensed for use in the United
States and in other countries (4). There are three types of aluminum salts used as adjuvants in vaccines, which differ
in their physical/chemical properties: (5)
• Aluminum hydroxide
• Aluminum phosphate
• Potassium aluminum sulfate (often called "alum")
There are also other adjuvants used in vaccines and other medications that have been licensed in the United States
and other countries.(4)
Medical Literature
A search of the published medical literature pertaining to GARD AS IL has identified references that are pertinent to
your inquiry. Summaries of these articles are provided below.
Mechanism of Action
The biologic activity of aluminum salts consists of three components:
• The formation of an antigen repository (depot) in the tissue for prolonged and sustained exposure to the
immune system,
• The production of particulate antigens to mobilize antigen presenting cells (APCs) to the vaccination depot
site and enhancement of phagocytic antigen uptake
• The activation of the complement pathway and the induction of chemokines which are soluble proteins that
mediate/control the immune response to infections. This results in the production of antibodies, especially
IgG 1 and IgE. (5-7)
Studies have shown that aluminum-containing vaccines produce a higher and more prolonged antibody response
than that observed with comparable vaccines without adjuvant. This benefit was more apparent during the primary
immunization series compared with the booster doses. As a result of this enhanced immune response, less antigen is
needed per dose, and the number ofrequired doses can be reduced.(5)
Safety Profile
Aluminum adjuvants have a demonstrated safety profile over 60 years of use (2;8) and Merck's aluminum adjuvant
has been used for approximately 25 years. Chapter 21 of the U.S. Code ofFederal Regulations (21 CFR 610.15)
describes the general requirements for adjuvants which are agents that augment specific immune responses to
antigens.(2)
Common adverse events include local reactions such as redness, swelling, and/or tenderness at the injection site.
More serious local reactions such as large areas of swelling, sterile abscesses, subcutaneous nodules, and allergic
responses have been reported rarely.( 4;8) Data from a review of adverse events after the administration of
aluminum-containing vaccines against diphtheria, tetanus, and pertussis show no evidence that aluminum salts cause
serious or long-lasting adverse events.(4) Additional papers have been published regarding aluminum adjuvants in
vaccines. (9;10)
Aluminum is an abundant element found in soil, air, food, and water; therefore, infants are exposed to aluminum in
the environment in addition to exposure through vaccination. For example, breast milk contains approximately
Page 164 of 637

APRlL 10, 2017 PETER DOSHI, PHD. 4
40 mcg of aluminum per liter, and infant formulas contain an average of approximately 225 mcg of aluminum per
liter. The amount of aluminum in vaccines is similar to that in infant formulas.(8)
The US Agency for Toxic Substances and Disease Registry (ASTDR) has determined that the minimum risk level
for human exposure to aluminum from all sources is 2 mg/kg/day. The half-life of elimination of aluminum from
the body is 24 hours. Therefore it has been established that the levels of aluminum that infants are exposed to
through food and vaccines are less than the minimal risk level established by ASTDR.( 4;8)
It has been suggested that aluminum may be associated with dementia and Alzheimer's disease. The Alzheimer's
Association notes that upon continued investigation, there is no evidence to suggest aluminum increases the risk of
dementia. http://www.alz.org/alzheimers disease myths about alzheimers.asp; accessed April, 2017)
General Recommendations
The Global Advisory Committee on Vaccine Safety (GACVS) stated that "From the most recent evidence available,
there is no reason to conclude that a health risk exists as a result of administration of aluminum-containing vaccines,
nor is there any good reason for changing current vaccination practice. The GACVS will continue to review the
evidence that might emerge from ongoing studies."
(http://www.who.int/vaccine safety/committee/topics/aluminium/statement 112002/en/; accessed April, 2017)
In the 'Ingredients of Vaccines Fact Sheet', the Centers for Disease Control and Prevention (CDC) states that
'Common substances found in vaccines include: Aluminum gels or salts of aluminum which are added as adjuvants
to help the vaccine stimulate a better response. Adjuvants help promote an earlier, more potent response, and more
persistent immune response to the vaccine." "To ensure the safety of vaccines, the CDC, the FDA, the National
Institutes of Health (NIH), and other Federal agencies routinely monitor and conduct research to examine any new
evidence that would suggest possible problems with the safety of vaccines." (http://www.cdc.gov/vaccines/vac-
gen/additives.htm; accessed April, 2017)
In the section about vaccine ingredients on 'Common Parental Concerns', the American Academy of Pediatrics
(AAP) notes that adjuvants help increase the body's immune response to a vaccine, and make it possible to use
smaller amounts of antigen and decrease the number of doses needed. In addition, the AAP states that the amount of
aluminum in vaccines is similar to that found in 33 ounces of infant formula. (bttps://www.aap.org/en-us/advocacy-
and-policy/aap-health-initiatives/immunization/Pages/Common- Parental-Concems.aspx; accessed April, 201 7.)
Reference List
(1) GARDASIL®9 Prescribing Information.
(2) Baylor NW, Egan W, Richman P. Aluminum salts in vaccines: US perspective. Vaccine 2002
May;20(Suppl 3):Sl8-S23.
(3) Ruiz W, McClements WL, Jansen KU, Esser MT. Kinetics and isotype profile of antibody responses in
rhesus macaques induced following vaccination with HPV 6, 11, 16, and 18 LI-virus-like particles
formulated with or without Merck aluminum adjuvant. J Immune Based Therapies Vaccines 2005 Apr
20;3(1):2.
(4) NNii. Vaccine Components: Aluminum adjuvants in vaccines. National Network for Immunization
Information (NNii) 2008 Nov 7;1-3.
(5) Eickhoff TC, Myers M. Workshop Summary: Aluminum in Vaccines. Vaccine 2002;20:Sl-S4.
(6) Hunter RL. Overview of vaccine adjuvants: present and future. Vaccine 2002;20:S7-Sl2.
(7) Fraser CK, Diener KR, Brown MP, Hayball JD. Improving vaccines by incorporating immunological
coadjuvants. Expert Rev Vaccines 2007;6(4):559-78.
Page 165 of 637

(8) Offit PA, Jew RK. Addressing Parents' Concerns: Do Vaccines Contain Harmful Preservatives, Adjuvants,
Additives, or Residuals? Pediatrics 2003;112(6):1394-401.
(9) Gherardi R, Eidi H, Crepeaux G, et al. Biopersistence and brain translocation of aluminum adjuvants of
vaccines. Front Neurol 2015;5(6):4.
(10) Shaw C, Li D, Tomljenovic L. Are there negative CNS impacts of aluminum adjuvants used in vaccines
and immunotherapy? Immunotherapy 2014;6(10):1055-71.
Page 166 of 637

APRJL 10, 2017 PETER DOSHI, PHD. 6
HPV DNA IN GARDASIL
Provided below, for your review, is the most current information regarding the presence ofHPV DNA in
GARDASIL.
The production of GARD AS IL does not use live human papillomavirus (HPV); rather, GARD AS IL contains virus
like particles (VLPs), which are protein shells that help to stimulate immunity (antibodies) to HPV.(1)
In general, recombinant DNA technology (using tiny fragments of DNA to generate antibodies) has been used in the
production of several vaccines for decades.(2) Vaccines that are manufactured using recombinant technology,
including GARDASIL, may potentially have detectable fragments of DNA, which are expected residuals from the
manufacturing process. The fragments are present in very small quantities and are not contaminants. There has
been no identified link between the presence of fragments of recombinant DNA and adverse events.
http://www.fda.gov/BiologicsBloodVaccinesN accines/ ApprovedProducts/ucm27 6859 .htin; accessed April, 2017)
A search of the published medical literature identified information pertaining to your inquiry, a selection of which is
provided below.
Lee (3) reported on the results ofHPV DNA testing performed on 16 samples ofGARDASIL from nine countries.
Three unopened vials and 13 unopened syringes were received from physicians in Australia, Bulgaria, France, India,
New Zealand, Poland, Russia, Spain, and the United States. A nested polymerase chain reaction (PCR) method was
used to test for HPV DNA. When the proteinase K-resistant insoluble part of the vaccine was tested, HPV LI gene
fragments were detected, suggesting the HPV DNA fragments were bound to the amorphous aluminum
hydroxyphosphate sulfate (AAHS) adjuvant. All positive amplifications were shown to contain a hypervariable
sequence of the L 1 gene open reading frame of an HPV-11 synthetic construct, a variant of HPV-18 DNA, or a
mixture of the two. No HPV-6 or HPV-16 DNA residues were detected. When the supernatant of the vaccine or the
supernatant of the proteinase K digestate of the insoluble particles of the vaccine were tested, they did not detect any
DNA residues.
On October 21, 2011, the Food and Drug Administration released a statement on this topic. The archived statement
can be found at http://www.fda.gov/B iologicsB loodV accinesN accines/ ApprovedProducts/ucm2 7685 9 .htin
(accessed April, 2017).
Reference List
(1) CDC, Atkinson W, Wolfe S, Hamborsky J. Human Papillomavirus In: Epidemiology and Prevention of
Vaccine-Preventable Diseases. 12th ed. Washington, DC: Public Health Foundation. 2011. p. 139-50.
(2) Centers for Disease Control and Prevention. Hepatitis B. In: Atkinson W, Wolfe S, Hamborsky J, editors.
Epidemiology and Prevention of Vaccine-Preventable Diseases. 12th ed. Washington DC: Public Health
Foundation; 2011.
(3) Lee S. Detection of human papillomavirus (HPV) LI gene DNA possibly bound to particulate aluminum
adjuvant in the HPV vaccine Gardasil. J Inorg Biochem Dec 2012;117:85-92.
Page 167 of 637

Merck& Co.,inc.
U.S.HumanHealth
P.O.Box1000
NorthWales,PA19454-1099
Tel:8006726372
Fax:8006372568
April 10,2017
Peter Doshi, PhD.

Assistant Professor
0MERCK
20 North Pine Street
Baltimore MD 21201
Dear Dr. Doshi:
This is in response to your recent inquiry regarding GARDASIL (human papillomavirus quadrivalent (types 6, 11,
16, and 18) vaccine, recombinant). Your inquiry concerned whether there are any long-term studies with
GARDASIL that evaluate cancer, such as cervical, anal, oropharyngeal, and penile, as a specific outcome.
Enclosed, for your review, is a summary of the most current information regarding the duration of efficacy of
GARDASIL, which includes data relevant to your inquiry.
The following information is provided as a professional service in response to your unsolicited inquiry. It is
intended to provide you with a review of the available scientific literature and/or data that you requested. Merck
does not recommend the use of its products in any manner other than as described in the prescribing information.
Enclosed for your convenience is prescribing information for GARDASIL.
If you require further information or need to report suspected adverse reactions, please contact the Merck National
Service Center at 1-800-672-6372.
Merck does not control nor assume any responsibility for the content of other Internet websites to which we provide
links.
Sincerely,
Richard Gersh, MD
Global Medical Information Physician
Enclosures:
GARDASIL Prescribing Information
Case Number USI 7-013927
Page 168 of 637

DURATION OF IMMUNITY/EFFICACY
The following is a summary of the information pertinent to your inquiry. For detailed information, please refer to
the appropriate sections outlined below.
Summary
• The duration of immunity following a complete schedule of immunization with GARDASIL has not been
established. The peak anti-HPV GMTs for HPV types 6, 11, 16, and 18 in clinical trials occurred at month
7. Anti-HPV GMTs for HPV types 6, 11, 16, and 18 were similar between measurements at month 24 and
month 60 in Study 2 (phase 2 study in 16- through 26-year-old girls and women).(1)
• The minimum anti-HPV titer that confers protective efficacy has not been determined. Because there were
few disease cases in individuals naYve(PCR negative and seronegative) to vaccine HPV types at baseline in
the group that received GARDASIL, it has not been possible to establish minimum anti-HPV 6, anti-HPV
11, anti-HPV 16, and anti-HPV 18 antibody levels that protect against clinical disease caused by HPV 6,
11, 16, and/or 18.(1)
• The protection of GARD ASIL against HPV-related disease continues to be studied over time in
populations who were enrolled in Phase 3 studies.(1)
o Females 16 to 23 years: Data from an ongoing extension of the Females United to Unilaterally
Reduce Endo/Ectocervical Disease (FUTURE) II study showed no cases ofHPV 6/11/16/18-related
cervical intraepithelial neoplasia (CIN) (any grade), adenocarcinoma in situ (AIS), cervical cancer,
vulvar cancer or vaginal cancer up to 8.4 years post-vaccination. Immunogenicity analyses 9 years
postdose 1 showed the percentage who remained seropositive to HPV 6/11/16/18 was 94.4%, 95.5%,
99.1 %, and 60.0% by competitive Luminex immunoassay (cLIA) and 97.6%, 96.4%, 100%, and
90.8% by IgG LIA.(1-4) An analysis at 10 years post-vaccination showed no new cases ofHPV
16/18-related CIN 2 or worse, or HPV 6/11/16/18-related CIN, vulvar cancer, and vaginal cancer.(5)
o Males and Females 9 to 15 years: The final data presented from an extension study demonstrated
no breakthrough cases of cervical/genital disease related to HPV types 6, 11, 16, and 18 observed
among preadolescents and adolescents vaccinated with GARDASIL during 10 years of follow-up.
Ten subjects were identified with persistent infection of 2: 6 month duration with vaccine HPV types.
Two of these 10 subjects had an infection that persisted for 2:12 months. Immunogenicity analyses
showed that seropositivity remained high for HPV types 6, 11, and 16. Lower anti-HPV 18 responses
were seen over time, but no cases of persistent infection due to HPV type 18 were observed. (6)
o Females 24 to 45 years: Long-term follow-up data in a subset of women from FUTURE III showed
no cases ofHPV 6/11/16/18-related CIN or external genital lesions during the extension phase of the
study (4 to 6 years post-vaccination). Immunogenicity data demonstrated no significant reduction in
seropositivity from month 48 (end of base study) to month 72.(7) An analysis at 8 years post-
vaccination showed no cases ofHPV 6/11/16/18-related CIN or condyloma, HPV16/18 related CIN 2
or worse, or HPV 6/11 related condyloma.(8)
o Males 16 to 26 years: Long-term follow-up data in young men showed no cases ofHPV Types 6/11-
related genital warts and no cases ofHPV Types 6/11/16/18-related external genital lesions up to
approximately 9 years post-vaccination. In a subpopulation evaluated for anal disease, a single case
of AINI was observed and no high-grade disease was reported. Seropositivity rates for HPV 6, 11,
16, and 18 remained high in the young male population. (9)
• Immune memory was assessed by administering a fourth dose ofGARDASIL to a subset of patients
originally enrolled and vaccinated with 3 doses in a phase II clinical trial. Among subjects who received
the antigen challenge at month 60, a strong memory (anamnestic) immune response was observed.(10)
Page 169 of 637

• The need for a booster dose of GARDASIL is not known.
Medical Literature
A search of the published medical literature pertaining to GARDASIL identified references that are pertinent to your
inquiry.
Clinical Trials
FUTURE II (Young Women)

FUTURE II was a randomized, double-blind, placebo-controlled phase 3 clinical trial that evaluated the efficacy,
immunogenicity and safety of GARDASIL in women 16 to 26 years of age (11 ). The study emolled 12,167 women
who received GARDASIL (n=6087) or placebo (n=6080) at day 1, month 2, and month 6. An ongoing extension of
FUTURE II (Long-term Follow-up [LFTU] study) is utilizing national healthcare registries in Denmark, Iceland,
Norway, and Sweden to monitor endpoint cases ofHPV 6/11/16/18-related CIN (any grade), AIS, cervical cancer,
vulvar cancer, or vaginal cancer among girls and women 16 through 23 years of age at emollment. For women who
received GARDASIL in FUTURE II, follow-up began after the last study visit (Cohort 1; n=2650); in women who
received placebo, follow-up began after three doses of GARD AS IL were administered subsequent to the completion
of the base study (Cohort 2).(1;2)
An interim analysis of the per-protocol effectiveness population included 1,902 subjects who completed the
vaccination series with GARDASIL within one year, were naYveto the relevant HPV type through 1 month postdose
3, had no protocol violations, and had follow-up data available. The median follow-up from initial vaccination was
6.7 years with a range of2.8 to 8.4 years. No cases ofHPV 6/11/16/18-related CIN (any grade), AIS, cervical
cancer, vulvar cancer, or vaginal cancer were observed over a total of 5,765 person-years at risk.(1 ;2) An additional
analysis showed that no cases ofHPV 6/11/16/18-related disease have been observed in Cohort 2 through year 8 (4).
Analyses of type-specific anti-HPV antibody titers at 9 years postdose 1 (range of 1,178 to 1,331 subjects with
evaluable data across HPV types) showed the percentage who remained seropositive to HPV 6/11/16/18 was 94.4,
95.5, 99.1, and 60.0 by cLIA and 97.6, 96.4, 100, and 90.8 by lgG LIA.(1;3)
In an analysis at 10 years post-vaccination among women in Cohort 1 (n=1281), no new cases ofHPV 16/18-related
CIN 2 or worse, or HPV 6/11/16/18-related CIN, vulvar cancer, and vaginal cancer were observed (5).
Adolescents
A follow-up study is evaluating the long-term immunogenicity, safety, and effectiveness of GARDASIL in
adolescents (12). The study is a continuation of a randomized, placebo-controlled trial that evaluated the safety and
immunogenicity of GARD AS IL in 1781 sexually naYvemale and female adolescents aged 9 to 15 years of age (13).
The primary objective of the long-term study is to evaluate the anti-HPV 6/11/16/18 serological levels at months 72,
96 and 126. The secondary objective is the long-term effectiveness of GARDASIL with regard to the incidence of
HPV 6/11/16/18-related persistent infection and disease (CIN [any grade], AIS, VIN, VaIN, cervical cancer, vulvar
cancer, vaginal cancer, genital lesions) from the initiation of sexual activity or age 16 onwards. The safety objective
is the incidence of serious adverse events and deaths considered vaccine- or procedure-related. The data presented
here are the month 96 (8 year) interim analyses (12).
Of the 1781 subjects in the baseline trial, 1661participated in the long-term follow-up study. Subjects vaccinated in
the base study were the early vaccination group (EVG; n=l 179); those vaccinated with placebo in the base study and
later vaccinated with a three dose regimen of GARDASIL (starting at month 30) were the catch-up vaccination
group (CVG; n=482). The mean age of vaccination with dose one was 12 and 15 years in the EVG and CVG,
respectively. All other baseline characteristics were similar.
A total of 1116 subjects in the EVG had follow-up immunogenicity data. As differences in seropositivity have been
observed between the cLIA and IgG total assays (attributed to the differences in measurement parameters and
sensitivity of each assay), antibodies were measured using both. Immunogenicity analyses 8 years postdose 1 (range
of 436 to 440 subjects with evaluable data across HPV types (1)) showed the percentage who remained seropositive
to HPV 6/11/16/18 was 88.4%, 89.1 %, 96.8% and 64.1 % with cLIA and 94.3%, 89.4%, 99.5%, and 88.8% with
Page 170 of 637

total IgG, respectively. Note the median follow-up time for the CVG was 4.7 years post dose 3 and was not
included in this analysis.
In the EVG, there were no cases ofHPV 6/11/16/18-related persistent infection :::,:12months' duration, CIN, AIS,
VIN, VaIN, cervical cancer, vulvar cancer, vaginal cancer or genital lesions (total of 1,105 person-years at risk (1)).
months' duration; none persisted to 12 months' duration. In the CVG,
There were 4 cases of persistent infection :::,:4
there was one case ofHPV-18 related CIN 1 and 7 cases of persistent infection. Note the median follow-up time to
assess persistent infection and disease was similar in both groups (4.1 and 3.9 years in the EVG and CVG groups,
respectively) as evaluation began when subjects were 16 years of age. Details are provided in the following table.
HPV 6/11/16/18-Related Persistent Infection or Disease

Endpoints EVG CVG
n cases n cases
Females
HPV 6/11/16/18-related persistent infection 240 2 121 6
HPV 16-related 240 2 121 2
HPV 18-related 240 0 121 4
HPV 6/11/16/18-related cervical disease 206 0 107 la
HPV 6/11/16/18-related genital warts or vulvar/vaginal disease 256 0 126 0
Males
HPV 6/11/16/18-related persistent infection 171 2 89 1
HPV 6-related 171 1 89 1
HPV 16-related 171 1 89 0
HPV 6/11/16/18-related disease 173 0 89 0
aHPV 18-related CINI
Among subjects evaluated for persistent infection or disease, the acquisition of new sexual partners was
approximately 1 per year. The incidence of Chlamydia (3%) and gonorrhea (4%) was similar between the EVG and
CVG and between male and female subjects. In the CVG, 4 of the 6 female subjects who developed persistent
infection or disease were also diagnosed with Chlamydia or gonorrhea during the study. The male diagnosed with
HPV 6-related persistent infection was also diagnosed with Chlamydia during the follow-up period.
Three serious adverse events occurred during the long~term follow-up study, one of which was determined by the
investigator to be vaccine-related (cranial nerve VII paralysis of2.7 weeks' duration [CVG, 131 days post dose 3]).
The subject fully recovered.
The authors noted this is the first study to present long-term immunogenicity, effectiveness and safety data for
GARDASIL from a monitored group of pre-adolescents and adolescents, the intended primary target group for
vaccination. The data showed that 3 doses of GARD AS IL provided protection against HPV 6/11/16/18-related
persistent infection and disease, with the only cases of persistent infection :::,:12months and disease (CIN 1) seen in
the CVG (vaccinated 3 years later than the EVG). The acquisition of new sexual partners and the incidence of other
sexually transmitted diseases indicate the emerging risks encountered by this age group. The authors discussed the
need for HPV vaccination, preferably before sexual debut and HPV acquisition, to reduce the risk ofHPV-related
cancers in the population.
The final 10-year data for the adolescent subjects in this study was presented by Das et al (6). A total of 1,245
subjects (821 in the EVG and 424 in the CVG) had visits in the long-term follow-up study. The median follow-up
time was 9.9 years in the EVG and 7.4 years in the CVG. No breakthrough cases of cervical/genital disease related
to HPV types 6, 11, 16, and 18 were observed among preadolescents and adolescents vaccinated with GARDASIL
during 10 years of follow-up. Ten subjects were identified with persistent infection of::=-:
6 month duration with
vaccine HPV types. Two of these 10 subjects had an infection that persisted for ::=-:12
months.
For HPV types 6, 11, and 16, 89%-96% of subjects remained seropositive through 10 years after vaccination. Lower
anti-HPV 18 responses were seen over time, consistent with observations in other studies with GARDASIL, but no
Page 171 of 637

cases of persistent infection due to HPV type 18 were observed. No serious adverse events were reported between 8
and 10 years of follow-up.
FUTURE III (Adult Women)
FUTURE III was a randomized, double-blind, placebo-controlled phase 3 clinical trial that evaluated the efficacy,
immunogenicity and safety ofGARDASIL in women aged 24 to 45 years of age (14). The study enrolled 3,819
women who received GARDASIL (n=1911) or placebo (n=1908) at day 1, month 2 and month 6. An analysis of
long-term follow-up in women who participated in FUTURE III evaluated data collected as of year 6 post-
vaccination (relative to day 1 of the base study) in subjects vaccinated in Columbia (7). Total enrollment in the base
study was 1610 Columbian women (804 randomized to GARDASIL, 806 randomized to placebo); 1360 participated
in the extension. Subjects vaccinated in the base study were the early vaccination group (EVG; n=684); subjects
vaccinated with placebo in the base study and later vaccinated with a three dose regimen of GARDASIL were the
catch-up vaccination group (CVG; n=651). The primary analyses for this follow-up were conducted in the EVG
PPE population as the CVG have not had sufficient follow-up as of this publication.
Results in the EVG population showed there were no cases ofHPV 6/11/16/18-related CIN or genital warts during
the follow-up period (4 to 6 years post-vaccination). Analysis ofnon-HPV 6/11/16/18 CIN or genital warts did not
reveal any evidence ofHPV type replacement. Immunogenicity data using cLIA and total lgG demonstrated no -
significant reduction in seropositivity from month 48 (end of base study) to month 72. Seropositivity at month 72
for each vaccine-related HPV type is provided in the following table.
Anti-HPV Seropositivity at Month 72

EVG Per-Protocol Immuno enicit Po ulation
Assay
Anti-HPV6
cLIA 468 89.1 % (85.9%, 91.8%)
Total I G 468 87.8% (84.5%, 90.6%)
Anti-HPV 11
cLIA 466 92.1% (89.2%, 94.3%)
Total I G 468 84.4% (80.8%, 87.6%)
Anti-HPV 16
cLIA 473 97.3% (95.3%, 98.5%)
Total I G 473 99.8% (98.8%, 100%
Anti-HPV 18
cLIA 530 45.3% (41.0%, 49.6%)
Total I G 530 81.5% (77.9%, 84.7%)
While some declines are seen in seropositivity after the vaccination course, month 72 cLIA seropositivity was
comparable to that observed at month 48. Other studies have noted that the proportion of subjects seropositive to
HPV 18 by cLIA declines over time due to the nature of the assay. As stated above, no cases ofHPV 18-related
disease have been observed in the EVG PPE population, either during the base study or during follow-up to date. In
addition, no serious adverse experiences were reported in the EVG between years 4 and 6.
An additional analysis at 8 years post-vaccination showed no cases ofHPV 6/11/16/18-related CIN or condyloma,
HPVl 6/18 related CIN 2 or worse, or HPV 6/11 related condyloma in the EVG (8).
Young Men
An analysis examined the long-term effectiveness, immunogenicity and safety ofGARDASIL in young men (15).
The study is a continuation of a randomized, double-blind, placebo-controlled trial which evaluated the efficacy and
safety of GARDASIL in 4065 men 16 to 26 years of age (16). As of the cut-off date for this analysis, 1,805 have
participated in the long-term study. Among these subjects, the median follow-up time post dose 3 was 5.8 years in
the early vaccination group (vaccinated in the base study); the range of follow-up was 3.1 to 6.8 years.
This analysis showed there were no cases ofHPV 6/11-related genital warts or HPV 6/11/16/18-related external
genital lesions, AIN or anal cancer up to 6.8 years post vaccination. Approximately 84% or more of subjects in the
early vaccination group remained seropositive for HPV types 6, 11, and 16 and approximately 48% of subjects were
Page 172 of 637

APRlL 10,2017 PETER DOSHI, PHD. 6
seropositive for HPV type 18 through month 72 post dose 1. Two serious adverse experiences have been reported
up to this time; neither adverse event was considered related to the vaccine.
Follow-up data for the young men in this study was presented by Das et al (9). There were 936 subjects in the EVG
followed for a median duration of8.9years and 867 CVG subjects were followed for 4.2 years. In the EVG per-
protocol population, no cases ofHPV Types 6/11-related genital warts and no cases ofHPV Types 6/11/16/18-
related external genital lesions were observed during the extension. In a subpopulation evaluated for anal disease,
no high-grade disease was reported and a single case of AIN 1 was observed (0.3/100 person-years-at-risk in this
extension, compared to 5.8 per 100 person-years-at- risk in the base study). Seropositivity rates for HPV 6, 11, 16,
and 18 remained high in the young male population. There were no vaccine-related serious adverse experiences
reported during this follow-up period.
Immune Memory
To assess immune memory, a fourth dose of GARD AS IL was administered at month 60 to a subset of241
women 16 to 23 years of age who were originally enrolled in a phase II clinical trial and received
GARDASIL or placebo at day 1, month 2, and month 6 (10). HPV antibody responses were measured at
1 and 4 weeks post-dose 4 (subjects who originally received vaccine) or post-dose 1 (subjects who
originally received placebo). Between month 37 and month 60, no breakthrough cases ofHPV-related
disease occurred in the group that received GARDASIL (10 new cases were observed in the placebo group
during the extension period). Among subjects who received the antigen challenge at month 60 and who
were DNA negative to HPV-6/11 /16 or 18 through month 60, a strong memory (anamnestic) immune
response was observed (Figure 1). Anti-HPV levels 1 week post-challenge were approximately 7- to 23-
fold higher than levels observed 1 month post-dose 3 (month 7). Anti-HPV GMTs 1 month post-dose 4
were approximately 10- to 3 8-fold higher than those observed at month 7.
Figure 1: Immune Memory at 60 Months Following a Three-dose Regimen of GARDASIL

Redacted on request by BMJ
Page 173 of 637

This figure was published in Vaccine, Vol. 25, Olsson, Villa, Costa, et al., Induction oflmmune Memory Following Administration of a
Prophylactic Quadrivalent Human Papillomavirus Types 6/11/16/18 LI Virus-Like Particle Vaccine, Pages 4931-4939, Copyright Elsevier
(2007).
Additional Prescribing Information (1)
Long-Term Follow-Up Studies (Section 14.9)
The protection of GARDASIL against HPV-related disease continues to be studied over time in populations
including adolescents (boys and girls) and women who were enrolled in the Phase 3 studies.
Persistence of Effectiveness
An extension of Study 4 used national healthcare registries in Denmark, Iceland, Norway, and Sweden to monitor
endpoint cases ofHPV 6-, 11-, 16-, or 18-related CIN (any grade), AIS, cervical cancer, vulvar cancer, or vaginal
cancer among 2,650 girls and women 16 through 23 years of age at enrollment who were randomized to vaccination
with GARDASIL and consented to be followed in the extension study. An interim analysis of the per-protocol
effectiveness population included 1,902 subjects who completed the GARDASIL vaccination series within one year,
were naive to the relevant HPV type through 1 month postdose 3, had no protocol violations, and had follow-up data
available. The median follow-up from initial vaccination was 6.7 years with a range of2.8 to 8.4 years. No cases of
HPV 6-, 11-, 16-, or 18-related CIN (any grade), AIS, cervical cancer, vulvar cancer, or vaginal cancer were
observed over a total of 5,765 person-years at risk.
An extension of a Phase 3 study (Study 7) in which 614 girls and 565 boys 9 through 15 years of age at enrollment
were randomized to vaccination with GARDASIL actively followed subjects for endpoint cases ofHPV 6-, 11-, 16-,
or 18-related persistent infection, CIN (any grade), AIS, VIN, VaIN, cervical cancer, vulvar cancer, vaginal cancer,
and genital lesions from the initiation of sexual activity or age 16 onwards. An interim analysis of the per-protocol
effectiveness population included 246 girls and 168 boys who completed the GARDASIL vaccination series within
one year, were seronegative to the relevant HPV type at initiation of the vaccination series, and had not initiated
sexual activity prior to receiving the third dose of GARDASIL. The median follow-up, from the first dose of
vaccine, was 7.2 years with a range of0.5 to 8.5 years. No cases of persistent infection ofat least 12 months'
duration and no cases ofHPV 6-, 11-, 16-, or 18-related CIN (any grade), AIS, VIN, VaIN, cervical cancer, vulvar
cancer, vaginal cancer, or genital lesions were observed over a total 1,105 person-years at risk. There were 4 cases
ofHPV 6-, 11-, 16-, or 18-related persistent infection of at least 6 months' duration, including 3 cases related to
HPV 16 and 1 case related to HPV 6, none of which persisted to 12 months' duration.
Persistence of the Immune Response

The interim reports of the two extension studies described above included analyses of type-specific anti-HPV
antibody titers at 9 years postdose 1 for girls and women 16 through 23 years of age at enrollment (range of 1,178 to
1,331 subjects with evaluable data across HPV types) and at 8 years postdose 1 for boys and girls 9 through 15 years
of age at enrollment (range of 436 to 440 subjects with evaluable data across HPV types). Anti-HPV 6, 11, 16, and
18 GMTs as measured by cLIA were decreased compared with corresponding values at earlier time points, but the
proportions of seropositive subjects ranged from 88.4% to 94.4% for anti-HPV 6, from 89.1 % to 95.5% for anti-
HPV 11, from 96.8% to 99.1% for anti-HPV 16, and from 60.0% to 64.1% for anti-HPV 18.
Reference List
(1) GARDASIL® Prescribing Information.
(2) Kjaer S. Long-term effectiveness ofGARDASIL in the Nordic countries. International Papillomavirus
Conference (IPC), November 30 - December 6, 2012, San Juan, Puerto Rico, EP-739.
(3) Nygard M, Saah A, Munk C, Tryggvadottird L, Enerly E, HortlundfM, et al. Evaluation of the Long-Term
Anti-Human Papillomavirus 6 (HPV6), 11, 16, and 18 Immune Responses Generated by the Quadrivalent
HPV Vaccine. Clin Vaccine Immunol 2015 Aug;22(8):943-8.
Page 174 of 637

a! • f ";
APRJL 10, 2017 PETER DOSHI, PHD. 8
(4) Nygard M. Long-term imnmnogenicity, safety, and effectiveness ofGARDASIL in the Nordic countries.
ESPID, May 28-June 1, 2013. Milan, Italy. Abstract 1249.
(5) Kjaer S, Nygard M, Dillner J, Munk C, et al. Long-term effectiveness and safety ofGardasil in the Nordic
Countries. EUROGIN Proceedings Feb 4-7 2015. European Research Organization on Genital Infection
and Neoplasia 2015 International Multidisciplinary Congress, Seville, Spain. p.171 Abstract OC 6-1 2015.
(6) Das R, Saah A, Iversen 0. Effectiveness, immunogenicity, and safety ofGardasil in pre-adolescents and
adolescents - 10 years of follow-up. Jun 15-18, 2016 - EUROGIN 2016 European Research Organization
on Genital Infection and Neoplasia, Salzburg, Austria. Abstract OC 13-03.
(7) Luna J, Plata M, Gonzalez M, Correa A, Maldonado I, et al. Long-Term Follow-up Observation of the
Safety, Immunogenicity, and Effectiveness ofGardasil in Adult Women. PLoS ONE 2013;8(12):e8341
doi:10.1371/journal.pone.008343 l[Epub ahead of print].
(8) Das R, Plata M, Gonzalez M, Correa A, Maldonado I, et al. Long-term effectiveness of Gardasil among
adult women in Colombia. EUROGIN 2015 Proceedings Feb 4-7 2015 European Research Organization on
Genital Infection and Neoplasia 2015 International Multidisciplinary Congress, Seville, Spain. p.163
Abstract OC4-9 2015.
(9) Das R. A long-term effectiveness, immunogenicity, and safety study of Gardasil (human papillomavirus
[types 6,11,16,18] recombinant vaccine) in young men (V501-020).Jun 15-18, 2016 - EUROGIN 2016
European Research Organization on Genital Infection and Neoplasia, Salzburg, Austria. Abstract OC 03-
03.
(10) Olsson SE, Villa LL, Costa RLR, Petta CA, Andrade RP, Mahn C, et al. Induction of immune memory
following administration of a prophylactic quadrivalent human papillomavirus (HPV) types 6/11/16/18 LI
virus-like particle (VLP) vaccine. Vaccine 2007 Jun 21;25(26):4931-9.
(11) Koutsky LAT, Villa LL, Perez G, Kjaer SK, Paavonen J, Lehtinen M, et al. Quadrivalent Vaccine against
Human Papillomavirus to Prevent High-Grade Cervical Lesions. N Engl J Med 2007 May
10;356(19): 1915-27.
(12) Ferris D, Samakoses R, Block S, Lazcano-Ponce E, et al. Long-term Study of a Quadrivalent Human
Papillomavirus Vaccine. Pediatrics 2014;134(3):e657-e665.
(13) Reisinger KS, Block SL, Lazcano-Ponce E, Samakoses R, Esser MT, Erick J, et al. Safety and Persistent
Immunogenicity ofa Quadrivalent Human Papillomavirus Types 6, 11, 16, 18 LI Virus-Like Particle
Vaccine in Preadolescents and Adolescents. Pediatr Infect Dis J 2007 Mar;26(3):201-9.
(14) Munoz N, Manalastas R, Pitisuttithum P, Tresukosol D. Safety, immunogenicity, and efficacy of

quadrivalent human papillomavirus (types 6, 11, 16, 18) recombinant vaccine in women aged 24-45 years:
A randomized, double-blind trial. Lancet 2009 Jun 6;373(9679):1949-57.
(15) Goldstone S. Long-term effectiveness, immunogenicity and safety of quadrivalent HPV vaccine in men.
International Papillomavirus Conference 2014, August 21-25, Seattle, WA. Abstract PH.PDOl.03.
(16) Giuliano A, Palefsky J, Goldstone S, Moreira E, et al. Efficacy ofQuadrivalent HPV Vaccine against HPV
Infection and Disease in Males. NEJM 2011;364(5):401-11.
Page 175 of 637

HIGHLIGHTS OF PRESCRIBING INFORMATION • Not all vulvar, vaginal, and anal cancers are caused by HPV, and
These highlights do not include all the information needed to use GARDASIL protects only against those vulvar, vaginal, and anal
GARDASIL safely and effectively. See full prescribing information cancers caused by HPV 16 and 18. (1.3)
for GARDASIL. • GARDASIL does not protect against genital diseases not caused
by HPV. (1.3)
GARDASIL® • Vaccination with GARDASIL may not result in protection in all
[Human Papillomavirus Quadrivalent (Types 6, 11, 16, and 18) vaccine recipients. (1.3)
Vaccine, Recombinant] • GARDASIL has not been demonstrated to prevent HPV-related
Suspension for intramuscular injection CIN 2/3 or worse in women older than 26 years of age. (14.7)
Initial U.S. Approval: 2006
------------------- DOSAGE AND ADMINISTRATION---------------------
--------------------------1 NDIGATIO NS AND USAGE ----------------------- 0.5-ml suspension for intramuscular injection at the following
GARDASIL is a vaccine indicated in girls and women 9 through 26 schedule: 0, 2 months, 6 months. (2.1)
years of age for the prevention of the following diseases caused by
------------- DOSAGE FORMS AND STRENGTHS-----------~-------
Human Papillomavirus (HPV) types included in the vaccine:
• 0.5-ml suspension for injection as a single-dose vial and prefilled
• Cervical, vulvar, vaginal, and anal cancer caused by HPV types 16
syringe. (3, 11)
and 18 (1.1)
• Genital warts (condyloma acuminata) caused by HPV types 6 and -------------------------------CONTRA! NDICATIO NS -----------------------
11 (1.1) • Hypersensitivity, including severe allergic reactions to yeast (a
And the following precancerous or dysplastic lesions caused by HPV vaccine component), or after a previous dose of GARDASIL. (4,
types 6, 11, 16, and 18: 11)
• Cervical intraepithelial neoplasia (CIN) grade 2/3 and Cervical
adenocarcinoma in situ (AIS) (1.1) ------------------------WARN! NGS AND PRECAUTIONS--------------------
• Cervical intraepithelial neoplasia (CIN) grade 1 (1.1) • Because vaccinees may develop syncope, sometimes resulting in
falling with injury, observation for 15 minutes after administration is
• Vulvar intraepithelial neoplasia (VIN) grade 2 and grade 3 (1.1)
recommended. Syncope, sometimes associated with tonic-clonic
• Vaginal intraepithelial neoplasia (ValN) grade 2 and grade 3 (1.1) movements and other seizure-like activity, has been reported
• Anal intraepithelial neoplasia (AIN) grades 1, 2, and 3 (1.1) following vaccination with GARDASIL. When syncope is
associated with tonic-clonic movements, the activity is usually
GARDASIL is indicated in boys and men 9 through 26 years of age for transient and typically responds to restoring cerebral perfusion by
the prevention of the following diseases caused by HPV types included maintaining a supine or Trendelenburg position. (5.1)
in the vaccine:
• Anal cancer caused by HPV types 16 and 18 (1.2) -------------------- ADVERSE REACTIONS------------------------
• Genital warts (condyloma acuminata) caused by HPV types 6 and The most common adverse reaction was headache. Common adverse
11 (1.2) reactions (frequency of at least 1.0% and greater than AAHS control or
And the following precancerous or dysplastic lesions caused by HPV saline placebo) are fever, nausea, dizziness; and injection-site pain,
types 6, 11, 16, and 18: swelling, erythema, pruritus, and bruising. (6.1)
• Anal intraepithelial neoplasia (AIN) grades 1, 2, and 3. (1.2)
To report SUSPECTED ADVERSE REACTIONS, contact Merck
Limitations of GARDASIL Use and Effectiveness: Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc., at 1-877-
• GARDASIL does not eliminate the necessity for women to 888-4231 orVAERS at 1-800-822-7967 orwww.vaers.hhs.gov.
continue to undergo recommended cervical cancer screening. (1.3, -------------------------DRUG INTERACTIONS-------------------------
17) GARDASIL may be administered concomitantly with RECOMBIVAX
• Recipients of GARDASIL should not discontinue anal cancer HB® (7.1) or with Menactra and Adacel. (7.2)
screening if it has been recommended by a health care provider.
(1.3, 17) -------------------- USE IN SPECIFIC POPULATIONS---------------------
• GARDASIL has not been demonstrated to provide protection Safety and effectiveness of GARDASIL have not been established in
against disease from vaccine and non-vaccine HPV types to which the following populations:
a person has previously been exposed through sexual activity. • Pregnant women. Women who receive GARDASIL during
(1.3, 14.4, 14.5) pregnancy are encouraged to contact Merck Sharp & Dahme
• GARDASIL is not intended to be used for treatment of active Corp., a subsidiary of Merck & Co., Inc., at 1-877-888-4231. (8.1)
external genital lesions; cervical, vulvar, vaginal, and anal cancers; • Children below the age of 9 years. (8.4)
CIN; VIN; ValN, or AIN. (1.3) • lmmunocompromised individuals. Response to GARDASIL may
• GARDASIL has not been demonstrated to protect against be diminished. (8.6)
diseases due to HPV types not contained in the vaccine. (1.3,
14.4, 14.5) See 17 for PATIENT COUNSELING INFORMATION and
FDA-approved patient labeling.
Revised: 04/2015
FULL PRESCRIBING INFORMATION: CONTENTS* 7 DRUG INTERACTIONS

7.1 Use with RECOMBIVAX HB
1 INDICATIONS AND USAGE 7.2 Use with Menactra and Adacel
1.1 Girls and Women 7.3 Use with Hormonal Contraceptives
1.2 Boys and Men 7.4 Use with Systemic lmmunosuppressive Medications
1.3 Limitations of GARDASIL Use and Effectiveness 8 USE IN SPECIFIC POPULATIONS
2 DOSAGE AND ADMINISTRATION 8.1 Pregnancy
2.1 Dosage 8.3 Nursing Mothers
2.2 Method of Administration 8.4 Pediatric Use
3 DOSAGE FORMS AND STRENGTHS 8.5 Geriatric Use
4 CONTRAINDICATIONS 8.6 lmmunocompromised Individuals
5 WARNINGS AND PRECAUTIONS 10 OVERDOSAGE
5.1 Syncope 11 DESCRIPTION
5.2 Managing Allergic Reactions 12 CLINICAL PHARMACOLOGY
6 ADVERSE REACTIONS 12.1 Mechanism of Action
6.1 Clinical Trials Experience 13 NONCLINICAL TOXICOLOGY
6.2 Postmarketing Experience 13.1 Carcinogenesis, Mutagenesis, Impairment of Fertility
Page 176 of 637

14 CLINICAL STUDIES 14.8 lmmunogenicity
14.1 Prophylactic Efficacy - HPV Types 6, 11, 16, and 18 in Girls 14.9 Long-Term Follow-Up Studies
and Women 16 through 26 Years of Age 14.10Studies with RECOMBIVAX HB [hepatitis B vaccine
14.2 Prophylactic Efficacy- HPV Types 6, 11, 16, and 18 in Boys (recombinant)]
and Men 16 through 26 Years of Age 14.11 Studies with Menactra [Meningococcal (Groups A, C, Y and
14.3 Prophylactic Efficacy -Anal Disease Caused by HPV Types W-135) Polysaccharide Diphtheria Toxoid Conjugate
6, 11, 16, and 18 in Boys and Men 16 through 26 Years of Vaccine] and Adacel [Tetanus Toxoid, Reduced Diphtheria
Age in the MSM Sub-study Toxoid and Acellular Pertussis Vaccine Adsorbed (Tdap)]
14.4 Population Impact in Girls and Women 16 through 26 Years 16 HOW SUPPLIED/STORAGE AND HANDLING
of Age 17 PATIENT COUNSELING INFORMATION
14.5 Population Impact in Boys and Men 16 through 26 Years of
Age *Sections or subsections omitted from the full prescribing information
14.6 Overall Population Impact are not listed.
14.7 Studies in Women 27 through 45 Years of Age
FULL PRESCRIBING INFORMATION
1 INDICATIONS AND USAGE
1.1 Girls and Women

GARDASIL® is a vaccine indicated in girls and women 9 through 26 years of age for the prevention of
the following diseases caused by Human Papillomavirus (HPV) types included in the vaccine:
• Cervical, vulvar, vaginal, and anal cancer caused by HPV types 16 and 18
• Genital warts (condyloma acuminata) caused by HPV types 6 and 11
And the following precancerous or dysplastic lesions caused by HPV types 6, 11, 16, and 18:
• Cervical intraepithelial neoplasia (CIN) grade 2/3 and Cervical adenocarcinoma in situ (AIS)
• Cervical intraepithelial neoplasia (CIN) grade 1
• Vulvar intraepithelial neoplasia (VIN) grade 2 and grade 3
• Vaginal intraepithelial neoplasia (ValN) grade 2 and grade 3
• Anal intraepithelial neoplasia (AIN) grades 1, 2, and 3
1.2 Boys and Men
GARDASIL is indicated in boys and men 9 through 26 years of age for the prevention of the following
diseases caused by HPV types included in the vaccine:
• Anal cancer caused by HPV types 16 and 18
• Genital warts (condyloma acuminata) caused by HPV types 6 and 11
And the following precancerous or dysplastic lesions caused by HPV types 6, 11, 16, and 18:
• Anal intraepithelial neoplasia (AIN) grades 1, 2, and 3
1.3 Limitations of GARDASIL Use and Effectiveness
The health care provider should inform the patient, parent, or guardian that vaccination does not
eliminate the necessity for women to continue to undergo recommended cervical cancer screening.
Women who receive GARDASIL should continue to undergo cervical cancer screening per standard of
care. [See Patient Counseling Information (17).J
Recipients of GARDASIL should not discontinue anal cancer screening if it has been recommended
by a health care provider. [See Patient Counseling Information (17).J
GARDASIL has not been demonstrated to provide protection against disease from vaccine and non-
vaccine HPV types to which a person has previously been exposed through sexual activity. [See Clinical
Studies (14.4, 14.5).]
GARDASIL is not intended to be used for treatment of active external genital lesions; cervical, vulvar,
vaginal, and anal cancers; CIN; VIN; ValN; or AIN.
GARDASIL has not been demonstrated to protect against diseases due to HPV types not contained in
the vaccine. [See Clinical Studies (14.4, 14.5).]
Not all vulvar, vaginal, and anal cancers are caused by HPV, and GARDASIL protects only against
those vulvar, vaginal, and anal cancers caused by HPV 16 and 18.
GARDASIL does not protect against genital diseases not caused by HPV.
Vaccination with GARDASIL may not result in protection in all vaccine recipients.
GARDASIL has not been demonstrated to prevent HPV-related CIN 2/3 or worse in women older than
26 years of age. [See Clinical Studies (14.7).]
Page 177 of 637

2 DOSAGE AND ADMINISTRATION
2.1 Dosage
GARDASIL should be administered intramuscularly as a 0.5-ml dose at the following schedule: 0, 2
months, 6 months. [See Clinical Studies (14.8).]
2.2 Method of Administration
For intramuscular use only.
Shake well before use. Thorough agitation immediately before administration is necessary to maintain
suspension of the vaccine. GARDASIL should not be diluted or mixed with other vaccines. After thorough
agitation, GARDASIL is a white, cloudy liquid. Parenteral drug products should be inspected visually for
particulate matter and discoloration prior to administration. Do not use the product if particulates are
present or if it appears discolored.
· GARDASIL should be administered intramuscularly in the deltoid region of the upper arm or in the
higher anterolateral area of the thigh.
Syncope has been reported following vaccination with GARDASIL and may result in falling with injury;
observation for 15 minutes after administration is recommended. [See Warnings and Precautions (5.1).]
Single-Dose Vial Use
Withdraw the 0.5-ml dose of vaccine from the single-dose vial using a sterile needle and syringe and
use promptly.
Preti/led Syringe Use
This package does not contain a needle. Shake well before use. Attach the needle by twisting in a
clockwise direction until the needle fits securely on the syringe. Administer the entire dose as per
standard protocol.
3 DOSAGE FORMS AND STRENGTHS
GARDASIL is a suspension for intramuscular administration available in 0.5-ml single dose vials and
prefilled syringes. See Description (11) for the complete listing of ingredients.
4 CONTRAINDICATIONS
Hypersensitivity, including severe allergic reactions to yeast (a vaccine component), or after a
previous dose of GARDASIL. [See Description (11).J
5 WARNINGS AND PRECAUTIONS
5.1 Syncope
Because vaccinees may develop syncope, sometimes resulting in falling with injury, observation for
15 minutes after administration is recommended. Syncope, sometimes associated with tonic-clonic
movements and other seizure-like activity, has been reported following vaccination with GARDASIL.
When syncope is associated with tonic-clonic movements, the activity is usually transient and typically
responds to restoring cerebral perfusion by maintaining a supine or Trendelenburg position.
5.2 Managing Allergic Reactions
Appropriate medical treatment and supervision must be readily available in case of anaphylactic
reactions following the administration of GARDASIL.
6 ADVERSE REACTIONS
Overall Summary of Adverse Reactions
Headache, fever, nausea, and dizziness; and local injection site reactions (pain, swelling, erythema,
pruritus, and bruising) occurred after administration with GARDASIL.
Syncope, sometimes associated with tonic-clonic movements and other seizure-like activity, has been
reported following vaccination with GARDASIL and may result in falling with injury; observation for
15 minutes after administration is recommended. [See Warnings and Precautions (5.1).J
Anaphylaxis has been reported following vaccination with GARDASIL.
Page 178 of 637

6.1 Clinical Trials Experience
Because clinical trials are conducted under widely varying conditions, adverse reaction rates observed
in the clinical trials of a vaccine cannot be directly compared to rates in the clinical trials of another
vaccine and may not reflect the rates observed in practice.
Studies in Girls and Women (9 Through 45 Years of Age) and Boys and Men (9 Through 26 Years of
Age)
In 7 clinical trials (5 Amorphous Aluminum Hydroxyphosphate Sulfate [AAHS]-controlled, 1 saline
placebo-controlled, and 1 uncontrolled), 18,083 individuals were administered GARDASIL or AAHS
control or saline placebo on the day of enrollment, and approximately 2 and 6 months thereafter, and
safety was evaluated using vaccination report cards (VRC)-aided surveillance for 14 days after each
injection of GARDASIL or AAHS control or saline placebo in these individuals. The individuals who were
monitored using VRC-aided surveillance included 10,088 individuals 9 through 45 years of age at
enrollment who received GARDASIL and 7,995 individuals who received AAHS control or saline placebo.
Few individuals (0.2%) discontinued due to adverse reactions. The race distribution of the 9- through 26-
year-old girls and women in the safety population was as follows: 62.3% White; 17.6% Hispanic (Black
and White); 6.8% Asian; 6.7% Other; 6.4% Black; and 0.3% American Indian. The race distribution of the
24- through 45-year-old women in the safety population of Study 6 was as follows: 20.6% White; 43.2%
Hispanic (Black and White); 0.2% Other; 4.8% Black; 31.2% Asian; and 0.1% American Indian. The race
distribution of the 9- through 26-year-old boys and men in the safety population was as follows: 42.0%
White; 19.7% Hispanic (Black and White); 11.0% Asian; 11.2% Other; 15.9% Black; and 0.1% American
Indian.
Common Injection-Site Adverse Reactions in Girls and Women 9 Through 26 Years of Age
The injection site adverse reactions that were observed among recipients of GARDASIL at a
frequency of at least 1.0% and also at a greater frequency than that observed among AAHS control or
saline placebo recipients are shown in Table 1.
Table 1: Injection-Site Adverse Reactions in Girls and Women 9 Through 26 Years of Age*
Saline
GARDASIL AAHS Controlt Placebo
Adverse Reaction (N = 5088) (N = 3470) (N = 320)
(1 to 5 Days Postvaccination) % % %
Injection Site
Pain 83.9 75.4 48.6
Swelling 25.4 15.8 7.3
Erythema 24. 7 18.4 12.1
Pruritus 3.2 2.8 0.6
Bruising 2.8 3.2 1.6
*The injection-site adverse reactions that were observed among recipients of GARDASIL were at a frequency
of at least 1.0% and also at a greater frequency than that observed among AAHS control or saline placebo
recipients.
tAAHS Control = Amorphous Aluminum Hydroxyphosphate Sulfate
Common Injection-Site Adverse Reactions in Boys and Men 9 Through 26 Years of Age
The injection site adverse reactions that were observed among recipients of GARDASIL at a
frequency of at least 1.0% and also at a greater frequency than that observed among AAHS control or
saline placebo recipients are shown in Table 2.
Table 2: Injection-Site Adverse Reactions in Boys and Men 9 Through 26 Years of Age*
Saline
GARDASIL AAHS Controlt Placebo
Adverse Reaction (N = 3093) (N = 2029) (N = 274)
(1 to 5 Days Postvaccination) % % %
Injection Site
Pain 61.4 50.8 41.6
Erythema 16.7 14.1 14.5
Swelling 13.9 9.6 8.2
Hematoma 1.0 0.3 3.3
Page 179 of 637

*The injection-site adverse reactions that were observed among recipients of GARDASIL were at a frequency
of at least 1.0% and also at a greater frequency than that observed among AAHS control or saline placebo
recipients.
tAAHS Control = Amorphous Aluminum Hydroxyphosphate Sulfate
Evaluation of Injection-Site Adverse Reactions by Dose in Girls and Women 9 Through 26 Years of Age
An analysis of injection-site adverse reactions in girls and women by dose is shown in Table 3. Of
those girls and women who reported an injection-site reaction, 94.3% judged their injection-site adverse
reaction to be mild or moderate in intensity.
Table 3: Postdose Evaluation of Injection-Site Adverse Reactions in Girls and Women 9 Through 26 Years of Age (1 to 5
Days Postvaccination)
GARDASIL AAHS Control* Saline Placebo
(% occurrence) (% occurrence) (% occurrence)
Post-
Post- Post- Post- Post- Post- Post- Post- Post-
dose
Adverse dose dose dose dose dose dose dose dose
1
Reaction Nt= 2 3 1 2 3 1 2 3
N =4924 N = 4818 N = 3410 N = 3351 N = 3295 N = 315 N = 301 N = 300
5011
Pain 63.4 60.7 62.7 57.0 47.8 49.6 33.7 20.3 27.3
Mild/Moderate 62.5 59.7 61.2 56.6 47.3 48.9 33.3 20.3 27.0
Severe 0.9 1.0 1.5 0.4 0.5 0.6 0.3 0.0 0.3
Swelling• 10.2 12.8 15.1 8.2 7.5 7.6 4.4 3.0 3.3
Mild/Moderate 9.6 11.9 14.2 8.1 7.2 7.3 4.4 3.0 3.3
Severe 0.6 0.8 0.9 0.2 0.2 0.2 0.0 0.0 0.0
Erythema• 9.2 12.1 14.7 9.8 8.4 8.9 7.3 5.3 5.7
Mild/Moderate 9.0 11.7 14.3 9.5 8.4 8.8 7.3 5.3 5.7
Severe 0.2 0.3 0.4 0.3 0.1 0.1 0.0 0.0 0.0
*AAHS Control= Amorphous Aluminum Hydroxyphosphate Sulfate
tN = Number of individuals with follow-up
t1ntensity of swelling and erythema was measured by size (inches): Mild = Oto s1; Moderate= >1 to sZ; Severe= >2.
Evaluation of Injection-Site Adverse Reactions by Dose in Boys and Men 9 Through 26 Years of Age
An analysis of injection-site adverse reactions in boys and men by dose is shown in Table 4. Of those
boys and men who reported an injection-site reaction, 96.4% judged their injection-site adverse reaction
to be mild or moderate in intensity.
Table 4: Postdose Evaluation of Injection-Site Adverse Reactions in Boys and Men 9 Through 26 Years of Age (1 to 5 Days
Postvaccination)
GARDASIL AAHS Control* Saline Placebo
(% occurrence) (% occurrence) % occurrence)
Post-
Post- Post- Post- Post- Post- Post- Post- Post-
dose
Adverse dose dose dose dose dose dose dose dose
1
Reaction Nt= 2 3 1 2 3 1 2 3
N = 2898 N = 2826 N = 1950 N = 1854 N = 1799 N =269 N =263 N = 259
3003
Pain 44.7 36.9 34.4 38.4 28.2 25.8 27.5 20.5 16.2
Mild/Moderate 44.5 36.4 34.1 37.9 28.2 25.5 27.5 20.2 16.2
Severe 0.2 0.5 0.3 0.4 0.1 0.3 0.0 0.4 0.0
Swelling• 5.6 6.6 7.7 5.6 4.5 4.1 4.8 1.5 3.5
Mild/Moderate 5.3 6.2 7.1 5.4 4.5 4.0 4.8 1.5 3.1
Severe 0.2 0.3 0.5 0.2 0.0 0.1 0.0 0.0 0.4
Erythema• 7.2 8.0 8.7 8.3 6.3 5.7 7.1 5.7 5.0
Mild/Moderate 6.8 7.7 8.3 8.0 6.2 5.6 7.1 5.7 5.0
Severe 0.3 0.2 0.3 0.2 0.1 0.1 0.0 0.0 0.0
*AAHS Control = Amorphous Aluminum Hydroxyphosphate Sulfate
t1ntensity of swelling and erythema was measured by size (inches): Mild= Oto s1; Moderate= >1 to s2; Severe= >2.
Common Systemic Adverse Reactions in Girls and Women 9 Through 26 Years of Age
Headache was the most commonly reported systemic adverse reaction in both treatment groups
(GARDASIL = 28.2% and AAHS control or saline placebo = 28.4% ). Fever was the next most commonly
reported systemic adverse reaction in both treatment groups (GARDASIL = 13.0% and AAHS control or
saline placebo= 11.2%).
Page 180 of 637

Adverse reactions that were observed among recipients of GARDASIL, at a frequency of greater than
or equal to 1.0% where the incidence in the GARDASIL group was greater than or equal to the incidence
in the AAHS control or saline placebo group, are shown in Table 5.
Table 5: Common Systemic Adverse Reactions in Girls and Women 9 Through 26 Years of Age
(GARDASIL .:Control)*
GARDASIL AAHS Control+ or Saline

Adverse Reactions (N = 5088) Placebo
(1 to 15 Days Postvaccination) % (N = 3790)
%
Pyrexia 13.0 11.2
Nausea 6.7 6.5
Dizziness 4.0 3.7
Diarrhea 3.6 3.5
Vomiting 2.4 1.9
Cough 2.0 1.5
Toothache 1.5 1.4
Upper respiratory tract infection 1.5 1.5
Malaise 1.4 1.2
Arthralgia 1.2 0.9
Insomnia 1.2 0.9
Nasal congestion 1.1 0.9
*The adverse reactions in this table are those that were observed among recipients of GARDASIL at a
frequency of at least 1.0% and greater than or equal to those observed among AAHS control or saline
placebo recipients.
tAAHS Control= Amorphous Aluminum Hydroxyphosphate Sulfate
Common Systemic Adverse Reactions in Boys and Men 9 Through 26 Years of Age
Headache was the most commonly reported systemic adverse reaction in both treatment groups
(GARDASIL = 12.3% and AAHS control or saline placebo= 11.2%). Fever was the next most commonly
reported systemic adverse reaction in both treatment groups (GARDASIL = 8.3% and AAHS control or
saline placebo= 6.5%).
Adverse reactions that were observed among recipients of GARDASIL, at a frequency of greater than
or equal to 1.0% where the incidence in the group that received GARDASIL was greater than or equal to
the incidence in the AAHS control or saline placebo group, are shown in Table 6.
Table 6: Common Systemic Adverse Reactions in Boys and Men 9 Through 26 Years of Age
(GARDASIL .:Control)*
GARDASIL AAHS Control+ or Saline

Adverse Reactions (N = 3093) Placebo
(1 to 15 Days Postvaccination) % (N = 2303)
%
Headache 12.3 11.2
Pyrexia 8.3 6.5
Oropharyngeal pain 2.8 2.1
Diarrhea 2.7 2.2
Nasopharyngitis 2.6 2.6
Nausea 2.0 1.0
Upper respiratory tract infection 1.5 1.0
Abdominal pain upper 1.4 1.4
Myalgia 1.3 0.7
Dizziness 1.2 0.9
Vomiting 1.0 0.8
*The adverse reactions in this table are those that were observed among recipients of GARDASIL at a
frequency of at least 1.0% and greater than or equal to those observed among AAHS control or saline
placebo recipients.
tAAHS Control= Amorphous Aluminum Hydroxyphosphate Sulfate
Evaluation of Fever by Dose in Girls and Women 9 Through 26 Years of Age

An analysis of fever in girls and women by dose is shown in Table 7.
Page 181 of 637

Table 7: Postdose Evaluation of Fever in Girls and Women 9 Through 26 Years of Age
1 to 5 Da s Postvaccination
GARDASIL AAHS Control* or Saline Placebo
(% occurrence) (% occurrence)
Temperature Postdose 1 Postdose 2 Postdose 3 Postdose 1 Postdose 2 Post
°F Nt = 4945 N = 4804 N = 4671 N = 3681 N = 3564 N=
2100 to <102 3.7 4.1 4.4 3.1 3.8
2102 0.3 0.5 0.5 0.2 0.4
Evaluation of Fever by Dose in Boys and Men 9 Through 26 Years of Age

An analysis of fever in boys and men by dose is shown in Table 8.
Table 8: Postdose Evaluation of Fever in Boys and Men 9 Through 26 Years of Age
(1 to 5 Davs Postvaccinationl
GARDASIL AAHS Control* or Saline Placebo
(% occurrence (% occurrence)
Temperature Postdose 1 Postdose 2 . Postdose 3 Postdose 1 Postdose 2 Postdose 3
(OF) Nt= 2972 N = 2849 N = 2792 N = 2194 N = 2079 N = 2046
>100 to <102 2.4 2.5 2.3 2.1 2.2 1.6
2102 0.6 0.5 0.5 0.5 0.3 0.3
Serious Adverse Reactions in the Entire Study Population

=
Across the clinical studies, 258 individuals (GARDASIL N 128 or 0.8%; placebo N 130 or 1.0%) =
= =
out of 29,323 (GARDASIL N 15,706; AAHS control N 13,023; or saline placebo N 594) individuals =
(9- through 45-year-old girls and women; and 9- through 26-year-old boys and men) reported a serious
systemic adverse reaction.
Of the entire study population (29,323 individuals), 0.04% of the reported serious systemic adverse
reactions were judged to be vaccine related by the study investigator. The most frequently (frequency of 4
cases or greater with either GARDASIL, AAHS control, saline placebo, or the total of all three) reported
serious systemic adverse reactions, regardless of causality, were:
Headache [0.02% GARDASIL (3 cases) vs. 0.02% AAHS control (2 cases)],
Gastroenteritis [0.02% GARDASIL (3 cases) vs. 0.02% AAHS control (2 cases)],
Appendicitis [0.03% GARDASIL (5 cases) vs. 0.01% AAHS control (1 case)],
Pelvic inflammatory disease [0.02% GARDASIL (3 cases) vs. 0.03% AAHS control (4 cases)],
Urinary tract infection [0.01% GARDASIL (2 cases) vs. 0.02% AAHS control (2 cases)],
Pneumonia [0.01% GARDASIL (2 cases) vs. 0.02% AAHS control (2 cases)],
Pyelonephritis [0.01% GARDASIL (2 cases) vs. 0.02% AAHS control (3 cases)],
Pulmonary embolism [0.01% GARDASIL (2 cases) vs. 0.02% AAHS control (2 cases)].
One case (0.006% GARDASIL; 0.0% AAHS control or saline placebo) of bronchospasm; and 2 cases
(0.01% GARDASIL; 0.0% AAHS control or saline placebo) of asthma were reported as serious systemic
adverse reactions that occurred following any vaccination visit.
In addition, there was 1 individual in the clinical trials, in the group that received GARDASIL, who
reported two injection-site serious adverse reactions (injection-site pain and injection-site joint movement
impairment).
Deaths in the Entire Study Population
Across the clinical studies, 40 deaths (GARDASIL N = 21 or 0.1%; placebo N = 19 or 0.1%) were
reported in 29,323 (GARDASIL N =
15,706; AAHS control N =
13,023, saline placebo N 594) =
individuals (9- through 45-year-old girls and women; and 9- through 26-year-old boys and men). The
events reported were consistent with events expected in healthy adolescent and adult populations. The
most common cause of death was motor vehicle accident (5 individuals who received GARDASIL and 4
individuals who received AAHS control), followed by drug overdose/suicide (2 individuals who received
GARDASIL and 6 individuals who received AAHS control), gunshot wound (1 individual who received
GARDASIL and 3 individuals who received AAHS control), and pulmonary embolus/deep vein thrombosis
(1 individual who received GARDASIL and 1 individual who received AAHS control). In addition, there
Page 182 of 637

were 2 cases of sepsis, 1 case of pancreatic cancer, 1 case of arrhythmia, 1 case of pulmonary
tuberculosis, 1 case of hyperthyroidism, 1 case of post-operative pulmonary embolism and acute renal
failure, 1 case of traumatic brain injury/cardiac arrest, 1 case of systemic lupus erythematosus, 1 case of
cerebrovascular accident, 1 case of breast cancer, and 1 case of nasopharyngeal cancer in the group
that received GARDASIL; 1 case of asphyxia, 1 case of acute lymphocytic leukemia, 1 case of chemical
poisoning, and 1 case of myocardial ischemia in the AAHS control group; and 1 case of medulloblastoma
in the saline placebo group.
Systemic Autoimmune Disorders in Girls and Women 9 Through 26 Years of Age
In the clinical studies, 9- through 26-year-old girls and women were evaluated for new medical
conditions that occurred over the course of follow-up. New medical conditions potentially indicative of a
systemic autoimmune disorder seen in the group that received GARDASIL or AAHS control or saline
placebo are shown in Table 9. This population includes all girls and women who received at least one
dose of GARDASIL or AAHS control or saline placebo, and had safety data available.
Table 9: Summary of Girls and Women 9 Through 26 Years of Age Who Reported an Incident Condition
Potentially Indicative of a Systemic Autoimmune Disorder After Enrollment in Clinical Trials of GARDASIL,
Regardless of Causality
GARDASIL AAHS Control* or Saline
(N = 10,706) Placebo
Conditions (N = 9412)
n (%) n(%)
Arthralgia/Arthritis/Arthropathy 1 120(1.1) 98 (1.0)
Autoimmune Thyroiditis 4 (0.0) 1 (0.0)
Celiac Disease 10 (0.1) 6 (0.1)
Diabetes Mellitus Insulin-dependent 2 (0.0) 2 (0.0)
Erythema Nodosum 2 (0.0) 4 (0.0)
Hyperthyroidism* 27 (0.3) 21 (0.2)
Hypothyroidism§ 35 (0.3) 38 (0.4)
Inflammatory Bowel Disease' 7 (0.1) 10(0.1)
Multiple Sclerosis 2 (0.0) 4 (0.0)
Nephritis# 2 (0.0) 5 (0.1)
Optic Neuritis 2 (0.0) 0 (0.0)
Pigmentation Disorder" 4 (0.0) 3 (0.0)
PsoriasisB 13 (0.1) 15 (0.2)
Raynaud's Phenomenon 3 (0.0) 4 (0.0)
Rheumatoid Arthritis" 6 (0.1) 2 (0.0)
Scleroderma/Morphea 2 (0.0) 1 (0.0)
Stevens-Johnson Syndrome 1 (0.0) 0 (0.0)
Systemic Lupus Erythematosus 1 (0.0) 3 (0.0)
Uveitis 3 (0.0) 1 (0.0)
All Conditions 245 (2.3) 218 (2.3)
tArthralgia/Arthritis/Arthropathy includes the following terms: Arthralgia, Arthritis, Arthritis reactive, and Arthropathy
*Hyperthyroidism includes the following terms: Basedow's disease, Goiter, Toxic nodular goiter, and Hyperthyroidism
§Hypothyroidism includes the following terms: Hypothyroidism and thyroiditis
11nflammatory bowel disease includes the following terms: Colitis ulcerative, Crohn's disease, and Inflammatory bowel
disease
#Nephritis includes the following terms: Nephritis, Glomerulonephritis minimal lesion, Glomerulonephritis proliferative
PPigmentation disorder includes the following terms: Pigmentation disorder, Skin depigmentation, and Vitiligo
BPsoriasis includes the following terms: Psoriasis, Pustular psoriasis, and Psoriatic arthropathy
"Rheumatoid arthritis includes juvenile rheumatoid arthritis. One woman counted in the rheumatoid arthritis group
reported rheumatoid arthritis as an adverse experience at Day 130.
N = Number of individuals enrolled
n = Number of individuals with specific new Medical Conditions
NOTE: Although an individual may have had two or more new Medical Conditions, the individual is counted only once
within a category. The same individual may appear in different categories.
Systemic Autoimmune Disorders in Boys and Men 9 Through 26 Years of Age

In the clinical studies, 9- through 26-year-old boys and men were evaluated for new medical
conditions that occurred over the course of follow-up. New medical conditions potentially indicative of a
systemic autoimmune disorder seen in the group that received GARDASIL or AAHS control or saline
placebo are shown in Table 10. This population includes all boys and men who received at least one
dose of GARDASIL or AAHS control or saline placebo, and had safety data available.
Page 183 of 637

Table 10: Summary of Boys and Men 9 Through 26 Years of Age Who Reported an Incident Condition
Potentially Indicative of a Systemic Autoimmune Disorder After Enrollment in Clinical Trials of GARDASIL,
Regardless of Causality
AAHS Contror or Saline
GARDASIL Placebo
Conditions (N = 3093) (N = 2303)
n (%) n(%)
Alopecia Areata 2 (0.1) 0 (0.0)
Ankylosing Spondylitis 1 (0.0) 2 (0.1)
Arthralgia/Arthritis/Reactive Arthritis 30 (1.0) 17 (0.7)
Autoimmune Thrombocytopenia 1 (0.0) 0 (0.0)
Diabetes Mellitus Type 1 3 (0.1) 2 (0.1)
Hyperthyroidism 0 (0.0) 1 (0.0)
Hypothyroidism t 3 (0.1) 0 (0.0)
Inflammatory Bowel Diseaset 1 (0.0) 2 (0.1)
Myocarditis 1 (0.0) 1 (0.0)
Proteinuria 1 (0.0) 0 (0.0)
Psoriasis 0 (0.0) 4 (0.2)
Skin Depigmentation 1 (0.0) 0 (0.0)
Vitilioo 2 (0.1) 5 (0.2)
All Conditions 46 (1.5) 34 (1.5)
tHypothyroidism includes the following terms: Hypothyroidism and Autoimmune thyroiditis
t1nt1ammatorybowel disease includes the following terms: Colitis ulcerative and Crohn's disease
N = Number of individuals who received at least one dose of either vaccine or placebo
n = Number of individuals with specific new Medical Conditions
NOTE: Although an individual may have had two or more new Medical Conditions, the individual is counted only once
within a category. The same individual may appear in different categories.
Safety in Concomitant Use with RECOMBIVAX HB® [hepatitis 8 vaccine (recombinant)] in Girls and
Women 16 Through 23 Years of Age
The safety of GARDASIL when administered concomitantly with RECOMBIVAX HB® [hepatitis B
vaccine (recombinant)] was evaluated in an AAHS-controlled study of 1871 girls and women with a mean
age of 20.4 years [see Clinical Studies (14.10)]. The race distribution of the study individuals was as
follows: 61.6% White; 23.8% Other; 11.9% Black; 1.6% Hispanic (Black and White); 0.8% Asian; and
0.3% American Indian. The rates of systemic and injection-site adverse reactions were similar among
girls and women who received concomitant vaccination as compared with those who received
GARDASIL or RECOMBIVAX HB [hepatitis B vaccine (recombinant)].
Safety in Concomitant Use with Menactra [Meningococcal (Groups A, C, Y and W-135) Polysaccharide
Diphtheria Toxoid Conjugate Vaccine] and Adacel [Tetanus Toxoid, Reduced Diphtheria Toxoid and
Acel/ular Pertussis Vaccine Adsorbed (Tdap)]
The safety of GARDASIL when administered concomitantly with Menactra [Meningococcal (Groups A,
C, Y and W-135) Polysaccharide Diphtheria Toxoid Conjugate Vaccine] and Adacel [Tetanus Toxoid,
Reduced Diphtheria Toxoid and Acellular Pertussis Vaccine Adsorbed (Tdap)] was evaluated in a
randomized study of 1040 boys and girls with a mean age of 12.6 years [see Clinical Studies (14.11)].
The race distribution of the study subjects was as follows: 77.7% White; 1.4% Multi-racial; 12.3% Black;
6.8% Hispanic (Black and White); 1.2% Asian; 0.4% American Indian, and 0.2% Indian.
There was an increase in injection-site swelling reported at the injection site for GARDASIL
(concomitant= 10.9%, non-concomitant= 6.9%) when GARDASIL was administered concomitantly with
Menactra and Adacel as compared to non-concomitant (separated by 1 month) vaccination. The majority
of injection-site swelling adverse experiences were reported as being mild to moderate in intensity.
Safety in Women 27 Through 45 Years of Age
The adverse reaction profile in women 27 through 45 years of age was comparable to the profile seen
in girls and women 9 through 26 years of age.
6.2 Postmarketing Experience
The following adverse events have been spontaneously reported during post-approval use of
GARDASIL. Because these events were reported voluntarily from a population of uncertain size, it is not
possible to reliably estimate their frequency or to establish a causal relationship to vaccine exposure.
Blood and lymphatic system disorders: Autoimmune hemolytic anemia, idiopathic thrombocytopenic
purpura, lymphadenopathy.
Page 184 of 637

Respiratory, thoracic and mediastinal disorders: Pulmonary embolus.
Gastrointestinal disorders: Nausea, pancreatitis, vomiting.
General disorders and administration site conditions: Asthenia, chills, death, fatigue, malaise.
Immune system disorders: Autoimmune diseases, hypersensitivity reactions including
anaphylactic/anaphylactoid reactions, bronchospasm, and urticaria.
Musculoskeletal and connective tissue disorders: Arthralgia, myalgia.
Nervous system disorders: Acute disseminated encephalomyelitis, dizziness, Guillain-Barre
syndrome, headache, motor neuron disease, paralysis, seizures, syncope (including syncope associated
with tonic-clonic movements and other seizure-like activity) sometimes resulting in falling with injury,
transverse myelitis.
Infections and infestations: cellulitis.
Vascular disorders: Deep venous thrombosis.
7 DRUG INTERACTIONS
7.1 Use with RECOMBIVAX HB

Results from clinical studies indicate that GARDASIL may be administered concomitantly (at a
separate injection site) with RECOMBIVAX HB [hepatitis B vaccine (recombinant)] [see Clinical Studies
(14.10)].
7.2 Use with Menactra and Adacel
Results from clinical studies indicate that GARDASIL may be administered concomitantly (at a
separate injection site) with Menactra [Meningococcal (Groups A, C, Y and W-135) Polysaccharide
Diphtheria Toxoid Conjugate Vaccine] and Adacel [Tetanus Toxoid, Reduced Diphtheria Toxoid and
Acellular Pertussis Vaccine Adsorbed (Tdap)] [see Clinical Studies (14. 11)].
7.3 Use with Hormonal Contraceptives
In clinical studies of 16- through 26-year-old women, 13,912 (GARDASIL N = 6952; AAHS control or
saline placebo N = 6960) who had post-Month 7 follow-up used hormonal contraceptives for a total of
33,859 person-years (65.8% of the total follow-up time in the studies).
In one clinical study of 24- through 45-year-old women, 1357 (GARDASIL N =690; AAHS control N =
667) who had post-Month 7 follow-up used hormonal contraceptives for a total of 3400 person-years
(31.5% of the total follow-up time in the study). Use of hormonal contraceptives or lack of use of hormonal
contraceptives among study participants did not impair the immune response in the per protocol
immunogenicity (PPI) population.
7.4 Use with Systemic lmmunosuppressive Medications
lmmunosuppressive therapies, including irradiation, antimetabolites, alkylating agents, cytotoxic
drugs, and corticosteroids (used in greater than physiologic doses), may reduce the immune responses
to vaccines [see Use in Specific Populations (8. 6)].
8 USE IN SPECIFIC POPULATIONS
8. 1 Pregnancy
Pregnancy Category B:
Reproduction studies have been performed in female rats at doses equivalent to the recommended
human dose and have revealed no evidence of impaired female fertility or harm to the fetus due to
GARDASIL. There are, however, no adequate and well-controlled studies in pregnant women. Because
animal reproduction studies are not always predictive of human responses, GARDASIL should be used
during pregnancy only if clearly needed.
An evaluation of the effect of GARDASIL on embryo-fetal, pre- and postweaning development was
conducted using rats. One group of rats was administered GARDASIL twice prior to gestation, during the
period of organogenesis (gestation Day 6) and on lactation Day 7. A second group of pregnant rats was
administered GARDASIL during the period of organogenesis (gestation Day 6) and on lactation Day 7
only. GARDASIL was administered at 0.5 mUrat/occasion (120 mcg total protein which is equivalent to
10
Page 185 of 637

the recommended human dose) by intramuscular injection. No adverse effects on mating, fertility,
pregnancy, parturition, lactation, embryo-fetal or pre- and postweaning development were observed.
There were no vaccine-related fetal malformations or other evidence of teratogenesis noted in this study.
In addition, there were no treatment-related effects on developmental signs, behavior, reproductive
performance, or fertility of the offspring.
Clinical Studies in Humans
In clinical studies, women underwent urine pregnancy testing prior to administration of each dose of
GARDASIL. Women who were found to be pregnant before completion of a 3-dose regimen of
GARDASIL were instructed to defer completion of their vaccination regimen until resolution of the
pregnancy.
GARDASIL is not indicated for women 27 years of age or older. However, safety data in women 16
through 45 years of age was collected, and 3819 women (GARDASIL N = 1894 vs. AAHS control or
saline placebo N = 1925) reported at least 1 pregnancy each.
The overall proportions of pregnancies that resulted in an adverse outcome, defined as the combined
numbers of spontaneous abortion, late fetal death, and congenital anomaly cases out of the total number
of pregnancy outcomes for which an outcome was known (and excluding elective terminations), were
22.6% (446/1973) in women who received GARDASIL and 23.1 % (460/1994) in women who received
AAHS control or saline placebo.
Overall, 55 and 65 women in the group that received GARDASIL or AAHS control or saline placebo,
respectively (2.9% and 3.4% of all women who reported a pregnancy in the respective vaccination
groups), experienced a serious adverse reaction during pregnancy. The most common events reported
were conditions that can result in Caesarean section (e.g., failure of labor, malpresentation, cephalopelvic
disproportion), premature onset of labor (e.g., threatened abortions, premature rupture of membranes),
and pregnancy-related medical problems (e.g., pre-eclampsia, hyperemesis). The proportions of
pregnant women who experienced such events were comparable between the groups receiving
GARDASIL and AAHS control or saline placebo.
There were 45 cases of congenital anomaly in pregnancies that occurred in women who received
GARDASIL and 34 cases of congenital anomaly in pregnancies that occurred in women who received
AAHS control or saline placebo.
Further sub-analyses were conducted to evaluate pregnancies with estimated onset within 30 days or
more than 30 days from administration of a dose of GARDASIL or AAHS control or saline placebo. For
pregnancies with estimated onset within 30 days of vaccination, 5 cases of congenital anomaly were
observed in the group that received GARDASIL compared to 1 case of congenital anomaly in the group
that received AAHS control or saline placebo. The congenital anomalies seen in pregnancies with
estimated onset within 30 days of vaccination included pyloric stenosis, congenital megacolon, congenital
hydronephrosis, hip dysplasia, and club foot. Conversely, in pregnancies with onset more than 30 days
following vaccination, 40 cases of congenital anomaly were observed in the group that received
GARDASIL compared with 33 cases of congenital anomaly in the group that received AAHS control or
saline placebo.
Women who receive GARDASIL during pregnancy are encouraged to contact Merck Sharp &
Dohme Corp., a subsidiary of Merck & Co., Inc., at 1-877-888-4231 or VAERS at 1-800-822-7967 or
www.vaers.hhs.gov.
8.3 Nursing Mothers
Women 16 Through 45 Years of Age
It is not known whether GARDASIL is excreted in human milk. Because many drugs are excreted in
human milk, caution should be exercised when GARDASIL is administered to a nursing woman.
GARDASIL or AAHS control were given to a total of 1133 women (vaccine N = 582, AAHS control N =
551) during the relevant Phase 3 clinical studies.
Overall, 27 and 13 infants of women who received GARDASIL or AAHS control, respectively
(representing 4.6% and 2.4% of the total number of women who were breast-feeding during the period in
which they received GARDASIL or AAHS control, respectively), experienced a serious adverse reaction.
In a post-hoc analysis of clinical studies, a higher number of breast-feeding infants (n = 7) whose
mothers received GARDASIL had acute respiratory illnesses within 30 days post vaccination of the
mother as compared to infants (n = 2) whose mothers received AAHS control.
11
Page 186 of 637

8.4 Pediatric Use
Safety and effectiveness have not been established in pediatric patients below 9 years of age.
8.5 Geriatric Use
The safety and effectiveness of GARDASIL have not been evaluated in a geriatric population, defined
as individuals aged 65 years and over.
8.6 lmmunocompromised Individuals
The immunologic response to GARDASIL may be diminished in immunocompromised individuals [see
Drug Interactions (7.4)].
10 OVERDOSAGE
There have been reports of administration of higher than recommended doses of GARDASIL.
In general, the adverse event profile reported with overdose was comparable to recommended single
doses of GARDASIL.
11 DESCRIPTION
GARDASIL, Human Papillomavirus Quadrivalent (Types 6, 11, 16, and 18) Vaccine, Recombinant, is
a non-infectious recombinant quadrivalent vaccine prepared from the purified virus-like particles (VLPs) of
the major capsid (L 1) protein of HPV Types 6, 11, 16, and 18. The L1 proteins are produced by separate
fermentations in recombinant Saccharomyces cerevisiae and self-assembled into VLPs. The fermentation
process involves growth of S. cerevisiae on chemically-defined fermentation media which include
vitamins, amino acids, mineral salts, and carbohydrates. The VLPs are released from the yeast cells by
cell disruption and purified by a series of chemical and physical methods. The purified VLPs are adsorbed
on preformed aluminum-containing adjuvant (Amorphous Aluminum Hydroxyphosphate Sulfate). The
quadrivalent HPV VLP vaccine is a sterile liquid suspension that is prepared by combining the adsorbed
VLPs of each HPV type and additional amounts of the aluminum-containing adjuvant and the final
purification buffer.
GARDASIL is a sterile suspension for intramuscular administration. Each 0.5-ml dose contains
approximately 20 mcg of HPV 6 L1 protein, 40 mcg of HPV 11 L1 protein, 40 mcg of HPV 16 L1 protein,
and 20 mcg of HPV 18 L1 protein.
Each 0.5-ml dose of the vaccine contains approximately 225 mcg of aluminum (as Amorphous
Aluminum Hydroxyphosphate Sulfate adjuvant), 9.56 mg of sodium chloride, 0.78 mg of L-histidine,
50 mcg of polysorbate 80, 35 mcg of sodium borate, <7 mcg yeast protein/dose, and water for injection.
The product does not contain a preservative or antibiotics.
After thorough agitation, GARDASIL is a white, cloudy liquid.
12 CLINICAL PHARMACOLOGY
12.1 Mechanism of Action

HPV only infects human beings. Animal studies with analogous animal papillomaviruses suggest that
the efficacy of L1 VLP vaccines may involve the development of humoral immune responses. Human
beings develop a humoral immune response to the vaccine, although the exact mechanism of protection
is unknown.
13 NONCLINICAL TOXICOLOGY
13.1 Carcinogenesis, Mutagenesis, Impairment of Fertility

GARDASIL has not been evaluated for the potential to cause carcinogenicity or genotoxicity.
GARDASIL administered to female rats at a dose of 120 mcg total protein, which is equivalent to the
recommended human dose, had no effects on mating performance, fertility, or embryonic/fetal survival.
The effect of GARDASIL on male fertility has been studied in male rats at an intramuscular dose of 0.5
ml/rat/occasion (120 mcg total protein which is equivalent to the recommended human dose). One group
of male rats was administered GARDASIL once, 3 days prior to cohabitation, and a second group of male
rats was administered GARDASIL three times, at 6 weeks, 3 weeks, and 3 days prior to cohabitation.
12
Page 187 of 637

There were no treatment-related effects on reproductive performance including fertility, sperm count, and
sperm motility. There were no treatment-related gross or histomorphologic and weight changes on the
testes.
14 CLINICAL STUDIES
CIN 2/3 and AIS are the immediate and necessary precursors of squamous cell carcinoma and
adenocarcinoma of the cervix, respectively. Their detection and removal has been shown to prevent
cancer; thus, they serve as surrogate markers for prevention of cervical cancer. In the clinical studies in
girls and women aged 16 through 26 years, cases of CIN 2/3 and AIS were the efficacy endpoints to
assess prevention of cervical cancer. In addition, cases of VIN 2/3 and ValN 2/3 were the efficacy
endpoints to assess prevention of HPV-related vulvar and vaginal cancers, and observations of external
genital lesions were the efficacy endpoints for the prevention of genital warts.
In clinical studies in boys and men aged 16 through 26 years, efficacy was evaluated using the
following endpoints: external genital warts and penile/perineal/perianal intraepithelial neoplasia (PIN)
grades 1/2/3 or penile/perineal/perianal cancer. In addition, cases of AIN grades 1/2/3 and anal cancer
made up the composite efficacy endpoint used to assess prevention of HPV-related anal cancer.
Anal HPV infection, AIN, and anal cancer were not endpoints in the studies conducted in women. The
similarity of HPV-related anal disease in men and women supports bridging the indication of prevention of
AIN and anal cancer to women.
Efficacy was assessed in 6 AAHS-controlled, double-blind, randomized Phase 2 and 3 clinical studies.
The first Phase 2 study evaluated the HPV 16 component of GARDASIL (Study 1, N = 2391 16- through
26-year-old girls and women) and the second evaluated all components of GARDASIL (Study 2, N = 551
16- through 26-year-old girls and women). Two Phase 3 studies evaluated GARDASIL in 5442 (Study 3)
and 12,157 (Study 4) 16- through 26-year-old girls and women. A third Phase 3 study, Study 5, evaluated
GARDASIL in 4055 16- through 26-year-old boys and men, including a subset of 598 (GARDASIL = 299;
placebo = 299) men who self-identified as having sex with men (MSM population). A fourth Phase 3
study, Study 6, evaluated GARDASIL in 3817 24- through 45-year-old women. Together, these six
studies evaluated 28,413 individuals (20,541 girls and women 16 through 26 years of age at enrollment
with a mean age of 20.0 years, 4055 boys and men 16 through 26 years of age at enrollment with a mean
age of 20.5 years, and 3817 women 24 through 45 years of age at enrollment with a mean age of 34.3
years). The race distribution of the 16- through 26-year-old girls and women in the clinical trials was as
follows: 70.4% White; 12.2% Hispanic (Black and White); 8.8% Other; 4.6% Black; 3.8% Asian; and 0.2%
American Indian. The race distribution of the 16- through 26-year-old boys and men in the clinical trials
was as follows: 35.2% White; 20.5% Hispanic (Black and White); 14.4% Other; 19.8% Black; 10.0%
Asian; and 0.1% American Indian. The race distribution of the 24- through 45-year-old women in the
clinical trials was as follows: 20.6% White; 43.2% Hispanic (Black and White); 0.2% Other; 4.8% Black;
31.2% Asian; and 0.1% American Indian.
The median duration of follow-up was 4.0, 3.0, 3.0, 3.0, 2.3, and 4.0 years for Study 1, Study 2, Study
3, Study 4, Study 5, and Study 6, respectively. Individuals received vaccine or AAHS control on the day of
enrollment and 2 and 6 months thereafter. Efficacy was analyzed for each study individually and for all
studies in girls and women combined according to a prospective clinical plan.
Overall, 73% of 16- through 26-year-old girls and women, 67% of 24- through 45-year-old women, and
83% of 16- through 26-year-old boys and men were naTve (i.e., PCR [Polymerase Chain Reaction]
negative and seronegative for all 4 vaccine HPV types) to all 4 vaccine HPV types at enrollment.
A total of 27% of 16- through 26-year-old girls and women, 33% of 24- through 45-year-old women,
and 17% of 16- through 26-year-old boys and men had evidence of prior exposure to or ongoing infection
with at least 1 of the 4 vaccine HPV types. Among these individuals, 74% of 16- through 26-year-old girls
and women, 71% of 24- through 45-year-old women, and 78% of 16- through 26-year-old boys and men
had evidence of prior exposure to or ongoing infection with only 1 of the 4 vaccine HPV types and were
naTve(PCR negative and seronegative) to the remaining 3 types.
In 24- through 45-year-old individuals, 0.4% had been exposed to all 4 vaccine HPV types.
In individuals who were naTve (PCR negative and seronegative) to all 4 vaccine HPV types, CIN,
genital warts, VIN, ValN, PIN, and persistent infection caused by any of the 4 vaccine HPV types were
counted as endpoints.
Among individuals who were positive (PCR positive and/or seropositive) for a vaccine HPV type at
Day 1, endpoints related to that type were not included in the analyses of prophylactic efficacy. Endpoints
13
Page 188 of 637

related to the remaining types for which the individual was na"fve(PGR negative and seronegative) were
counted.
For example, in individuals who were HPV 18 positive (PGR positive and/or seropositive) at Day 1,
lesions caused by HPV 18 were not counted in the prophylactic efficacy evaluations. Lesions caused by
HPV 6, 11, and 16 were included in the prophylactic efficacy evaluations. The same approach was used
for the other types.
14.1 Prophylactic Efficacy- HPV Types 6, 11, 16, and 18 in Girls and Women 16 through 26 Years
of Age
GARDASIL was administered without prescreening for presence of HPV infection and the efficacy
trials allowed enrollment of girls and women regardless of baseline HPV status (i.e., PGR status or
serostatus). Girls and women with current or prior HPV infection with an HPV type contained in the
vaccine were not eligible for prophylactic efficacy evaluations for that type.
The primary analyses of efficacy with respect to HPV types 6, 11, 16, and 18 were conducted in the
per-protocol efficacy (PPE) population, consisting of girls and women who received all 3 vaccinations
within 1 year of enrollment, did not have major deviations from the study protocol, and were na"iVe(PGR
negative in cervicovaginal specimens and seronegative) to the relevant HPV type(s) (Types 6, 11, 16,
and 18) prior to dose 1 and through 1 month Postdose 3 (Month 7). Efficacy was measured starting after
the Month 7 visit.
GARDASIL was efficacious in reducing the incidence of GIN (any grade including GIN 2/3); AIS;
genital warts; VIN (any grade); and ValN (any grade) related to vaccine HPV types 6, 11, 16, or 18 in
those who were PGR negative and seronegative at baseline (Table 11).
In addition, girls and women who were already infected with 1 or more vaccine-related HPV types
prior to vaccination were protected from precancerous cervical lesions and external genital lesions
caused by the other vaccine HPV types.
14
Page 189 of 637

Table 11: Analysis of Efficacy of GARDASIL in the PPE* Populationt of 16-Through 26-Year-Old Girls and Women for
Vaccine HPV Types
Population
GARDASIL AAHS Control I % Efficacy {95% Cl)
N Number of cases N Number of cases I
HPV 16- or 18-related GIN 2/3 or AIS
Study 1"' 755 0 750 12 100.0 (65.1, 100.0)
Study 2 231 0 230 1 100.0 (-3744.9, 100.0)
Studv 3 2201 0 2222 36 100.0 (89.2, 100.0)
Study4 5306 2 5262 63 96.9 (88.2, 99.6)
Combined Protocolss 8493 2 8464 112 98.2 (93.5, 99.8)
HPV 16-related GIN 2/3 or AIS
Combined Protocols' 7402 2 7205 93 97.9 (92.3, 99.8)
HPV 18-related GIN 2/3 or AIS
HPV 16- or 18-related VIN 2/3
Study2 231 0 230 0 Not calculated
Study 3 2219 0 2239 6 100.0 (14.4, 100.0)
Studv4 5322 0 5275 4 100.0 (-50.3, 100.0)
HPV 16- or 18-related ValN 2/3
Study 2 231 0 230 0 Not calculated
Studv3 2219 0 2239 5 100.0 (-10.1, 100.0)
Study4 5322 0 5275 4 100.0 (-50.3, 100.0)
HPV 6-, 11-, 16-, or 18-related GIN (GIN 1, GIN 2/3) or AIS
Study 2 235 0 233 3 100.0 (-138.4, 100.0)
Study3 2241 0 2258 77 100.0 (95.1, 100.0)
Study4 5388 9 5374 145 93.8 (88.0, 97.2)
HPV 6-, 11-, 16-, or 18-related Genital Warts
Study 2 235 0 233 3 100.0 (-139.5, 100.0)
Study 3 2261 0 2279 58 100.0 (93.5, 100.0)
Studv4 5404 2 5390 132 98.5 (94.5, 99.8)
HPV 6- and 11-related Genital Warts
.. ..
*The PPE populat1on consisted of 1nd1v1dualswho received all 3 vaccinations within 1 year of enrollment, did not have maier
deviations from the study protocol, and were na"ive (PCR negative and seronegative) to the relevant HPV type(s) (Types 6, 11, 16,
and 18) prior to dose 1 and through 1 month postdose 3 (Month 7).
tsee Table 14 for analysis of vaccine impact in the general population.
*Evaluated only the HPV 16 L1 VLP vaccine component of GARDASIL
§Analyses of the combined trials were prospectively planned and included the use of similar study entry criteria.
N = Number of individuals with at least 1 follow-up visit after Month 7
Cl = Confidence Interval
Note 1: Point estimates and confidence intervals are adjusted for person-time of follow-up.
Note 2: The first analysis in the table (i.e., HPV 16- or 18-related CIN 2/3, AIS or worse) was the primary endpoint of the vaccine
development plan.
Note 3: Table 11 does not include cases due to non-vaccine HPV types.
AAHS Control = Amorphous Aluminum Hydroxyphosphate Sulfate
Prophylactic efficacy against overall cervical and genital disease related to HPV 6, 11, 16, and 18 in
an extension phase of Study 2, that included data through Month 60, was noted to be 100% (95% Cl:
12.3%, 100.0%) among girls and women in the per protocol population naTveto the relevant HPV types.
GARDASIL was efficacious against HPV disease caused by HPV types 6, 11, 16, and 18 in girls and
women who were na"ivefor those specific HPV types at baseline.
14.2 Prophylactic Efficacy - HPV Types 6, 11, 16, and 18 in Boys and Men 16 through 26 Years of
Age
The primary analyses of efficacy were conducted in the per-protocol efficacy (PPE) population. This
population consisted of boys and men who received all 3 vaccinations within 1 year of enrollment, did not
have major deviations from the study protocol, and were na"ive (PCR negative and seronegative) to the
relevant HPV type(s) (Types 6, 11, 16, and 18) prior to dose 1 and through 1 month postdose 3 (Month
7). Efficacy was measured starting after the Month 7 visit.
15
Page 190 of 637

GARDASIL was efficacious in reducing the incidence of genital warts related to vaccine HPV types 6
and 11 in those boys and men who were PCR negative and seronegative at baseline (Table 12). Efficacy
against penile/perineal/perianal intraepithelial neoplasia (PIN) grades 1/2/3 or penile/perineal/perianal
cancer was not demonstrated as the number of cases was too limited to reach statistical significance.
Table 12: Analysis of Efficacy of GARDASIL in the PPE* Population of 16- Through 26-Year-Old Boys and Men for Vaccine HPV
Types
Endpoint I GARDASIL I AAHS Control I % Efficacy (95% Cl)
I NT I Number of cases I N I Number of cases I
External Genital Lesions HPV 6-, 11-, 16-, or 18- related
External Genital Lesions I 1394 I 3 I 1404 I 32 I 90.6 (70.1, 98.2)
Condyloma I 1394 I 3 I 1404 I 28 I 89.3 (65.3, 97.9)
PIN 1/2/3 I 1394
.. I 0 I 1404 I 4 I 100.0 (-52.1, 100.0)
*The PPE population consisted of ind1v1dualswho received all 3 vaccinations within 1 year of enrollment, did not have major dev1at1ons
from the study protocol, and were naTve (PCR negative and seronegative) to the relevant HPV type(s) (Types 6, 11, 16, and 18) prior to
dose 1 and through 1 month postdose 3 (Month 7).
tN = Number of individuals with at least 1 follow-up visit after Month 7
Cl = Confidence lnteNal
14.3 Prophylactic Efficacy - Anal Disease Caused by HPV Types 6, 11, 16, and 18 in Boys and
Men 16 through 26 Years of Age in the MSM Sub-study
A sub-study of Study 5 evaluated the efficacy of GARDASIL against anal disease (anal intraepithelial
neoplasia and anal cancer) in a population of 598 MSM. The primary analyses of efficacy were conducted
in the per-protocol efficacy (PPE) population of Study 5.
GARDASIL was efficacious in reducing the incidence of anal intraepithelial neoplasia (AIN) grades 1
(both condyloma and non-acuminate), 2, and 3 related to vaccine HPV types 6, 11, 16, and 18 in those
boys and men who were PCR negative and seronegative at baseline (Table 13).
Table 13: Analysis of Efficacy of GARDASIL for Anal Disease in the PPE* Population of 16-Through 26-Year-Old Boys and Men
in the MSM Sub-study for Vaccine HPV Types
HPV 6-, 11-, 16-, or 18- related GARDASIL AAHS Control
% Efficacy (95% Cl)
Endpoint w Number of cases N Number of cases
AIN 1/2/3 194 5 208 24 77.5 (39.6, 93.3)
AIN 2/3 194 3 208 13 74.9 (8.8, 95.4)
AIN 1 194 4 208 16 73.0 (16.3, 93.4)
Condyloma Acuminatum 194 0 208 6 100.0 (8.2, 100.0)
Non-acuminate 194 4 208 11 60.4 (-33.5, 90.8)
..
*The PPE population consisted of ind1v1dualswho received all 3 vaccinations w1th1n1 year of enrollment, did not have major dev1at1ons
from the study protocol, and were naTve (PCR negative and seronegative) to the relevant HPV type(s) (Types 6, 11, 16, and 18) prior to
dose 1 and through 1 month postdose 3 (month 7).
tN = Number of individuals with at least 1 follow-up visit after Month 7
Cl = Confidence lnteNal
14.4 Population Impact in Girls and Women 16 through 26 Years of Age

Effectiveness of GARDASIL in Prevention of HPV Types 6-, 11-, 16-, or 18-Re/ated Genital Disease in
Girls and Women 16 Through 26 Years of Age, Regardless of Current or Prior Exposure to Vaccine HPV
Types
The clinical trials included girls and women regardless of current or prior exposure to vaccine HPV
types, and additional analyses were conducted to evaluate the impact of GARDASIL with respect to HPV
6-, 11-, 16-, and 18-related cervical and genital disease in these girls and women. Here, analyses
included events arising among girls and women regardless of baseline PCR status and serostatus,
including HPV infections that were present at the start of vaccination as well as events that arose from
infections that were acquired after the start of vaccination.
The impact of GARDASIL in girls and women regardless of current or prior exposure to a vaccine HPV
type is shown in Table 14. Impact was measured starting 1 month Postdose 1. Prophylactic efficacy
denotes the vaccine's efficacy in girls and women who are na'ive (PCR negative and seronegative) to the
relevant HPV types at Day 1. Vaccine impact in girls and women who were positive for vaccine HPV
16
Page 191 of 637

infection, as well as vaccine impact among girls and women regardless of baseline vaccine HPV PCR
status and serostatus are also presented. The majority of CIN and genital warts, VIN, and ValN related to
a vaccine HPV type detected in the group that received GARDASIL occurred as a consequence of HPV
infection with the relevant HPV type that was already present at Day 1.
There was no clear evidence of protection from disease caused by HPV types for which girls and
women were PCR positive regardless of serostatus at baseline.
Table 14: Effectiveness of GARDASIL in Prevention of HPV 6, 11, 16, or 18-Related Genital Disease in Girls and Women 16
Through 26 Years of Age, Regardless of Current or Prior Exposure to Vaccine HPV Tvpes
GARDASIL or HPV 16
AAHS Control % Reduction
Endpoint Analysis L1 VLP Vaccine
(95% Cl)
N Cases N Cases
HPV 16- or 18-

related CIN 2/3 or
AIS
Prophylactic Efficacv*
HPV 16 and/or HPV 18 Positive at Dav 1T
Girls and Women Regardless of Current or Prior
Exposure to HPV 16 or 181
9346
2870
9836
4
142
146
9407
2898
9904
155
148...
303
97.4
-·
(93.3, 99.3)
51.8 (41.1, 60.7)
Prophylactic Efficacy* 8642 1 8673 34 97.0 (82.4, 99.9)

HPV 16- or 18- __§
HPV 16 and/or HPV 18 Positive at Day 1t 1880 8 1876 4
related VIN 2/3 or
ValN 2/3 Girls and Women Regardless of Current or Prior
8955 9 8968 38 76.3 (50.0, 89.9)
Exposure to HPV 16 or 181
ProPhvlactic Efficacv* 8630 16 8680 309 94.8 (91.5, 97.1)
HPV 6-, 11-, 16-,
HPV6, HPV 11, HPV 16, and/orHPV 18 186# 213# _§
18-related CIN 2466 2437
Positive at Dav 1t
(CIN 1, CIN 2/3)
orAIS 8819 202 8854 522 61.5 (54.6, 67.4)
Exposure to Vaccine HPV Tvpes 1
HPV 6-, 11-, 16-, HPV6, HPV 11, HPV 16, and/orHPV 18 __§
2501 51P 2475 55P
or 18-related Positive at Day 1t
Genital Warts Girls and Women Regardless of Current or Prior
8955 61 8968 307 80.3 (73.9, 85.3)
HPV6-or11-
related Genital
HPV 6 and/or HPV 11 Positive at Dav 1T
1186 51 1176 54
--·
Warts 8955 60 8968 300 80.1 (73.7, 85.2)
. . ..
*Includes all 1nd1v1duals who received at least 1 vaccination and who were HPV-na1ve (1.e., seronegat1ve and PCR negative) at Day 1
to the vaccine HPV type being analyzed. Case counting started at 1 month postdose 1.
t1ncludes all individuals who received at least 1 vaccination and who were HPV positive or had unknown HPV status at Day 1, to at
least one vaccine HPV type. Case counting started at Day 1.
"'out of the 148 AAHS control cases of 16/18 CIN 2/3, 2 women were missing serology or PCR results for Day 1.
§There is no expected efficacy since GARDASIL has not been demonstrated to provide protection against disease from vaccine HPV
~pes to which a person has previously been exposed through sexual activity.
Includes all individuals who received at least 1 vaccination (regardless of baseline HPV status at Day 1). Case counting started at 1
month postdose 1.
#Includes 2 AAHS control women with missing serology/PCR data at Day 1.
PIncludes 1 woman with missing serology/PCR data at Day 1.
N = Number of individuals who have at least one follow-up visit after Day 1
Note 1: The 16- and 18-related CIN 2/3 or AIS composite endpoint included data from studies 1, 2, 3, and 4. All other endpoints only
included data from studies 2, 3, and 4.
Note 2: Positive status at Day 1 denotes PCR positive and/or seropositive for the respective type at Day 1.
Note 3: Table 14 does not include disease due to non-vaccine HPV types.
Effectiveness of GARDASIL in Prevention of Any HPV Type Related Genital Disease in Girls and Women
16 Through 26 Years of Age, Regardless of Current or Prior Infection with Vaccine or Non-Vaccine HPV
Types
The impact of GARDASIL against the overall burden of dysplastic or papillomatous cervical, vulvar,
and vaginal disease regardless of HPV detection, results from a combination of prophylactic efficacy
against vaccine HPV types, disease contribution from vaccine HPV types present at time of vaccination,
the disease contribution from HPV types not contained in the vaccine, and disease in which HPV was not
detected.
Additional efficacy analyses were conducted in 2 populations: (1) a generally HPV-naTve population
(negative to 14 common HPV types and had a Pap test that was negative for SIL [Squamous
17
Page 192 of 637

lntraepithelial Lesion] at Day 1), approximating a population of sexually-narve girls and women and (2)
the general study population of girls and women regardless of baseline HPV status, some of whom had
HPV-related disease at Day 1.
Among generally HPV-na"ive girls and women and among all girls and women in the study population
(including girls and women with HPV infection at Day 1), GARDASIL reduced the overall incidence of CIN
2/3 or AIS; of VIN 2/3 or ValN 2/3; of CIN (any grade) or AIS; and of Genital Warts (Table 15). These
reductions were primarily due to reductions in lesions caused by HPV types 6, 11, 16, and 18 in girls and
women narve (seronegative and PCR negative) for the specific relevant vaccine HPV type. Infected girls
and women may already have CIN 2/3 or AIS at Day 1 and some will develop CIN 2/3 or AIS during
follow-up, either related to a vaccine or non-vaccine HPV type present at the time of vaccination or
related to a non-vaccine HPV type not present at the time of vaccination.
Table 15: Effectiveness of GARDASIL in Prevention of Any HPV Type Related Genital Disease in Girls and Women 16
. Infect1on
Th roug h 26 Y ears of A,ge, ReJar diess of C urrent or Prior . wit. h V accme
. or Non-V accme
. HPV T1ypes
Endpoints Caused by Vaccine or GARDASIL AAHS Control % Reduction
Analysis
Non-vaccine HPV Types N Cases N Cases (95% Cl)
Girls and Women
CIN 2/3 or AIS Regardless of Current
or Prior Exposure to 8559 421 8592 516 18.4 (7.0, 28.4)
Vaccine or Non-
Vaccine HPV Tvoes t
Girls and Women
VIN 2/3 and ValN 2/3 Regardless of Current
Vaccine or Non-
Vaccine HPV Types t
Girls and Women
CIN (Any Grade) or AIS Regardless of Current
Vaccine or Non-
Vaccine HPVTvpest
Girls and Women
Genital Warts Regardless of Current
Vaccine or Non-
Vaccine HPVTypest
..
*Includes all md1v1dualswho received at least 1 vaccination and who had a Pap test that was negative for SIL [Squamous
lntraepithelial Lesion] at Day 1 and were narve to 14 common HPV types at Day 1. Case counting started at 1 month postdose 1.
t1ncludes all individuals who received at least 1 vaccination (regardless of baseline HPV status or Pap test result at Day 1). Case
counting started at 1 month postdose 1.
Cl= Confidence Interval
14.5 Population Impact in Boys and Men 16 through 26 Years of Age

Effectiveness of GARDASIL in Prevention of HPV Types 6-, 11-, 16-, or 18-Related Anogenital Disease in
Boys and Men 16 Through 26 Years of Age, Regardless of Current or Prior Exposure to Vaccine HPV
Types
Study 5 included boys and men regardless of currel")t or prior exposure to vaccine HPV types, and
additional analyses were conducted to evaluate the impact of GARDASIL with respect to HPV 6-, 11-,
16-, and 18-related anogenital disease in these boys and men. Here, analyses included events arising
among boys and men regardless of baseline PCR status and serostatus, including HPV infections that
were present at the start of vaccination as well as events that arose from infections that were acquired
after the start of vaccination.
The impact of GARDASIL in boys and men regardless of current or prior exposure to a vaccine HPV
type is shown in Table 16. Impact was measured starting at Day 1. Prophylactic efficacy denotes the
18
Page 193 of 637

vaccine's efficacy in boys and men who are naTve (PCR negative and seronegative) to the relevant HPV
types at Day 1. Vaccine impact in boys and men who were positive for vaccine HPV infection, as well as
vaccine impact among boys and men regardless of baseline vaccine HPV PCR status and serostatus are
also presented. The majority of anogenital disease related to a vaccine HPV type detected in the group
that received GARDASIL occurred as a consequence of HPV infection with the relevant HPV type that
was already present at Day 1.
There was no clear evidence of protection from disease caused by HPV types for which boys and men
were PCR positive regardless of serostatus at baseline.
Table 16: Effectiveness of GARDASIL in Prevention of HPV Types 6-, 11-, 16-, or 18-Related Anogenital Disease
.m Boysan d Men 16 Th rough 26 Years of A,ge, Rei ar diess of C urrent or Pnor . Exposure t o Vaccme . HPV Types
AAHS % Reduction
GARDASIL
Endpoint Analysis Control (95% Cl)
N Cases N Cases
HPV 6, HPV 11, HPV 16, and/or __+
460 14 453 26
External HPV 18 Positive at Dav 1t
Genital Boys and Men Regardless of
Lesions Current or Prior Exposure to
1943 27 1937 80 66.7 (48.0, 79.3)
Vaccine or Non-Vaccine HPV
Tvpes§
HPV 6, HPV 11, HPV 16, and/or _+
460 14 453 25
HPV 18 Positive at Dav 1t
Condyloma Boys and Men Regardless of
Current or Prior Exposure to
1943 24 1937 74 68.1 (48.8, 80.7)
Tvpes§
Prophylactic Efficacy* 1775 4 1770 5 20.7 (-268.4, 84.3)
HPV 6, HPV 11, HPV 16, and/or _+
460 2 453 1
PIN 1/2/3 Boys and Men Regardless of
1943 6 1937 6 0.3 (-272.8, 73.4)
Types§
HPV 6, HPV 11, HPV 16, and/or •.+
103 29 116 38
AIN 1/2/3 Boys and Men Regardless of
275 38 276 77 50.3 (25.7, 67.2)
Tvpes§
HPV 6, HPV 11, HPV 16, and/or __+
103 11 116 20
HPV 18 Positive at Day 1t
AIN 2/3 Boys and Men Regardless of
275 18 276 39 54.2 (18.0, 75.3)
Types§
*Includes all individuals who received at least 1 vaccination and who were HPV-na"ive (i.e., seronegative and PCR
negative) at Day 1 to the vaccine HPV type being analyzed. Case counting started at Day 1.
t1ncludes all individuals who received at least 1 vaccination and who were HPV positive or had unknown HPV status at
Day 1, to at least one vaccine HPV type. Case counting started at Day 1.
"'There is no expected efficacy since GARDASIL has not been demonstrated to provide protection against disease from
vaccine HPV types to which a person has previously been exposed through sexual activity.
§Includes all individuals who received at least 1 vaccination. Case counting started at Day 1.
Effectiveness of GARDASIL in Prevention of Any HPV Type Related Anogenital Disease in Boys and
Men 16 Through 26 Years of Age, Regardless of.Current or Prior Infection with Vaccine or Non-Vaccine
HPVTypes
The impact of GARDASIL against the overall burden of dysplastic or papillomatous anogenital disease
regardless of HPV detection, results from a combination of prophylactic efficacy against vaccine HPV
19
Page 194 of 637

types, disease contribution from vaccine HPV types present at time of vaccination, the disease
contribution from HPV types not contained in the vaccine, and disease in which HPV was not detected.
Additional efficacy analyses from Study 5 were conducted in 2 populations: (1) a generally HPV-na"ive
population that consisted of boys and men who are seronegative and PCR negative to HPV 6, 11, 16,
and 18 and PCR negative to HPV 31, 33, 35, 39, 45, 51, 52, 56, 58 and 59 at Day 1, approximating a
population of sexually-naTve boys and men and (2) the general study population of boys and men
regardless of baseline HPV status, some of whom had HPV-related disease at Day 1.
Among generally HPV-naTve boys and men and among all boys and men in Study 5 (including boys
and men with HPV infection at Day 1), GARDASIL reduced the overall incidence of anogenital disease
(Table 17). These reductions were primarily due to reductions in lesions caused by HPV types 6, 11, 16,
and 18 in boys and men na"ive (seronegative and PCR negative) for the specific relevant vaccine HPV
type. Infected boys and men may already have anogenital disease at Day 1 and some will develop
anogenital disease during follow-up, either related to a vaccine or non-vaccine HPV type present at the
time of vaccination or related to a non-vaccine HPV type not present at the time of vaccination.
Table 17: Effectiveness of GARDASIL in Prevention of Any HPV Type Related Anogenital Disease in Boys
and Men 16 Through 26 Years of Age, Regardless of Current or Prior Infection with Vaccine or Non-Vaccine
HPVT ypes
AAHS
GARDASIL % Reduction
Endpoint Analysis Control
(95%CI)
N Cases N Cases
External Boys and Men Regardless of
Genital Current or Prior Exposure to
1943 38 1937 92 59.3 (40.0, 72.9)
Lesions Vaccine or Non-Vaccine HPV
Typest
Boys and Men Regardless of
Condyloma Current or Prior Exposure to
1943 33 1937 85 61.8 (42.3, 75.3)
Typest
Prophylactic Efficacv* 1275 2 1270 4 50.7 (-244.3, 95.5)
PIN 1/2/3 Current or Prior Exposure to
1943 8 1937 7 -13.9 (-269.0, 63.9)
Typest
Prophylactic Efficacv* 129 12 126 28 54.9 (8.4, 79.1)
AIN 1/2/3 Current or Prior Exposure to
275 74 276 103 25.7 (-1.1, 45.6)
Typest
Proohvlactic Efficacv* 129 8 126 18 52.5 (-14.8, 82.1)
AIN 2/3 Current or Prior Exposure to
275 44 276 59 24.3 (-13.8, 50.0)
Tvoest
..
*Includes all 1nd1v1duals who received at least 1 vaccination and who were seronegative and PCR negative at
enrollment to HPV 6, 11, 16 and 18, and PCR negative at enrollment to HPV 31, 33, 35, 39, 45, 51, 52, 56, 58 and
59. Case counting started at Day 1.
t1ncludes all individuals who received at least 1 vaccination. Case counting started at Day 1.
14.6 Overall Population Impact

The subject characteristics (e.g. lifetime sex partners, geographic distribution of the subjects)
influence the HPV prevalence of the population and therefore the population benefit can vary widely.
The overall efficacy of GARDASIL will vary with the baseline prevalence of HPV infection and disease,
the incidence of infections against which GARDASIL has shown protection, and those infections against
which GARDASIL has not been shown to protect.
The efficacy of GARDASIL for HPV types not included in the vaccine (i.e., cross-protective efficacy) is
a component of the overall impact of the vaccine on rates of disease caused by HPV. Cross-protective
20
Page 195 of 637

efficacy was not demonstrated against disease caused by non-vaccine HPV types in the combined
database of the Study 3 and Study 4 trials.
GARDASIL does not protect against genital disease not related to HPV. One woman who received
GARDASIL in Study 3 developed an external genital well-differentiated squamous cell carcinoma at
Month 24. No HPV DNA was detected in the lesion or in any other samples taken throughout the study.
In 18,150 girls and women enrolled in Study 2, Study 3, and Study 4, GARDASIL reduced definitive
cervical therapy procedures by 23.9% (95% Cl: 15.2%, 31.7%).
14.7 Studies in Women 27 through 45 Years of Age
Study 6 evaluated efficacy in 3253 women 27 through 45 years of age based on a combined endpoint
of HPV 6-, 11-, 16- or 18-related persistent infection, genital warts, vulvar and vaginal dysplastic lesions
of any grade, CIN of any grade, AIS, and cervical cancer. These women were randomized 1:1 to receive
either GARDASIL or AAHS control. The efficacy for the combined endpoint was driven primarily by
prevention of persistent infection. There was no statistically significant efficacy demonstrated for CIN 2/3,
AIS, or cervical cancer. In post hoc analyses conducted to assess the impact of GARDASIL on the
individual components of the combined endpoint, the results in the population of women na"ive to the
relevant HPV type at baseline were as follows: prevention of HPV 6-, 11-, 16- or 18-related persistent
infection (80.5% [95% Cl: 68.3, 88.6]), prevention of HPV 6-, 11-, 16- or 18-related CIN (any grade)
(85.8% [95% Cl: 52.4, 97.3]), and prevention of HPV 6-, 11-, 16- or 18-related genital warts (87.6% [95%
Cl: 7.3, 99.7]).
Efficacy for disease endpoints was diminished in a population impact assessment of women who were
vaccinated regardless of baseline HPV status (full analysis set). In the full analysis set (FAS), efficacy
was not demonstrated for the following endpoints: prevention of HPV 16- and 18-related CIN 2/3, AIS, or
cervical cancer and prevention of HPV 6- and 11-related condyloma. No efficacy was demonstrated
against CIN 2/3, AIS, or cervical cancer in the general population irrespective of HPV type (FAS any type
analysis).
14.8 lmmunogenicity
Assays to Measure Immune Response
The minimum anti-HPV titer that confers protective efficacy has not been determined.
Because there were few disease cases in individuals na"ive (PCR negative and seronegative) to
vaccine HPV types at baseline in the group that received GARDASIL, it has not been possible to
establish minimum anti-HPV 6, anti-HPV 11, anti-HPV 16, and anti-HPV 18 antibody levels that protect
against clinical disease caused by HPV 6, 11, 16, and/or 18.
The immunogenicity of GARDASIL was assessed in 23,951 9- through 45-year-old girls and women
= =
(GARDASIL N 12,634; AAHS control or saline placebo N 11,317) and 5417 9- through 26-year-old
=
boys and men (GARDASIL N 3109; AAHS control or saline placebo N 2308). =
Type-specific immunoassays with type-specific standards were used to assess immunogenicity to
each vaccine HPV type. These assays measured antibodies against neutralizing epitopes for each HPV
type. The scales for these assays are unique to each HPV type; thus, comparisons across types and to
other assays are not appropriate.
Immune Response to GARDASIL
The primary immunogenicity analyses were conducted in a per-protocol immunogenicity (PPI)
population. This population consisted of individuals who were seronegative and PCR negative to the
relevant HPV type(s) at enrollment, remained HPV PCR negative to the relevant HPV type(s) through 1
month postdose 3 (Month 7), received all 3 vaccinations, and did not deviate from the study protocol in
ways that could interfere with the effects of the vaccine.
lmmunogenicity was measured by (1) the percentage of individuals who were seropositive for
antibodies against the relevant vaccine HPV type, and (2) the Geometric Mean Titer (GMT).
In clinical studies in 16- through 26-year-old girls and women, 99.8%, 99.8%, 99.8%, and 99.4% who
received GARDASIL became anti-HPV 6, anti-HPV 11, anti-HPV 16, and anti-HPV 18 seropositive,
respectively, by 1 month postdose 3 across all age groups tested.
In clinical studies in 27- through 45-year-old women, 98.2%, 97.9%, 98.6%, and 97.1% who received
GARDASIL became anti-HPV 6, anti-HPV 11, anti-HPV 16, and anti-HPV 18 seropositive, respectively,
by 1 month postdose 3 across all age groups tested.
21
Page 196 of 637

In clinical studies in 16- through 26-year-old boys and men, 98.9%, 99.2%, 98.8%, and 97.4% who
received GARDASIL became anti-HPV 6, anti-HPV 11, anti-HPV 16, and anti-HPV 18 seropositive,
respectively, by 1 month postdose 3 across all age groups tested.
Across all populations, anti-HPV 6, anti-HPV 11, anti-HPV 16, and anti-HPV 18 GMTs peaked at
Month 7 (Table 18 and Table 19). GMTs declined through Month 24 and then stabilized through Month 36
at levels above baseline. Tables 20 and 21 display the persistence of anti-HPV cLIA geometric mean
titers by gender and age group. The duration of immunity following a complete schedule of immunization
with GARDASIL has not been established.
Table 18: Summary of Month 7 Anti-HPV cLIA Geometric Mean Titers in the PPI* Pooulation of Girls and Women
% Seropositive GMT
Population Nt ni (95% Cl) (95% Cl) mMU§/ml
Anti-HPV 6
9- throuah 15--vear-old airls 1122 917 99.9 /99.4, 100.0) 929.2 (874.6, 987.3)
16- throuqh 26-year-old qirls and women 9859 3329 99.8 (99.6, 99.9) 545.0 (530.1, 560.4)
27-through 34-vear-old women 667 439 98.4 (96.7, 99.4) 435.6 (393.4, 482.4)
35--throuah 45-vear-old women 957 644 98.1 /96.8, 99.0) 397.3 /365.2, 432.2)
Anti-HPV 11
9- throuqh 15-vear-old airls 1122 917 99.9 (99.4, 100.0) 1304.6 (1224.7, 1389.7)
16- throuah 26-vear-old airls and women 9859 3353 99.8 (99.5, 99.9) 748.9 (726.0, 772.6)
27- throuah 34-vear-old women 667 439 98.2 /96.4, 99.2) 577.9 /523.8, 637.5)
35- throuqh 45-year-old women 957 644 97.7 (96.2, 98.7) 512.8 (472.9, 556.1)
Anti-HPV 16
9- throuah 15-vear-old airls 1122 915 99.9 (99.4, 100.0\ 4918.5 (4556.6, 5309.1)
16- thrquqh 26-vear-old qirls and women 9859 3249 99.8 (99.6, 100.0) 2409.2 (2309.0, 2513.8)
27- throuqh 34-year-old women 667 435 99.3 (98.0, 99.9) 2342.5 (2119.1, 2589.6)
35- throuah 45-vear-old women 957 657 98.2 (96.8, 99.1) 2129.5 (1962.7, 2310.5)
Anti-HPV 18
9- throuQh 15-vear-old qirls 1122 922 99.8 (99.2, 100.0) 1042.6 (967.6, 1123.3)
16- throuqh 26-year-old girls and women 9859 3566 99.4 (99.1, 99.7) 475.2 (458.8, 492.1)
27- throuah 34-vear-old women 667 501 98.0 (96.4, 99.0) 385.8 (347.6, 428.1)
35- throuah 45-vear-old women 957 722 96.4 /94.8, 97.6) 324.6 /297.6, 354.0\
..
*The PPI population consisted of ind1v1dualswho received all 3 vaccinations w1th1npre-defined day ranges, did not have
major deviations from the study protocol, met predefined criteria for the interval between the Month 6 and Month 7 visit,
and were naTve (PCR negative and seronegative) to the relevant HPV type(s) (types 6, 11, 16, and 18) prior to dose 1
and through 1 month Postdose 3 (Month 7).
tNumber of individuals randomized to the respective vaccination group who received at least 1 injection.
~Number of individuals contributing to the analysis.
cLIA = Competitive Luminex Immunoassay
Cl= Confidence Interval
GMT= Geometric Mean Titers
§mMU = milli-Merck Units
22
Page 197 of 637

Table 19: Summa of Month 7 Anti-HPV cLIA Geometric Mean Titers in the PPI* Po ulation of Bo sand Men
GMT
Po ulation Nt n* 95% Cl mMU§/ml
*The PPI population consisted of individuals who received all 3 vaccinations within pre-defined day ranges, did not have
major deviations from the study protocol, met predefined criteria for the interval between the Month 6 and Month 7 visit,
and were naTve (PCR negative and seronegative) to the relevant HPV type(s) (types 6, 11, 16, and 18) prior to dose 1 and
through 1 month Postdose 3 (Month 7).
tNumber of individuals randomized to the respective vaccination group who received at least 1 injection.
tNumber of individuals contributing to the analysis.
GMT = Geometric Mean Titers
§mMU = milli-Merck Units
23
Page 198 of 637

Ta b le 20: Persistence of Anti-HPV cl IA Geometric Mean Titers in 9- Throu~h 45-Year-OI dG iris and Women
16- to 26-Year-Old Girls and 27- to 34-Year-Old 35- to 45-Year-Old
9-to 15-Year-Old Girls
Women Women Women
(N* = 1122)
Assay (cLIA)/ (N* = 9859) (N* = 667) (N* = 957)
Time Point GMT GMT GMT GMT
nt (95% Cl) nt (95% Cl) nt (95%CI) nt (95%CI)
mMU*/ml mMU*/ml mMU*/mL mMU*/ml
Anti-HPV 6
Month 07 917 929.2 3329 545.0 439 435.6 644 397.3
(874.6, 987.3) (530.1, 560.4) (393.4, 482.4) (365.2, 432.2)
Month 24 214 156.1 2788 109.1 421 70.7 628 69.3
(135.6, 179.6) (105.2, 113.1) (63.8, 78.5) (63.7, 75.4)
Month 36' 356 129.4 - - 399 79.5 618 81.1
(115.6, 144.8) (72.0, 87.7) (75.0, 87.8)
Month 48 11 - - 2514 73.8 391 58.8 616 62.0
(70.9, 76.8) (52.9, 65.3) (57.0, 67.5)
Anti-HPV 11
Month 07 917 1304.6 3353 748.9 439 577.9 644 512.8
(1224.7, (726.0, 772.6) (523.8, 637.5) (472.9, 556.1)
1389.7)
Month 24 214 218.0 2817 137.1 421 79.3 628 73.4
{188.3, 252.4) {132.1, 142.3) {71.5, 87.8) (67.4, 79.8)
Month 36" 356 148.0 - - 399 81.8 618 77.4
(131.1, 167.1) (74.3, 90.1) {71.6, 83.6)
Month 48 11 - - 2538 89.4 391 67.4 616 62.7
(85.9, 93.1) (60.9, 74.7) (57.8, 68.0)
Anti-HPV 16
Month 07 915 4918.5 3249 2409.2 435 2342.5 657 2129.5
(4556.6, (2309.0, 2513.8) (2119.1, 2589.6) (1962.7, 2310.5)
5309.1)
Month 24 211 944.2 2721 442.6 416 285.9 642 271.4
{804.4, 1108.3) (425.0, 460.9) (254.4, 321.2) {247.1, 298.1)
Month 36' 353 642.2 - - 399 291.5 631 276.7
(562.8, 732.8) (262.5, 323.8) (254.5, 300.8)
Month 48 11 - - 2474 326.2 394 211.8 628 192.8
(311.8, 341.3) {189.5, 236.8) (176.5, 210.6)
Anti-HPV 18
Month 07 922 1042.6 3566 475.2 501 385.8 722 324.6
(967.6, 1123.3) {458.8, 492.1) {347.6, 428.1) (297.6, 354.0)
Month 24 214 137.7 3002 50.8 478 31.8 705 26.0
{114.8, 165.1) (48.2, 53.5) {28.1, 36.0) (23.5, 28.8)
Month 36" 357 87.0 - - 453 32.1 689 27.0
(74.8, 101.2) (28.5, 36.3) (24.5, 29.8)
Month 4811 - - 2710 33.2 444 25.2 688 21.2
(31.5, 35.0)
. . .. (22.3, 28.5) (19.2, 23.4)
*N = Number of 1nd1v1duals randomized 1nthe respective group who received at least 1 1nJect1on.
t n = Number of individuals in the indicated immunogenicity population.
:l:mMU= milli-Merck Units
§Month 37 for 9- to 15-year-old girls. No serology samples were collected at this time point for 16- to 26-year-old girls and women.
~Month 48/End-of-study visits for 16- to 26-year-old girls and women were generally scheduled earlier than Month 48. Mean visit
timing was Month 44. The studies in 9- to 15-year-old girls were planned to end prior to 48 months and therefore no serology
samples were collected.
24
Page 199 of 637

. t ence ofA n t"I• HPV C LIAG eome t"M
T a bl e 21 : Pers1s -
nc ean rt1ers m "9Th roug1h 26-Year- Old Boys an dM en
9- to 15-Year-Old Boys 16-to 26-Year-Old Boys and Men
(N* = 1072) (N* = 2026)
Assay (cLIA)/ Time Point
nt GMT nt GMT
(95% Cl) mMU*/ml (95% Cl) mMU*/ml
Anti-HPV 6
Month 07 884 1037.5 1094 447.2
(963.5, 1117.3) (418.4, 477.9)
Month 24 323 134.1 907 80.3
(119.5, 150.5) (74.9, 86.0)
Month 36' 342 126.6 654 72.4
(111.9, 143.2) (68.0, 77.2)
Month 48 11 - - - -
Anti-HPV 11
Month 07 885 1386.8 1094 624.5
(1298.5, 1481.0) (588.6, 662.5)
Month 24 324 188.5 907 94.6
(168.4, 211.1) (88.4, 101.2)
Month 36' 342 148.8 654 80.3
(131.1, 169.0) (75.7, 85.2)
Month 48 11 - - - -
Anti-HPV 16
Month 07 882 6056.5 1137 2401.5
(5601.4, 6548.6) (2241.8, 2572.6)
Month 24 322 938.2 938 347.7
(825.0, 1067.0) (322.5, 374.9)
Month 36' 341 708.8 672 306.7
(613.9, 818.3) (287.5, 327.1)
Month 48 11
- - - -
Anti-HPV 18
Month 07 887 1357.4 1176 402.6
(1249.4, 1474.7) (374.6, 432.6)
Month 24 324 131.9 967 38.7
(112.1, 155.3) (35.2, 42.5)
Month 36' 343 113.0 690 33.4
(94.7, 135.0) (30.9, 36.1)
Month 48 11 - - - -
*N = Number of individuals randomized in the respective group who received at least 1 injection.
tn = Number of individuals in the indicated immunogenicity population.
*mMU = milli-Merck Units
§Month 36 time point for 16- to 26-year-old boys and men; Month 37 for 9- to 15-year-old boys.
1The studies in 9-to 15-year-old boys and girls and 16- to 26-year-old boys and men were planned to end prior to 48 months and
therefore no serology samples were collected.
Tables 18 and 19 display the Month 7 immunogenicity data for girls and women and boys and men.
Anti-HPV responses 1 month postdose 3 among 9- through 15-year-old adolescent girls were non-inferior
to anti-HPV responses in 16- through 26-year-old girls and women in the combined database of
immunogenicity studies for GARDASIL. Anti-HPV responses 1 month postdose 3 among 9- through 15-
year-old adolescent boys were non-inferior to anti-HPV responses in 16- through 26-year-old boys and
men in Study 5.
On the basis of this immunogenicity bridging, the efficacy of GARDASIL in 9- through 15-year-old
adolescent girls and boys is inferred.
GMT Response to Variation in Dosing Regimen in 18- Through 26-Year-Old Women
Girls and women evaluated in the PPE population of clinical studies received all 3 vaccinations within
1 year of enrollment. An analysis of immune response data suggests that flexibility of ±1 month for Dose
2 (i.e., Month 1 to Month 3 in the vaccination regimen) and flexibility of ±2 months for Dose 3 (i.e., Month
4 to Month 8 in the vaccination regimen) do not impact the immune responses to GARDASIL.
Duration of the Immune Response to GARDASIL
The duration of immunity following a complete schedule of immunization with GARDASIL has not
been established. The peak anti-HPV GMTs for HPV types 6, 11, 16, and 18 occurred at Month 7. Anti-
25
Page 200 of 637

HPV GMTs for HPV types 6, 11, 16, and 18 were similar between measurements at Month 24 and Month
60 in Study 2.
14.9 Long-Term Follow-Up Studies
The protection of GARDASIL against HPV-related disease continues to be studied over time in
populations including adolescents (boys and girls) and women who were enrolled in the Phase 3 studies.
Persistence of Effectiveness
An extension of Study 4 used national healthcare registries in Denmark, Iceland, Norway, and
Sweden to monitor endpoint cases of HPV 6-, 11-, 16-, or 18-related CIN (any grade), AIS, cervical
cancer, vulvar cancer, or vaginal cancer among 2,650 girls and women 16 through 23 years of age at
enrollment who were randomized to vaccination with GARDASIL and consented to be followed in the
extension study. An interim analysis of the per-protocol effectiveness population included 1,902 subjects
who completed the GARDASIL vaccination series within one year, were naTveto the relevant HPV type
through 1 month postdose 3, had no protocol violations, and had follow-up data available. The median
follow-up from initial vaccination was 6.7 years with a range of 2.8 to 8.4 years. No cases of HPV 6-, 11-,
16-, or 18-related CIN (any grade), AIS, cervical cancer, vulvar cancer, or vaginal cancer were observed
over a total of 5,765 person-years at risk.
An extension of a Phase 3 study (Study 7) in which 614 girls and 565 boys 9 through 15 years of age
at enrollment were randomized to vaccination with GARDASIL actively followed subjects for endpoint
cases of HPV 6-, 11-, 16-, or 18-related persistent infection, CIN (any grade), AIS, VIN, ValN, cervical
cancer, vulvar cancer, vaginal cancer, and genital lesions from the initiation of sexual activity or age 16
onwards. An interim analysis of the per-protocol effectiveness population included 246 girls and 168 boys
who completed the GARDASIL vaccination series within one year, were seronegative to the relevant HPV
type at initiation of the vaccination series, and had not initiated sexual activity prior to receiving the third
dose of GARDASIL. The median follow-up, from the first dose of vaccine, was 7.2 years with a range of
0.5 to 8.5 years. No cases of persistent infection of at least 12 months' duration and no cases of HPV 6-,
11-, 16-, or 18-related CIN (any grade), AIS, VIN, ValN, cervical cancer, vulvar cancer, vaginal cancer, or
genital lesions were observed over a total 1,105 person-years at risk. There were 4 cases of HPV 6-, 11-,
16-, or 18-related persistent infection of at least 6 months' duration, including 3 cases related to HPV 16
and 1 case related to HPV 6, none of which persisted to 12 months' duration.
Persistence of the Immune Response
The interim reports of the two extension studies described above included analyses of type-specific
anti-HPV antibody titers at 9 years postdose 1 for girls and women 16 through 23 years of age at
enrollment (range of 1,178 to 1,331 subjects with evaluable data across HPV types) and at 8 years
postdose 1 for boys and girls 9 through 15 years of age at enrollment (range of 436 to 440 subjects with
evaluable data across HPV types). Anti-HPV 6, 11, 16, and 18 GMTs as measured by cLIA were
decreased compared with corresponding values at earlier time points, but the proportions of seropositive
subjects ranged from 88.4% to 94.4% for anti-HPV 6, from 89.1% to 95.5% for anti-HPV 11, from 96.8%
to 99.1% for anti-HPV 16, and from 60.0% to 64.1 % for anti-HPV 18.
14.10 Studies with RECOMBIVAX HB [hepatitis B vaccine (recombinant)]
The safety and immunogenicity of co-administration of GARDASIL with RECOMBIVAX HB [hepatitis B
vaccine (recombinant)] (same visit, injections at separate sites) were evaluated in a randomized, double-
blind, study of 1871 women aged 16 through 24 years at enrollment. The race distribution of the girls and
women in the clinical trial was as follows: 61.6% White; 1.6% Hispanic (Black and White); 23.8% Other;
11.9% Black; 0.8% Asian; and 0.3% American Indian.
Subjects either received GARDASIL and RECOMBIVAX HB (n = 466), GARDASIL and
RECOMBIVAX HS-matched placebo (n = 468), RECOMBIVAX HB and GARDASIL-matched placebo (n
=467) or RECOMBIVAX-matched placebo and GARDASIL-matched placebo (n =470) at Day 1, Month 2
and Month 6. lmmunogenicity was assessed for all vaccines 1 month post completion of the vaccination
series.
Concomitant administration of GARDASIL with RECOMBIVAX HB [hepatitis B vaccine (recombinant)]
did not interfere with the antibody response to any of the vaccine antigens when GARDASIL was given
concomitantly with RECOMBIVAX HB or separately.
26
Page 201 of 637

14.11 Studies with Menactra [Meningococcal (Groups A, C, Y and W-135) Polysaccharide
Diphtheria Toxoid Conjugate Vaccine] and Adacel [Tetanus Toxoid, Reduced Diphtheria Toxoid
and Acellular Pertussis Vaccine Adsorbed (Tdap)]
The safety and immunogenicity of co-administration of GARDASIL with Menactra [Meningococcal
(Groups A, C, Y and W-135) Polysaccharide Diphtheria Toxoid Conjugate Vaccine] and Adacel [Tetanus
Toxoid, Reduced Diphtheria Toxoid and Acellular Pertussis Vaccine Adsorbed (Tdap)] (same visit,
injections at separate sites) were evaluated in an open-labeled, randomized, controlled study of 1040
boys and girls 11 through 17 years of age at enrollment. The race distribution of the subjects in the
clinical trial was as follows: 77.7% White; 6.8% Hispanic (Black and White); 1.4% Multi-racial; 12.3%
Black; 1.2% Asian; 0.2% Indian; and 0.4% American Indian.
One group received GARDASIL in one limb and both Menactra and Adacel, as separate injections, in
the opposite limb concomitantly on Day 1 (n = 517). The second group received the first dose of
GARDASIL on Day 1 in one limb then Menactra and Adacel, as separate injections, at Month 1 in the
opposite limb (n = 523). Subjects in both vaccination groups received the second dose of GARDASIL at
Month 2 and the third dose at Month 6. lmmunogenicity was assessed for all vaccines 1 month post
completion of the vaccination series (1 dose for Menactra and Adacel and 3 doses for GARDASIL).
Concomitant administration of GARDASIL with Menactra [Meningococcal (Groups A, C, Y and W-135)
Polysaccharide Diphtheria Toxoid Conjugate Vaccine] and Adacel [Tetanus Toxoid, Reduced Diphtheria
Toxoid and Acellular Pertussis Vaccine Adsorbed (Tdap)] did not interfere with the antibody response to
any of the vaccine antigens when GARDASIL was given concomitantly with Menactra and Adacel or
separately.
16 HOW SUPPLIED/STORAGE AND HANDLING

All presentations for GARDASIL contain a suspension of 120 mcg L1 protein from HPV types 6, 11,
16, and 18 in a 0.5-mL dose. GARDASIL is supplied in vials and syringes.
Carton of one 0.5-mL single-dose vial. NDC 0006-4045-00.
Carton of ten 0.5-mL single-dose vials. NDC 0006-4045-41.
Carton of six 0.5-mL single-dose prefilled Luer-Lok® syringes with tip caps. NDC 0006-4109-09.
Carton of ten 0.5-mL single-dose prefilled Luer-Lok® syringes with tip caps. NDC 0006-4109-02.
Store refrigerated at 2 to 8°C (36 to 46°F). Do not freeze. Protect from light.
GARDASIL should be administered as soon as possible after being removed from refrigeration.
GARDASIL can be out of refrigeration (at temperatures at or below 25°C/77°F), for a total time of not
more than 72 hours.
17 PATIENT COUNSELING INFORMATION

Advise the patient to read the FDA-approved patient labeling (Patient Information).
Inform the patient, parent, or guardian:

• Vaccination does not eliminate the necessity for women to continue to undergo recommended
cervical cancer screening. Women who receive GARDASIL should continue to undergo cervical
cancer screening per standard of care.
• Recipients of GARDASIL should not discontinue anal cancer screening if it has been
recommended by a health care provider.
• GARDASIL has not been demonstrated to provide protection against disease from vaccine and
non-vaccine HPV types to which a person has previously been exposed through sexual activity.
• Since syncope has been reported following vaccination sometimes resulting in falling with injury,
observation for 15 minutes after administration is recommended.
• Vaccine information is required to be given with each vaccination to the patient, parent, or
guardian.
• Information regarding benefits and risks associated with vaccination.
• GARDASIL is not recommended for use in pregnant women.
• Importance of completing the immunization series unless contraindicated.
• Report any adverse reactions to their health care provider.
27
Page 202 of 637

Manuf.andDist.by: MerckSharp& DahmeCorp.,a subsidiaryof
"".,,,,.
MERCK&CO.,INC.,
WhitehouseStation,NJ 08889,USA
For patent information: www.merck.com/product/patent/home.html
The trademarks depicted herein are owned by their respective companies.
Copyright© 2006, 2009, 2010, 2011 Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc.
All rights reserved.
uspi-v501-i-1504r021
Printed in USA
28
Page 203 of 637

9883616
USPPI
Patient Information about
GARDASIL ® (pronounced "gard-Ah-sill")
Generic name: [Human Papillomavirus Quadrivalent (Types 6, 11, 16, and 18) Vaccine,
Recombinant]
1
Read this information with care before getting GARDAS1L . You (the person getting GARDASIL) will need
3 doses of the vaccine. It is important to read this leaflet when you get each dose. This leaflet does not
take the place of talking with your health care provider about GARDASIL.
What is GARDASIL?
GARDASIL is a vaccine (injection/shot) that is used for girls and women 9 through 26 years of age to help
protect against the following diseases caused by Human Papillomavirus (HPV):
• Cervical cancer
• Vulvar and vaginal cancers
• Anal cancer
• Genital warts
• Precancerous cervical, vaginal, vulvar, and anal lesions
GARDASIL is used for boys and men 9 through 26 years of age to help protect against the following
diseases caused by HPV:
• Anal cancer
• Genital warts
• Precancerous anal lesions
o The diseases listed above have many causes, and GARDASIL only protects against diseases
caused by certain kinds of HPV (called Type 6, Type 11, Type 16, and Type 18). Most of the
time, these 4 types of HPV are responsible for the diseases listed above.
o GARDASIL cannot protect you from a disease that is caused by other types of HPV, other
viruses, or bacteria.
o GARDASIL does not treat HPV infection.
o You cannot get HPV or any of the above diseases from GARDASIL.
What important information about GARDASIL should I know?

• You should continue to get routine cervical cancer screening.
• GARDASIL may not fully protect everyone who gets the vaccine.
• GARDASIL will not protect against HPV types that you already have.
Who should not get GARDASIL?

You should not get GARDASIL if you have, or have had:
• an allergic reaction after getting a dose of GARDASIL.
• a severe allergic reaction to yeast, amorphous aluminum hydroxyphosphate sulfate, polysorbate
80.
What should I tell my health care provider before getting GARDASIL?

Tell your health care provider if you:
• are pregnant or planning to get pregnant. GARDASIL is not recommended for use in pregnant
women.
• have immune problems, like HIV infection, cancer, or you take medicines that affect your immune
system.
• have a fever over 100°F (37 .8°C).
• had an allergic reaction to another dose of GARDASIL.
1
Registered trademark of Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc.
Copyright© 2006, 2009 Merck Sharp & Doh me Corp., a subsidiary of Merck & Co., Inc.
All rights reserved
Page 204 of 637

GARDASIL®
[Human Papillomavirus Quadrivalent (Types 6, 11, 16, and 18) Vaccine, Recombinant] 9883616
• take any medicines, even those you can buy over the counter.
Your health care provider will help decide if you should get the vaccine.
How is GARDASIL given?

GARDASIL is a shot that is usually given in the arm muscle. You will need 3 shots given on the following
schedule:
• Dose 1: at a date you and your health care provider choose.
• Dose 2: 2 months after Dose 1.
• Dose 3: 6 months after Dose 1.
Fainting can happen after getting GARDASIL. Sometimes people who faint can fall and hurt themselves.
For this reason, your health care provider may ask you to sit or lie down for 15 minutes after you get
GARDASIL. Some people who faint might shake or become stiff. This may require evaluation or
treatment by your health care provider.
Make sure that you get all 3 doses on time so that you get the best protection. If you miss a dose, talk to
your health care provider.
Can other vaccines and medications be given at the same time as GARDASIL?
GARDASIL can be given at the same time as RECOMBIVAX HB®1 [hepatitis B vaccine (recombinant)] or
Menactra [Meningococcal (Groups A, C, Y and W-135) Polysaccharide Diphtheria Toxoid Conjugate
Vaccine] and Adacel [Tetanus Toxoid, Reduced Diphtheria Toxoid and Acellular Pertussis Vaccine
Adsorbed (Tdap)].
What are the possible side effects of GARDASIL?
The most common side effects with GARDASIL are:

• pain, swelling, itching, bruising, and redness at the injection site
• headache
• fever
• nausea
• dizziness
• vomiting
• fainting
There was no increase in side effects when GARDASIL was given at the same time as RECOMBIVAX HB
[hepatitis B vaccine (recombinant)].
There was more injection-site swelling at the injection site for GARDASIL when GARDASIL was given at
the same time as Menactra [Meningococcal (Groups A, C, Y and W-135) Polysaccharide Diphtheria Toxoid
Conjugate Vaccine] and Adacel [Tetanus Toxoid, Reduced Diphtheria Toxoid and Acellular Pertussis
Vaccine Adsorbed (Tdap)].
Tell your health care provider if you have any of the following problems because these may be signs of an
allergic reaction:
• difficulty breathing
• wheezing (bronchospasm)
• hives
• rash
Page 205 of 637

GARDASIL®
[Human Papillomavirus Quadrivalent (Types 6, 11, 16, and 18) Vaccine, Recombinant] 9883616
Tell your health care provider if you have:

• swollen glands (neck, armpit, or groin)
• joint pain
• unusual tiredness, weakness, or confusion
• chills
• generally feeling unwell
• leg pain
• shortness of breath
• chest pain
• aching muscles
• muscle weakness
• seizure
• bad stomach ache
• bleeding or bruising more easily than normal
• skin infection
Contact your health care provider right away if you get any symptoms that concern you, even several
months after getting the vaccine.
For a more complete list of side effects, ask your health care provider.
What are the ingredients in GARDASIL?

The ingredients are proteins of HPV Types 6, 11, 16, and 18, amorphous aluminum hydro:xyphosphate
sulfate, yeast protein, sodium chloride, L-histidine, polysorbate 80, sodium borate, and water for injection.
This leaflet is a summary of information about GARDASIL. If you would like more information, please talk
to your health care provider or visit www.gardasil.com.
Manufactured and Distributed by: Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc.
Whitehouse Station, NJ 08889, USA
Issued April 2011
Page 206 of 637

Page 207 of 637
Product Case Number: (Located on bottom of Cover Letter) _______ _
Merck Global Medical Information responds to unsolicited requests from health care professionals for information regarding Merck prescription
products. Please take a moment to complete the following survey. The information you provide will be used to improve our service.
In your survey response, consider that there may not always be information available for a particular inquiry.
Quality of Response Very Satisfied Satisfied Neutral Dissatisfied Very Dissatisfied Not Applicable
Inquiry Addressed D D D D D D
Timeliness D D D D D D
Length D D D D D D
Clarity D D D D D D
Objectivity D D D D D D
Overall Satisfaction D D D D D D
Additional Feedback
(Use other side if necessary)
Thank you for your time and response.

D I would like to be contactedregardingmy survey response.
Name: Phonenumber
-- Fold Here First --
NO POSTAGE
NECESSARY
IF MAILED
INTHE
UNITED STATES
BUSINESS REPLY MAIL

FIRST -CLASS MAIL PERMIT NO. 51 WEST POINT PA
POSTAGE WILL BE PAID BY ADDRESSEE
UG1B-50
GMA-GLOBAL MEDICAL INFORMATION
MERCK & CO INC
P.O. B0X4
WEST POINT, PA 19486-9970
I,,, I II, I,,, I,, I I,, I,, I I,, I, I,, I, I,, I,,, I II,,, I,,, I I
Please Do Not Staple, Tape Only!
Page 208 of 637

Executive Director
U.S.A.
April 24, 2017
Dear Dr. Saah,
This week we received Dr. Richard Gersh’s mailed letters dated April 10, in response to our
eight questions sent March 14.
We are grateful for your assistance towards our systematic review (“Benefits and harms of
human papillomavirus vaccines: systematic review of industry clinical study reports and non-
industry published and unpublished reports.”1) We aim to maintain a balanced approach to the
evidence and to consider all factors when assessing the trials and follow-up studies of Gardasil
and Gardasil 9.
Unfortunately, after reading Dr. Richard Gersh’s replies to our eight questions we still are
unclear as to the answers to seven of them.
We would be grateful, if you would have another look at the questions and provide answers in
a more direct manner. Please see the following explanations for where clarifications are
needed:
1. Why did Merck not use an inert placebo vaccine in any of the Gardasil 4 clinical trials?
Instead of an inert placebo, vaccines containing the adjuvant amorphous aluminium hydroxide
sulphate (AAHS), other vaccines and carrier solution comparators were used in the trials. Dr.
Gersh did not explain why Merck made the decision to use these non-inert placebo vaccines as
comparators. This information is not found in the Prescribing Information that he referred us
to. To our knowledge, only trial V503-006 from the Gardasil 9 study program used pure saline
as a comparator. We assume that the decision to use non-inert comparators for the trial
designs was thoroughly considered, however the rationale for using active comparators
remains unclear to us.
1
Lars Jørgensen, Peter Christian Gøtzsche, Tom Jefferson. Benefits and harms of human
published and unpublished reports. PROSPERO 2017:CRD42017056093 Available from
http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42017056093
Page 209 of 637

2. Was AAHS ever approved by regulators before being used as an adjuvant in vaccines (not
necessarily limited to Gardasil) on the market?
We are not sure whether regulators ever approved AAHS, and if so, based upon what data.
The statement that aluminum has been in use as an adjuvant for decades does not answer the
question specifically about AAHS that to our understanding is a highly complex adjuvant.
3. What are the clinical effects of the AAHS-containing control solution versus inert placebo?
Given the central role of AAHS in triggering and sustaining high antibody responses (as
documented in Dr. Gersh’s reply), we wonder what safety studies Merck has undertaken of
AAHS and where we can find them?
4. Are the clinical effects of AAHS consistent and predictable?

A 2013 Merck patent application for AAHS describes “intra-batch” and “batch-to-batch”
variability of the concentration of AAHS in its manufacturing process. From the patent text it
appears that AAHS may be chemically inconsistent in form. Can you confirm your
understanding of this point and its relevance to the issue of whether the clinical effects of AAHS
are consistent and predictable? This question has not been addressed in Dr. Gersh’s reply to us,
despite the evidence of variability in the manufacturing process.
5. When did Merck become aware that the adjuvant in its adjuvanted vaccines was in fact
AAHS?
We note that a number of vaccine descriptions refer to AAHS as “Amorphous Aluminum
Hydroxyphosphate sulfate (previously called aluminum hydroxide).” However, the two
(Amorphous Aluminum Hydroxyphosphate sulfate and aluminum hydroxide) are not
synonymous. Does Merck truly believe aluminum hydroxide and AAHS are identical and the
only difference is a name change? If so, at what point did Merck start using the new name of
AAHS and why? However if on the other hand Merck believes, as we do, that aluminum
hydroxide and AAHS are different, when did Merck start using AAHS in its products? The
description of three types of aluminium-based adjuvants in Dr. Gersh’s letter does not answer
our question. There appears to have been no consistency in terminology.
6. Why does Merck describe the comparator in trial V501-018 as “normal saline” when it
contains more than water and sodium chloride?
Dr. Gersh’s letter does not answer our question and repeatedly refers to the
control/comparator used in study V501-018 as “normal saline”. We know that Merck calls this
“normal saline,” but as we pointed out in our previous letter, this so-called “normal saline”
comparator was a carrier solution, i.e., a control dose of 0.5 milliliters containing 9.56
milligrams of sodium chloride, 0.78 milligrams of L-histidine, 50 micrograms of polysorbate 80,
35 micrograms of sodium borate, <7 micrograms of yeast protein, and water [see PDF. page 64
in the clinical study report of V501-018]). We are asking why Merck refers to this carrier
solution as “normal saline.” Is there any medical dictionary that defines normal saline as
containing these ingredients?
Page 210 of 637

7. Does Gardasil contain DNA/RNA and, if so, what is the clinical significance of this?
Thank you for your clear answer to this question.
8. Are there any long-term Gardasil studies with cancer (i.e., cervical, anal, oropharyngeal, and
penile cancer) as a specific outcome and, if so, what are the studies identification numbers?
We apologize if this question was unclear or seemed very broad. We interpret Dr. Gersh’s
seven page response to this question as (1) that there were no cases of HPV related cancer, of
any type, in either experimental or control arms, in Gardasil studies; and (2) that no Gardasil
studies included cancer (of any type) as a pre-specified endpoint/outcome measure of interest.
We would prefer a confirmation of our interpretation, given the clinical importance of this
matter.
We would be grateful for your response by 15 May 2017. And we request you kindly send a
copy of your response by email to help expedite the correspondence.
Thank you for your time.
With best wishes,

Oxford OX2 6GG United Kingdom

Blegdamsvej 9

Blegdamsvej 9

220 Arch Street, Floor 12 Room 01-228
Page 211 of 637

Page 212 of 637
Page 213 of 637
Page 214 of 637
Page 215 of 637
Page 216 of 637
Page 217 of 637
Page 218 of 637
Page 219 of 637
Page 220 of 637
Page 221 of 637
Page 222 of 637
Page 223 of 637
Page 224 of 637
Page 225 of 637
Page 226 of 637
Page 227 of 637
Page 228 of 637
Page 229 of 637
Page 230 of 637
Page 231 of 637
Page 232 of 637
Page 233 of 637
Page 234 of 637
Page 235 of 637
Page 236 of 637
Page 237 of 637
Page 238 of 637
Page 239 of 637
Page 240 of 637
Page 241 of 637
Page 242 of 637
Page 243 of 637
Page 244 of 637
Page 245 of 637
Executive Director
U.S.A.
7 June 2017
Dear Dr. Saah,
1. We emailed you on March 14, 2017 with 8 questions regarding the Gardasil
development program.
2. Merck responded by post on April 10.
3. We emailed again on April 24 and explained that 7 of our 8 questions remained

unanswered.
4. You emailed us on May 15 explained that a response was on its way and to "please
consider this response as our final effort to respond to the enquiries sent".
5. Merck responded by post on May 23, stating that Merck's previous letter answered our
questions. ("In response to your letter of March 14, we provided thorough responses
that included Merck data and references to publicly available information, including
scientific literature and regulatory documents, which pertain to your questions.")
Merck's April 10 letter was lengthy, but the content does not answer 7 of our 8 original
questions, as we explained on April 24. We are alarmed that answers to these important
questions do not appear to be available in publicly available information. This is why we wrote
to you, seeking answers.
We appreciate that Merck may not have the answers to all of our questions, but surely Merck
must have the answer to straightforward questions such as the choice of control intervention in
the Gardasil clinical trials (Question 1: “Why did Merck not use an inert placebo vaccine in any
of the Gardasil 4 clinical trials?”)
Although your May 15 email indicates Merck intends to discontinue correspondence on this
matter, we shall continue to await your answers to our questions. And in the meantime, we
will endeavor to give this matter the attention it deserves.
Page 246 of 637

Thank you for your time.
With best wishes,

Oxford OX2 6GG United Kingdom

Blegdamsvej 9

Blegdamsvej 9

220 Arch Street, Floor 12 Room 01-228
Page 247 of 637

Professor Patrick Vallance
GlaxoSmithKline
980 Great West Road
Brentford, Middlesex,
TW8 9GS
United Kingdom
patrick.5.vallance@gsk.com
Dear Professor Vallance,
We are researchers carrying out a systematic review titled “Benefits and harms of human
published and unpublished reports.”
1
We have recently registered the protocol in PROSPERO and are in the process of gathering clinical
study reports to carry out the review, which covers all the HPV vaccines: Cervarix, Gardasil, Gardasil 9,
and experimental HPV vaccines.
We are contacting you with some inquiries based on our preliminary research and hope you might help us
in answering them.
Questions we have come up with thus far:
1. Why did GlaxoSmithKline not use an inert placebo vaccine in any of the Cervarix clinical trials? Patients,
doctors, and other decision makers are considering Cervarix against the option of Cervarix. As such, we
were surprised to find that none of the randomized Cervarix trials used an inert placebo control. All trials
used a control group that either received a solution containing the vaccine adjuvant aluminium hydroxide
(i.e.,Al[OH]3) or another vaccine (for example, the hepatitis A vaccine, Havrix ).
2. Are there any completed clinical trials of Cervarix which have not been registered and are not listed on
your website?
3. Was the adjuvant system, AS04, ever approved by regulators before being used as an adjuvant in
vaccines (not necessarily limited to Cervarix ) on the market?
We are not sure whether regulators ever approved AS04 and, if so, based upon what data.
4. What are the clinical effects of the AS04 versus inert placebo?
We understand from correspondence with regulators (i.e., the European Medicines Agency, EMA,
Australia’s Therapeutic Goods Administration, TGA, Health Canada and New Zealand’s MedSafe) that
there is no separate registration for adjuvants (including AS04). Thus, we wonder what safety studies
GlaxoSmithKline has undertaken of AS04?
5. Does Cervarix contain DNA/RNA and, if so, what is the clinical significance of this?
According to a document by Japan’s Pharmaceuticals and Medical Devices Agency (PMDA) regulator
there are viral DNA and RNA strands in unknown concentrations in Gardasil. The concentrations were
redacted so we do not know what these concentrations were and we do not know if this finding has any
clinical significance. However, we know that RNA/DNA alone injected intramuscularly into BALB-C mice
activates the immune response. We have not seen any information regarding this issue with respect to
Cervarix, so are asking just in case.
6. Are there any long-term Cervarix studies with cancer (i.e., cervical, anal, oropharyngeal, and penile
cancer) as a specific outcome and, if so, what are the studies identification numbers?
We have been unable to locate any study where actual cancer (i.e., not a surrogate for cancer, such as,
cervical/anal/penile intraepithelial neoplasia [CIN/AIN/PIN] etc.) is reported as the specific outcome. We are
aware that, in 2004, two years before the regulatory approval of the first HPV vaccine Gardasil, the World
Health Organization (WHO) approved a surrogate outcome for cervical cancer (i.e., cervical intraepithelial
neoplasia grade two or more, CIN2+: CIN2, CIN3, adenocarcinoma in situ and cervical cancer) for
2
regulatory approval.
We would be grateful for your response by 1 July 2017. Thank you for or your time,
Page 248 of 637

With best wishes,

Oxford OX2 6GG
United Kingdom

Blegdamsvej 9
2100 Copenhagen Ø,
Denmark
Lars Jørgensen, MD, PhD

Student Nordic Cochrane Centre
Blegdamsvej 9
2100 Copenhagen Ø,
Denmark
Peter Doshi, PhD,

Assistant Professor
Baltimore,
Maryland,
21201 U.S.A.
16 June 2017
1.Lars Jørgensen, Peter Christian Gøtzsche, Tom Jefferson. Benefits and harms of human papillomavirus vaccines:
systematic review of industry clinical study reports and non-industry published and unpublished reports. PROSPERO
2017:CRD42017056093 Available from http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42017056093
2.Pagliusi SR, Teresa Aguado M. Efficacy and other milestones for human papillomavirus vaccine introduction. Vaccine. 2004 Dec
16;23(5):569–78.
Page 249 of 637

Page 250 of 637
GSK responses to Dr. Jefferson July 2017
Question 1
Why did GlaxoSmithKline not use an inert placebo vaccine in any of the Cervarix clinical trials?
Patients, doctors, and other decision makers are considering Cervarix against the option of Cervarix.
As such, we were surprised to find that none of the randomized Cervarix trials used an inert placebo
control. All trials used a control group that either received a solution containing the vaccine adjuvant
aluminium hydroxide (i.e.,Al[OH]3) or another vaccine (for example, the hepatitis A vaccine, Havrix ).
An extensive clinical programme has been undertaken with Cervarix to support its licensure
worldwide and to assess its safety and efficacy profiles. In most randomised, controlled studies of the
development programme, GSK chose as control either aluminium hydroxide or an active vaccine
containing aluminium hydroxide such as Havrix or Engerix B, with well established and documented
safety profiles including reactogenicity. The choice of the comparator was made in accordance with
European Medicines Agency recommendations to use, when possible, alternative vaccines that do
not protect against the disease under study but still provide another potential benefit to vaccinees.1
If, as suggested, an inert placebo (e.g. saline water) was used, it would likely un-blind the studies to
Investigators and/or study participants due to the absence of reactogenicity following injection and
would lead to a bias in the characterization of both vaccine’ safety and efficacy profiles.
All studies were conducted in accordance with ethical principles that have their origins in the
Declaration of Helsinki, the principles of good clinical practice (GCP) and all applicable regulatory
requirements. Protocols were subject to local EC and regulatory authorities notification and/or
approval which was done and /or obtained prior to study start.
1. European Medicines Agency. Note for guidance on clinical evaluation of vaccines 2005
http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003875
.pdf last accessed in July 2017
Page 251 of 637

Question 2
Are there any completed clinical trials of Cervarix which have not been registered and are not listed
on your website?
We analysed web disclosure status on July 2017 of all studies that ever evaluated Cervarix. In scope
are all GSK sponsored and MedImmune initiated studies. The exceptions are discussed below. As an
appendix we provide a complete overview of all studies including individual disclosure status.
GSK initiated studies
Interventional studies
There are in total 70 interventional studies that evaluated Cervarix. Results of 66 studies have been
posted on the GSK website. There are four studies for which results have not been posted:
For one Study analysis has just been completed and results are due in 2017. The Protocol for this
study is posted on the GSK website
• MENACWY-TT-054 https://www.gsk-clinicalstudyregister.com/study/113823?search=study&search_terms=113823#ps
3 studies are ongoing and results are due in 2018/2019. The protocols for these studies are posted
on the GSK website
• HPV-062 EXT:015, https://www.gsk-clinicalstudyregister.com/study/113617?search=study&search_terms=113617#ps
• HPV-066 EXT:015, https://www.gsk-clinicalstudyregister.com/study/113621?search=study&search_terms=113621#ps
• HPV-067 EXT:015 https://www.gsk-clinicalstudyregister.com/study/113618?search=study&search_terms=113618#ps
Out of the 66 studies for which results are posted on the GSK website, 56 studies results are also
disclosed on Clinical Trials.gov.
There are 9 interventional studies that are registered on ClinicalTrials.gov, for which results have
been posted on the GSK website (see links below), but not on ClinicalTrials.gov, as studies were
completed before applicable date of FDA regulation (Sep 2007) or did not fall under the regulation
(e.g. Phase 1 trial):
• HPV-NG-001PRI, https://www.gsk-clinicalstudyregister.com/study/109836?search=study&search_terms=109836#rs
• HPV-001, https://www.gsk-clinicalstudyregister.com/study/580299/001?search=study&search_terms=580299%2F001#rs
• HPV-033, https://www.gsk-clinicalstudyregister.com/study/104951?search=study&search_terms=104951#rs
• HPV-035, https://www.gsk-clinicalstudyregister.com/study/106001?search=study&search_terms=106001#rs
• HPV-TETRA-051, https://www.gsk-clinicalstudyregister.com/study/102115?search=study&search_terms=102115#rs
There is one study for which results have been submitted to ClinicalTrials.gov but are currently under
review. Results are available on GSK website (see link below)
• HPV-019 PRI, https://www.gsk-clinicalstudyregister.com/study/109823?search=study&search_terms=109823#rs
Page 252 of 637

Non-interventional studies
There are in total 11 non-interventional studies that evaluated Cervarix. Results of 9 studies have
been posted on the GSK website. Results of 2 studies have not been posted :
• 1 completed meta-analysis (HPV-094 MA); Protocol is posted on the website. Study results are due in
2018 https://www.gsk-clinicalstudyregister.com/study/207644?search=study&search_terms=207644#ps
• 1 ongoing meta-analysis (HPV-091 MA); Protocol is posted on the website. Study results are due in
2018 https://www.gsk-clinicalstudyregister.com/study/205206?search=study&search_terms=205206#ps
Results of observational studies are generally not in scope of disclosure on ClinicalTrials.gov
MedImmune initiated studies
The first Phase I and II studies in the clinical development of Cervarix were initiated in 1999 by
MedImmune Inc. (Cervarix corresponds to the investigational product MEDI-517). MedImmune
studies comprise the following studies:
The MedImmune study results are all posted on the GSK website (see website links above). HPV-003,
004 and 005 are also registered on ClinicalTrials.gov.
Page 253 of 637

Question 3.
Was the adjuvant system, AS04, ever approved by regulators before being used as an adjuvant in
vaccines (not necessarily limited to Cervarix ) on the market?
We are not sure whether regulators ever approved AS04 and, if so, based upon what data.
The Adjuvant system AS04 is currently used in the two licensed vaccines Cervarix (HPV 16/18) and
Fendrix (Hepatis B), but has not been submitted for approval as a separate component to any
regulators (FDA, EMA, Japan PMDA) as there is no regulatory requirement nor pathway to register
vaccine adjuvants alone.
The Adjuvant System AS04 as used in Cervarix is composed of 50µg of 3-O-desacyl-4’-

monophosphoryl lipid A (MPL) adsorbed on aluminium hydroxide, hydrated (Al(OH)3) (0.5 milligrams
Al 3+ in total).
The first EPAR for Cervarix (2007) available on

http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_-
_Scientific_Discussion/human/000721/WC500024636.pdf provides details on the specifications,
controls and non-clinical evaluation as performed on the adjuvant and more specifically on the MPL.
EMA concluded based on a review of the analytical methods and their validation for the MPL powder
that the methods used can be considered adequate to control the MPL powder on a routine basis.
The EPAR for Cervarix states that: “ non-clinical data reveal no special hazard for humans based on
conventional studies of safety pharmacology, acute and repeated dose toxicity, local tolerance,
fertility, embryo-foetal and postnatal toxicity (up to the end of the lactation period).” *
The clinical safety evaluation of Cervarix, like for any aluminium adjuvanted vaccine, has been
performed on the final vaccine product (i.e., antigen and adjuvant/adjuvant system combination) and
reflects the safety of the vaccine as a whole.
Overall the Scientific Discussion of the EPAR concludes that satisfactory specifications and controls
have been applied as appropriate to the adjuvants: O-desacyl-4’- monophosphoryl lipid A (MPL) and
aluminium hydroxide, hydrated (Al(OH)3) and that Non-Pharmacopoeial methods have been
satisfactorily validated.
Based on the documentation provided on the adjuvant and an acceptable safety profile of the final
product, Cervarix was granted approval for use as an AS04 adjuvanted HPV vaccine in Europe, in the
US and in Japan.
* Additional information on production and non-clinical assessment of MPL can be found in the EPAR
of Hepatitis B vaccine Fendrix, which is similarly adjuvanted with an AS04 variant.
(http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_-
_Scientific_Discussion/human/000550/WC500021700.pdf
Page 254 of 637

Question 4.
What are the clinical effects of the AS04 versus inert placebo?
We understand from correspondence with regulators (i.e., the European Medicines Agency, EMA,
Australia’s Therapeutic Goods Administration, TGA, Health Canada and New Zealand’s MedSafe) that
there is no separate registration for adjuvants (including AS04). Thus, we wonder what safety studies
GlaxoSmithKline has undertaken of AS04?
Per licensing authorities (European Medicines Agency, Food and Drug Administration and World
Health Organisation) recommendations for non-clinical toxicology studies, AS04 was examined
separately and in combination with antigens. In these animal studies, local tolerance, single and
repeated dose toxicity studies demonstrated that AS04 was well tolerated and no signs of systemic
toxicity were observed at doses up to 30 times the human dose per body weight.
AS04 has not been evaluated as an independent component in clinical trials in humans [Descamps D,
Hardt K, Spiessens B, et al. 2009. Safety of human papillomavirus (HPV)-16/18 AS04-adjuvanted
vaccine for cervical cancer prevention: A pooled analysis of 11 clinical trials. Hum Vaccin, 5(5):332-
340.
Garcon]. Only the vaccine (antigen and AS04 combined) was tested for safety in humans. The safety
of AS04 (with antigen) has been evaluated during the development of several candidate vaccines
including HPV vaccine.
As of November 2016, approximately 64,000 subjects were included in the Company’s clinical
development programme for the AS04-adjuvanted HPV16/18 vaccine. An initial pooled analysis of
this large database of approximately 30,000 girls and women aged 10 years and above show the
HPV-16/18 AS04-adjuvanted cervical cancer vaccine Cervarix to be generally well-tolerated across all
age groups, with a favorable safety profile in women of all ages (Descamps et al. 2009). A second
pooled analysis of clinical trial data was performed 5 years later. This included data of 57,580
subjects and 96,704 vaccine doses and showed the incidence and distribution of AEs was similar
among HPV-16/18-vaccine recipients and controls. No new safety signals were identified in this
updated analysis (Angelo et al. 2014).
The analysis of the data from the clinical development programme confirms the AS04-adjuvanted
HPV16/18 vaccine has an acceptable benefit-risk profile.
Reference
Angelo M-G, David M-P, Zima J, Baril L, Dubin G, Arellano F, Struyf F. Pooled analysis of large and
long-term safety data from the AS04-adjuvanted human papillomavirus vaccine clinical trial program.
Pharmacoepidemiol Drug Saf. 2014 (a);23(5):466–479
Descamps D, Hardt K, Spiessens B, et al. 2009. Safety of human papillomavirus (HPV)-16/18 AS04-
adjuvanted vaccine for cervical cancer prevention: A pooled analysis of 11 clinical trials. Hum Vaccin,
5(5):332-340.
Garcon N, Segal L, Tavares F, et al. The safety evaluation of adjuvants during vaccine development:
The AS04 experience. Vaccine 2011; 29(27): 4453-4459.
Page 255 of 637

Verstraeten, T, Descamps, D, David, MP, et al. 2008. Analysis of adverse events of potential
autoimmune aetiology in a large integrated safety database of AS04 adjuvanted vaccines. Vaccine,
26(51):6630-6638.
Question 5
Does Cervarix contain DNA/RNA and, if so, what is the clinical significance of this?
According to a document by Japan’s Pharmaceuticals and Medical Devices Agency (PMDA) regulator
there are viral DNA and RNA strands in unknown concentrations in Gardasil. The concentrations were
redacted so we do not know what these concentrations were and we do not know if this finding has
any clinical significance. However, we know that RNA/DNA alone injected intramuscularly into BALB-C
mice activates the immune response. We have not seen any information regarding this issue with
respect to Cervarix, so are asking just in case.
To address this question we need to first provide some background information about the
expression system that is used for VLP production. It is important to state that the expression system
used by GSK differs from the expression system used for VLP production for the Gardasil vaccine.
Cervarix is the first vaccine for human use that has been produced with a Baculovirus Expression
Vector System. Hence, an extensive characterization was performed to confirm the safety and
applicability of the integral elements of this novel system. The biological properties and
characteristics of the Hi-5 Rix4446 cell line were extensively investigated. In particular, the Hi-5
Rix4446 insect cell line has been examined for the presence of adventitious agents not only by
applying the classical testing protocol but also by a variety of assays specifically designed for the
detection of insect-specific viral contaminants. In addition, the tumorigenic potential of the cell line
was investigated. The baculovirus seeds were also checked for classical and insect-specific
contaminating viruses. In summary, all measures and precautions have been taken to ensure the
safety of the HPV vaccine against non viral as well as viral adventitious agents.
With respect to viral adventitious agents:
The main source of potential viral contamination of the HPV vaccines lies within the production of
the HPV-16 L1 VLP and HPV-18 L1 VLP antigens, which makes use of starting materials of animal
origin such as the Hi-5 Rix4446 Cell Banks or the HPV-16 or HPV-18 Baculovirus Seeds.
The investigations performed demonstrated efficient viral clearance capacity of the HPV-16 L1 VLP
and HPV-18 L1 VLP antigen production processes. Clearance studies have demonstrated, during
development, that the first chromatography step was capable of removing most if not all the residual
DNA which could be present in the clarified extract from cell expression. Absence of infectivity during
QC testing or specific infectivity experiments has also been demonstrated both in the starting
materials as well as during routine production; thereby supporting the viral safety of the HPV
vaccine.
The DNA content was consistently below the LOQ (limit of quantification, 2 pg/dose) of the threshold
method used for its detection. This LOQ is below the WHO or CBER accepted limits:
Page 256 of 637

• WHO: 10 ng/dose, Technical report series No 878, 1998 – Annex 1: Requirements for the use
of animal cells as in vitro substrates for production of Biologicals
• CBER, FDA : 100 pg/dose, Points to consider in the characterization of cell lines to produce
Biologicals, 1993.
Page 257 of 637

6. Are there any long-term Cervarix studies with cancer (i.e., cervical, anal, oropharyngeal, and penile
cancer) as a specific outcome and, if so, what are the studies identification numbers?
We have been unable to locate any study where actual cancer (i.e., not a surrogate for cancer, such
as, cervical/anal/penile intraepithelial neoplasia [CIN/AIN/PIN] etc.) is reported as the specific
outcome. We are aware that, in 2004, two years before the regulatory approval of the first HPV
vaccine Gardasil, the World Health Organization (WHO) approved a surrogate outcome for cervical
cancer (i.e., cervical intraepithelial neoplasia grade two or more, CIN2+: CIN2, CIN3, adenocarcinoma
in situ and cervical cancer) for regulatory approval.
Prevention of cervical cancer would be the most clinically relevant endpoint for a preventive HPV
vaccine comprising oncogenic types. However as discussed at the FDA in 2001(Pratt D et al. 2001),
based on the low frequency and a protracted natural history, cervical cancer is not considered as a
feasible endpoint in the context of efficacy studies. Furthermore, in clinical studies with active follow
up, it would be unethical not to treat high grade precancerous lesions and wait for invasive cancer to
develop.
Based on the above cancer outcomes can only be studied in larger observational studies, where
women are passively followed up.
A study that may partially address this question is the HPV-027 trial
(https://clinicaltrials.gov/ct2/show/NCT01393470) , a long-term follow-up study conducted to
evaluate the long-term impact of Cervarix on the occurrence of cervical pre-cancerous lesions and
cervical cancer. 2409 Finnish women that have been enrolled in the PATRICIA study and have
received Cervarix will be passively followed up until 2024 via national Pathology registries. Incidence
of CIN3+ and Cervical cancer is compared to an unvaccinated control cohort. However, it needs to be
mentioned that the study is not powered to study cervical cancer as an independent outcome.
In the long term, data concerning vaccine impact on cervical cancer will become available from
countries that can link up vaccination registries and cancer registries (examples are Scotland and the
Scandinavian countries). As most vaccination programs have been implemented between 2007 and
2012 and as it takes 1 to 3 decades for cervical cancer to develop (Moscicki et al. 2006) first evidence
on cancer outcomes may be obtained in the next 5 to 10 years.
To address the question if advanced CIN can be considered a surrogate marker of cervical cancer we
want to mention an academic study conducted at the National Women's Hospital, Auckland, New
Zealand, without any involvement of GSK, where treatment of CIN3 was withheld from a substantial
number of women between 1965 and 1974 . 30 to 50% of the enrolled women developed invasive
cervical cancer during follow up. While this study is considered highly unethical, a thorough analysis
of the outcomes provided strong evidence that CIN3 is indeed a direct precursor of cervical cancer
(McCredie et al. 2008) and supports the WHO and FDA position concerning endpoint choices in HPV
vaccine efficacy trials.
McCredie, M.R., Sharples, K.J., Paul, C., Baranyai, J., Medley, G., Jones, R.W., & Skegg, D.C. Natural
history of cervical neoplasia and risk of invasive cancer in women with cervical intraepithelial
neoplasia 3: a retrospective cohort study. Lancet Oncol. 2008; 9, (5) 425-434
Moscicki, A.B., Schiffman, M., Kjaer, S., & Villa, L.L. Chapter 5: Updating the natural history of HPV
and anogenital cancer. Vaccine 2006; 24 (Suppl. 3), S3-42-S3/51
Page 258 of 637

Pratt D, Goldenthal K, & Gerber A. A discussion of possible endpoints for licensure of human
papillpmavirus (HPV) vaccines. Food and Drug Administration, 2001 (FDA briefing document No.1).
Available at: http://www.fda.gov/ohrms/dockets/ac/01/briefing/3805b1.htm, accessed July 2017.
Page 259 of 637

Letter to: 4 July 2017

Professor Guido Rasi, Executive Director
European Medicines Agency (EMA)
30 Churchill Place
Canary Wharf
London E14 5EU
United Kingdom

Dear Professor Rasi and Executive Team of the EMA,

We are researchers carrying out a systematic review (see our protocol at PROSPERO1) titled, “Benefits and
harms of human papillomavirus vaccines: systematic review of industry clinical study reports and non-
industry published and unpublished reports.”

We have identified 159 industry studies of the HPV vaccines (76 Cervarix, GlaxoSmithKline; and 83
Gardasil/Gardasil 9, Merck Sharp & Dohme) of which 79 (50%) are randomized clinical trials. We found a
high risk of reporting bias. Thus, we decided that our review ought to focus on clinical study reports rather
than journal publications.

We are contacting you with some inquiries based on our ‘access to documents’ requests for the HPV
vaccines’ clinical study reports under POLICY/00432: ASK-3483 (Cervarix), ASK-3998 (Gardasil) and ASK-
21232 (Gardasil 9), and hope you might help us.

(We have also requested studies relating to the Gardasil adjuvant, amorphous aluminium hydroxyl-
phosphate sulfate, in ASK-24805, and periodical safety update reports for all three HPV vaccines in ASK-
26139.)

On 14 May 2014, we requested the HPV vaccines’ clinical study reports from you.

By 1 July 2017, we had received 18 reports of which 13 were eligible for our review (see Table 1).

We are grateful for the received reports; however, none of the 18 reports (that were sent to us in 50
batches containing 128 different files) are complete and all are redacted. The redactions include allocation
numbers and vaccine batch numbers. These are important to assess harms related to particular
participants and vaccine batches.

Up until now, we have received approximately 29,000 pages from you including clinical study reports of
different versions (see Table 1), but we estimate that the total number of pages in the 18 reports is likely to
exceed 50,000. In addition, we have only acquired study protocols for four trials, serious adverse event
(SAE) narratives for five trials, and no complete individual participant data listings for any trial (see Table 1).
These three reporting elements are very important to assess harms.

We are aware that the EMA's current data sharing policies (i.e., POLICY/00432 and POLICY/00703) stress
that, “Openness and transparency are paramount values,”2 and that, “…access to documents or parts
thereof may be granted whenever an overriding public interest in disclosure can be identified by the
Agency,”2 which certainly is the case for the HPV vaccines.

We agree with the EMA that, “Access to CT [clinical trial] data in an analysable format will benefit public
health in the future,”3 and that the EMA, “…should focus on the science and the best interests of patients.”3

1
Website: https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20170030.pdf
2
Website: http://www.ema.europa.eu/docs/en_GB/document_library/Other/2010/11/WC500099473.pdf
Website: http://www.ema.europa.eu/docs/en_GB/document_library/Other/2013/06/WC500144730.pdf
3
Page 260 of 637

Questions in relation to our study requests:

1) How many clinical study reports of Cervarix, Gardasil and Gardasil 9 does the EMA have in total?

2) Which of the clinical study reports that the EMA has of Cervarix, Gardasil and Gardasil 9 are complete
with protocols, main bodies, appendices, serious adverse events narratives and individual participant data?

3) Is there a mechanism by which the EMA can provide us with all of its unredacted clinical study reports of
Cervarix, Gardasil and Gardasil 9 (e.g., if we sign a confidentiality agreement that entails that we only share
non-sensitive information with third parties), so that we can perform our systematic review in an unbiased
manner?

4) Will the EMA assist us by acquiring from the manufacturers the missing parts of the 13 aforementioned
clinical study reports that meet our inclusion criteria (see Table 1)?

5) Does the EMA know of additional clinical study reports of Cervarix, Gardasil and Gardasil 9 than those
the EMA already possesses?

We would be grateful for your response by 1 August 2017.


With best wishes,

Blegdamsvej 9

Oxford OX2 6GG
United Kingdom

Peter C. Gøtzsche, DrMedSci, MSc, Director Nordic Cochrane Centre
Blegdamsvej 9

University of Maryland School of Pharmacy Baltimore
Maryland, 21201
U.S.A.

Page 261 of 637

Table 1: Clinical study reports acquired from the EMA
Cervarix
No. Trial Protocol Main body Appendices SAE narratives Individual participant data listings Pages received
1. HPV-001 Missing* ACQUIRED ACQUIRED ACQUIRED Missing 5,592
2. HPV-008 Missing ACQUIRED ACQUIRED ACQUIRED Missing 3,741

HPV-012
3. Missing* ACQUIRED ACQUIRED ACQUIRED Missing 2,304
[Not eligible]
4. HPV-013 Missing ACQUIRED Missing* Missing Missing 382
5. HPV-014 Missing ACQUIRED Missing Missing Missing 238
6. HPV-015 Missing ACQUIRED Missing Missing Missing 543

HPV-040
7. Missing ACQUIRED Missing Missing Missing 128
[Not eligible]
HPV-070
[Not eligible]
Gardasil
Trial Protocol Main body Appendices SAE narratives Individual participant data listings Pages received
9. V501-005 Missing ACQUIRED Missing Missing Missing 357
11. V501-013 ACQUIRED ACQUIRED Missing Missing Missing 1,797
12. V501-015 ACQUIRED ACQUIRED Missing Missing Missing 713

V501-016
[Not eligible]
15. V501-019 ACQUIRED ACQUIRED ACQUIRED Missing Missing 3,845
16. V501-020 Missing* ACQUIRED ACQUIRED ACQUIRED Missing 2,595
Gardasil 9
Trial Protocol Main body Appendices SAE narratives Individual participant data listings Pages received
V503-001
17. ACQUIRED ACQUIRED Missing Missing Missing 3,942
[Not eligible]
18. V503-006 Missing ACQUIRED Missing ACQUIRED Missing 467
Total received 4/18 18/18 5/18 5/18 0/18 28,938

*The EMA informed us that these files were sent to us, but we have not received the files.
Page 262 of 637

As of 5 August 2018, we have not received a response rom A to our questions.
Page 263 of 637

Paper 4
Systematic review of the HPV vaccines’ clinical study reports
Page 264 of 637

Jørgensen L, Gøtzsche PC and Jefferson T. Benefits and harms of the human papillomavirus (HPV) vaccines:
systematic review with meta-analyses of trial data from clinical study reports. for publication.
2018. Protocol: https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20170030.pdf.
Amendment: https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20171116.pdf.

(A,B,C)
development


November professor
2018
November MRCGP,
Page 265 of 637
2018 FFPHM

2
Page 266 of 637
1
4 Benefits and harms of the human papillomavirus (HPV) vaccines: systematic review with meta-analyses
5 of trial data from clinical study reports
7 Word count: 5,293
10 Lars Jørgensen, LJ, MD (lj@cochrane.dk)1,2
11 Peter C. Gøtzsche, PCG, Professor (pcg@cochrane.dk)1
12 Tom Jefferson, TJ, MD (tj@cochrane.dk)1
13
14
15 Our systematic review protocol was registered on PROSPERO on January 2017:
16 https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20170030.pdf
17
18 Two protocol amendments were registered on PROSPERO on November 2017:
20
21 Our index of the HPV vaccine studies was published in Systematic Reviews on January 2018:
22 https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-018-0675-z
23
24 A description of the challenges obtaining the data was published on September 2018:
25 https://www.bmj.com/content/362/bmj.k3694.full?ijkey=0ibTwph3m0aErxL&keytype=ref
26
27 1Nordic Cochrane Centre, Rigshospitalet 7811, Tagensvej 21, 2100 Copenhagen, Denmark.
28 2Corresponding author (ORCID: 0000-0002-9737-0555; Researcher ID: T-6254-2017); email: lj@cochrane.dk; larsjorgensenmd@gmail.com.
1
Page 267 of 637
29 Abstract
30 Objective: To assess the benefits and harms of the human papillomavirus (HPV) vaccines.
31 Data sources: Clinical study reports obtained from the European Medicines Agency and GlaxoSmithKline
32 from 2014 to 2017.
33 Eligibility criteria: Randomised trials that compared an HPV vaccine with a placebo or active comparator in
34 healthy participants of all ages.
35 Appraisal and synthesis: Two researchers extracted data and judged risk of bias with the Cochrane tool.
36 Risk ratio (RR) estimates were pooled using random effects meta-analysis.
37 Outcomes: Clinically relevant outcomes in intention to treat populations—including HPV-related cancer
38 precursors irrespective of involved HPV types, treatment procedures and serious and general harms.
39 Results: 24 of 50 eligible clinical study reports were obtained with 58,412 pages of 22 trials and two follow-
40 up studies including 95,670 participants: 79,102 females and 16,568 males age 8-72; 393,194 person-years;
41 and 49 months mean weighted follow-up. We judged all 24 studies to be at high risk of bias.
42 Serious harms were incompletely reported for 72% of participants (68,610/95,670). Nearly all control
43 participants received active comparators (48,289/48,595, 99%). No clinical study report included complete
44 case report forms. At four years follow-up, the HPV vaccines reduced HPV-related carcinoma in situ (367 in
45 the HPV vaccine group vs. 490 in the comparator group, RR 0.73 [95% confidence interval, CI, 0.53 to 1.00],
46 number needed to vaccinate [NNV] 387, P=0.05, I2=67%) and HPV-related treatment procedures (1,081 vs.
47 1,416, RR 0.71 [95% CI 0.63 to 0.80], NNV 75, P<0.00001, I2=45%). The HPV vaccines increased serious
48 nervous system disorders (72 vs. 46, RR 1.49 [1.02 to 2.16], number needed to harm [NNH] 1,325, P=0.040,
49 I2=0%) and general harms (13,248 vs. 12,394, RR 1.07 [95% CI 1.03 to 1.11], NNH 51, P<0.00001, I2=77%)
50 but did not significantly increase fatal harms (45 vs. 38, RR 1.19 [95% CI 0.65 to 2.19], P=0.58, I2=30%) or
51 serious harms (1,404 vs. 1,357, RR 1.01 [95% CI 0.94 to 1.08], P=0.79, I2=0%).
52 Conclusion: At four years follow-up, the HPV vaccines decreased HPV-related cancer precursors and
53 treatment procedures but increased serious nervous system disorders and general harms. The extent to
54 which the HPV vaccines’ benefits outweigh their harms is unclear, as the included trials were primarily
55 designed to assess benefits and were not adequately designed to assess harms. Limited access to clinical
2
Page 268 of 637
56 study reports and trial data with case report forms prevented a thorough assessment. An independent
57 assessment of the complete individual participant data is needed.
58 Systematic review registration: CRD42017056093.
59
60 Key words
61 Human papillomavirus vaccine, systematic review, meta-analysis, randomised clinical trial and clinical study
62 report.
63
64 Introduction
65 The approved human papillomavirus (HPV) vaccines—GlaxoSmithKline’s Cervarix™ and Merck Sharp and
66 Dohme’s Gardasil™ and Gardasil 9™—are considered safe and effective (1–3). Recent evidence suggests
67 that the vaccines have significant and long-lasting effects (>12 years) on cervical cancer (4,5), better
68 effectiveness when vaccinated below the age of 17 (6) and are possibly able to substantially reduce the
69 global incidence of cervical cancer (7). However, there are important uncertainties regarding both the
70 benefits and harms of the vaccines.
71
72 Uncertainties of the benefits of the HPV vaccines
73 The HPV vaccines’ regulatory approvals were mainly based on per-protocol populations and surrogate
74 outcomes of HPV-related lesions, e.g., ‘cervical intraepithelial neoplasia or worse’ (CIN2+) infected with an
75 HPV vaccine-specific HPV type, such as HPV types 16 and 18 that are associated with the majority of HPV-
76 related cancers (8–10). It was considered unfeasible and unethical to use HPV-related cancer as the primary
77 outcome (11,12), since it takes many years for a cancer to develop after an HPV infection and also because
78 cervical screening is an established secondary prevention method that leads to removal of precancerous
79 lesions before they become cancerous. Up to 15% of HPV-related cervical cancers may not contain HPV
80 (13), but HPV may be identified in more cases with newer and more sensitive analysis methods (14). HPV-
81 related lesions are often infected with more than one HPV type, some of which may not be targeted by the
82 vaccines (15). This makes it impossible to assess which HPV type caused the lesion. The regulatory vaccine
83 approvals were not based on HPV-related lesions irrespective of HPV type in intention to treat populations,
3
Page 269 of 637
84 and factors such as antigenic shift, antigenic drift and herd immunity may be important in the long-term
85 perspective, as the approved HPV vaccines only target up to nine of the 25 HPV types considered
86 carcinogenic (1).
87
88 Uncertainties of the harms of the HPV vaccines
89 A Cochrane review from 2018 (3) and most large epidemiological studies (16–20) did not find serious or
90 general harms associated with the HPV vaccines. The Cochrane review was mainly based on journal
91 publications that are often influenced by reporting bias (21–24), and epidemiological studies are influenced
92 by confounding (25).
93 Acknowledged rare serious harms include anaphylaxis and syncope (8–10). Some case studies have
94 reported rare neurological harms such as postural orthostatic tachycardia syndrome (POTS) (26,27) and
95 complex regional pain syndrome (CRPS) (28). Cluster analyses of individual case safety reports from the
96 World Health Organization’s (WHOs) VigiBase® revealed additional harms—often serious in nature—that
97 overlapped with the symptomatology of POTS and CRPS (29). Although the European Medicines Agency’s
98 (EMA) investigation of POTS and CRPS did not find an association with the HPV vaccines (2), EMA’s
99 investigation was based on the HPV vaccine manufacturers’ own assessments (30), and about 30 cases of
100 POTS and CRPS were not recognised in the HPV vaccine manufacturers’ trials (31,32). Other reported rare
101 harms have included chronic fatigue syndrome (CFS), Guillain–Barré syndrome (GBS) and premature
102 ovarian failure (POF) (33–35).
103
104 Addressing the uncertainties of the HPV vaccines
105 To address the uncertainties of the benefits and harms of the HPV vaccines, we conducted a systematic
106 review with meta-analyses of trial data from clinical study reports. As of July 2017, about one third of the
107 HPV vaccine studies had not been published and study results were not posted for about half of the
108 completed studies on ClinicalTrials.gov (36). Therefore, we based our review on study programmes in order
109 to identify all trials (36) and on clinical study reports (37), as these reports provide vastly more information
110 about a study than a corresponding journal publication (21–24).
111
4
Page 270 of 637
112 Methods
113 Search strategy and study eligibility
114 Using a six-step process, we constructed and published an index of the HPV vaccine study programmes (36)
115 that included 206 comparative prospective studies (see Figure 1). Two researchers (LJ and TJ) conducted the
116 six steps that included searches of trial registers, journal publication databases and correspondence with
117 regulators and HPV vaccine manufacturers. It was not feasible to account for duplicate entries, as we
118 indexed studies and searched databases that used different IDs for a unique study (e.g., register ID, study
119 programme ID, manufacturer ID and publication ID) (36).
120 In May 2014, we requested the study programmes’ corresponding clinical study reports from the
121 European Medicines Agency (EMA; via its policy 0043) and obtained those reports that were freely available
122 on GlaxoSmithKline’s online trial register. We did not request clinical study reports from the manufacturers,
123 as this would limit our ability to use and share the data (38). In January 2017, we registered our systematic
124 review protocol in PROSPERO (International prospective register of systematic reviews): CRD42017056093
125 (37).
126 We included those trials and their follow-up studies of the 206 comparative studies from our index
127 that were randomised clinical phase II, III or IV trials. We aimed to include studies for which we obtained
128 industry clinical study reports or similar non-industry reports. We also aimed to include periodical safety
129 update reports. PICO criteria (participants, interventions, comparisons and outcomes) were used to select
130 trials that compared an HPV vaccine with a placebo (normal saline) or active comparator (adjuvant or non-
131 HPV vaccine such as a hepatitis vaccine) in healthy participants (see Additional file 1 for our PRISMA
132 checklist).
133
134 Data extraction and risk of bias assessment
135 One researcher performed trial selection and data extraction (LJ); a second researcher (TJ) checked the
136 selection and extraction; a third researcher (PCG) arbitrated. Cochrane’s tool (version 2011) was used for
137 risk of bias assessments (25).
138
139 Outcome assessment
5
Page 271 of 637
140 We assessed the following primary outcomes: all-cause mortality, deaths from and incidence of HPV-related
141 cancers, incidence of histologically confirmed carcinoma in situ and moderate intraepithelial neoplasia, fatal
142 harms, serious harms and harms of special interest (anaphylaxis, chronic fatigue syndrome [CFS], complex
143 regional pain syndrome [CRPS], Guillain-Barré syndrome [GBS], postural orthostatic tachycardia syndrome
144 [POTS], premature ovarian failure [POF] and syncope). Histological outcomes were assessed irrespective of
145 which HPV types were involved.
146 Secondary outcomes included HPV-related external genital lesions and referral procedures, new
147 onset diseases (reported in the included clinical study reports as ‘medically significant conditions’ and ‘new
148 medical history’) and general harms (reported as ‘solicited,’ ‘unsolicited’ and ‘systemic adverse events’). We
149 did not consider cytological, serological or virological outcomes or local harms due to their lower clinical
150 importance.
151 The clinical study reports included over 3,000 different types of harms that were classified with
152 MedDRA (Medical Dictionary for Regulatory Activities) preferred terms. Harms were often incompletely and
153 heterogeneously reported (see Table 1). We extracted and assessed all individual harms classified with
154 MedDRA preferred terms. We performed meta-analyses for the five most commonly occurring fatal and
155 serious harms, the five fatal and serious harms that the HPV vaccines increased the most and the five fatal
156 and serious harms that the HPV vaccines decreased the most. For new onset diseases and general harms, we
157 performed meta-analyses for the three most common, increased and decreased harms for each category
158 (‘medically significant conditions’ and ‘new medical history’; and ‘solicited,’ ‘unsolicited’ and ‘systemic
159 adverse events’). MedDRA preferred terms and total harms were reported as the number of participants
160 with one or more harms over the total number of participants.
161 To check for possible harm clustering on an organ system level, we meta-analysed the MedDRA
162 preferred terms in their respective system organ classes (for example, the MedDRA preferred terms
163 ‘dizziness,’ ‘pain’ and ‘syncope’ were part of and therefore included in the MedDRA system organ class
164 ‘nervous system disorders’). Only Merck clinical study reports included aggregate numbers for participants
165 with MedDRA system organ class harms, and only for new onset diseases (‘new medical history’) and general
166 harms (‘systemic adverse events’). For all GlaxoSmithKline clinical study reports and for serious harms for
167 Merck clinical study reports, we pooled MedDRA preferred terms in their respective system organ classes. A
6
Page 272 of 637
168 participant could potentially be included more than once in a separate analysis (e.g., if a participant
169 experienced a serious ‘headache’ and serious ‘dizziness,’ the participant would be counted twice in the
170 MedDRA system organ class analysis of serious nervous system disorders); we therefore consider the
171 MedDRA system organ class analyses exploratory.
172
173 Post hoc exploratory outcome assessment
174 As we did not obtain complete case report forms or individual participant data for any trial, and as the trials’
175 harm assessments had low internal and external validity (see Table 1 and Discussion), we performed post
176 hoc exploratory outcome analyses where we: 1) compared the clinical study report data with
177 pharmacovigilance data; and 2) assessed signs and symptoms of POTS and CRPS (see protocol amendment
178 on PROSPERO (39)).
179 1) We compared the three largest harm clusters reported from pharmacovigilance up to 1 January
180 2015 to the World Health Organization’s (WHO) VigiBase® (29) with the clinical study report data (for
181 example, VigiBase’s largest HPV vaccine harm cluster—'expected systemic reactions’—consists of the
182 MedDRA preferred terms headache, nausea, pyrexia, dizziness and vomiting). This was done to assess if the
183 pharmacovigilance data were comparable to the clinical study report data. We used the individual harm
184 cluster terms and found the corresponding MedDRA preferred terms in the clinical study report data. The
185 data was synthesized for those MedDRA preferred terms included in each harm cluster.
186 2) POTS and CRPS are rare syndromes that are difficult to identify; as mentioned, about 30 cases of
187 POTS and CRPS were not recognised in the HPV vaccine manufacturers’ trials (31,32) and there were no
188 reports of POTS and CRPS in the clinical study reports (see Results). To assess whether signs and symptoms
189 consistent with POTS and CRPS were present in the data, we asked a physician (Louise Brinth) with clinical
190 expertise in POTS and CRPS to assess the reported MedDRA preferred terms as ‘definitely,’ ‘probably,’
191 ‘probably not’ or ‘definitely not’ associated with the syndromes. As an example, the physician judged the
192 MedDRA preferred terms ‘dizziness postural’ and ‘pain in extremity’ to be ‘definitely’ associated with POTS
193 and CRPS, respectively. The physician was blinded to the allocation groups and outcome data. The data was
194 synthesized for those MedDRA preferred terms that the physician judged ‘definitely’ associated with POTS
195 or CRPS.
7
Page 273 of 637
196 The synthesis of two or more different MedDRA preferred term categories may include a participant
197 more than once in an analysis.
198
199 Data synthesis and analysis
200 Risk ratios were meta-analysed with the random effects inverse variance method. As small trials carry more
201 weight with this method, we compared random effects to a fixed effect risk ratio for all outcomes. Absolute
202 risk estimates were calculated as the number needed to vaccinate (NNV) or harm (NNH). Review Manager
203 5 was used for data synthesis and the intention to treat principle to calculate effect estimates. Sensitivity
204 and sub-group analyses were conducted to investigate potential sources of heterogeneity by taking
205 account of age, gender, risk of bias (25) and type of HPV vaccine and comparator.
206
207 Results
208 Characteristics of included trials
209 We identified 50 eligible studies: 43 industry trials, five industry follow-up studies and two non-industry
210 trials (see Figure 1). We obtained 24 clinical study reports of 58,412 pages from EMA and GlaxoSmithKline
211 for 22 industry trials and two industry follow-up studies (17 Cervarix™, five Gardasil™, one Gardasil 9™ and
212 one Merck HPV type 16 vaccine) with a total of 95,670 participants (79,102 females and 16,568 males age
213 8-72) and 393,194 person-years (see Tables 2 and 3 and Additional file 2). The mean follow-up time was 49
214 months (weighted by sample size). About two fifths of the participants in the control groups received
215 aluminium-based adjuvants (18,403/48,595) and three fifths received hepatitis vaccines (29,877)—less than
216 a thousand participants received carrier solution (597) or saline placebo (306).
217
218 Characteristics of excluded studies
219 For the 26 remaining and potentially eligible studies (23 trials and three follow-up studies) for which no
220 clinical study reports were obtained (or similar reports for the two non-industry trials), numbers of
221 participants were identified for 20 of the 23 industry and one of the two non-industry trials. The trials
222 included 25,632 and 139 participants, respectively, which was equal to 21% (25,632/121,441) of the total
223 eligible sample (see Additional file 3).
8
Page 274 of 637
224
225 Risk of bias of included trials
226 We judged all 22 trials and the two follow-up studies to be at high risk of bias (see Figures 2 and 3 and
227 Additional file 2). Nearly all control participants (48,289/48,595, 99%) received an active comparator such
228 as HPV vaccine aluminium-containing adjuvants or hepatitis vaccines. This distorted—to an unknown
229 extent—the assessment of harms, which we have clarified elsewhere (38). Furthermore, serious harms
230 were incompletely reported for 72% of the participants (68,610/95,670; see Table 1 and Additional file 2).
231 All 24 clinical study reports contained redactions—especially of harms—and lacked significant parts such as
232 serious harm narratives and case report forms (except for two reports: HPV-001 and HPV-008, which,
233 however, included less than half of the participants’ case report forms) (38).
234
235 Benefits
236 Seven clinical study reports assessed histological outcomes of which four reported HPV-related cancer
237 outcomes irrespective of involved HPV types. At four years follow-up, the HPV vaccines did not decrease
238 HPV-related cancer (7 in the HPV vaccine groups vs. 3 in the comparator groups, risk ratio [RR] 1.68 [95%
239 confidence interval, CI, 0.51 to 5.49], P=0.39, I2=0%) or deaths hereof (2 vs. 1, RR 1.44 [95% CI 0.23 to 9.12],
240 P=0.70, I2=0%); whereas they decreased HPV-related carcinoma in situ (367 vs. 490, RR 0.73 [95% CI 0.53 to
241 1.00], number needed to vaccinate [NNV] 387, P=0.05, I2=67%) and the composite surrogate outcome of
242 HPV-related moderate intraepithelial neoplasia or worse (952 vs. 1,239, RR 0.78 [95% CI 0.66 to 0.91], NNV
243 190, P=0.002, I2=53%). The HPV vaccines also decreased HPV-related external genital lesions (289 vs. 582,
244 RR 0.56 [95% CI 0.39 to 0.82], NNV 47, P=0.003, I2=83%) and HPV-related treatment procedures such as
245 cervical conisations (1,081 vs. 1,416, RR 0.71 [95% CI 0.63 to 0.80], NNV 75, P<0.00001, I2=45%) (see Table 4
246 and Additional file 4).
247
248 Harms
249 Serious harms
250 The HPV vaccines increased serious nervous system disorders grouped in the MedDRA system organ class
251 (72 vs. 46, RR 1.49 [95% CI 1.02 to 2.16], number needed to harm [NNH] 1,325, P=0.04, I2=0%) but did not
9
Page 275 of 637
252 significantly increase fatal harms (45 vs. 38, RR 1.19 [95% CI 0.65 to 2.19], P=0.58, I2=30%) or serious harms
253 (1,404 vs. 1,357, RR 1.01 [95% CI 0.94 to 1.08], P=0.79, I2=0%). No individual fatal or serious harm classified
254 with a MedDRA preferred term was significantly increased or decreased by the HPV vaccines (see Tables 5
255 and 8 and Additional file 4).
256
257 New onset diseases
258 The HPV vaccines increased new onset back pain (397 vs. 336, RR 1.15 [95% CI 1.00 to 1.33], NNH 589,
259 P=0.05, I2=0%) but decreased new onset gynaecological chlamydia infection (1,409 vs. 1,512, RR 0.93 [95%
260 CI 0.87 to 1.00], NNV 176, P=0.05, I2=0%) and vaginal infection (369 vs. 420, 0.87 [95% CI 0.76 to 1.00], NNV
261 150, P=0.05, I2=0%). The vaccines also decreased vascular disorders grouped in the MedDRA system organ
262 class (234 vs. 294, RR 0.80 [95% CI 0.67 to 0.94], NNV 439, P=0.009, I2=0%) (see Tables 6 and 8 and
263 Additional file 4).
264
265 General harms
266 The HPV vaccines increased general harms (13,248 vs. 12,394, RR 1.07 [95% CI 1.03 to 1.11], NNH 51,
267 P<0.00001, I2=77%)—especially myalgia (3,989 vs. 3,047, RR 1.41 [95% CI 1.24 to 1.60], NNH 26, P<0.00001,
268 I2=80%), fatigue (4,933 vs. 4,489, RR 1.13 [95% CI 1.08 to 1.18], NNH 67, P<0.00001, I2=22%) and headache
269 (5,561 vs. 5,246, RR 1.06 [95% CI 1.02 to 1.11], NNH 83, P=0.009, I2=40%) (see Table 7 and Additional file 4).
270
271 Harms of special interest
272 Cases of anaphylaxis and syncope were evenly distributed. There were no cases of chronic fatigue
273 syndrome (CFS), complex regional pain syndrome (CRPS), Guillain-Barré syndrome (GBS) or postural
274 orthostatic tachycardia syndrome (POTS), but there was one case of premature ovarian failure (POF) in the
275 HPV vaccine group (see Table 9 and Additional file 4).
276
277 Post hoc exploratory harm analyses
278 The data from the included clinical study reports that corresponded to the three largest harm clusters
279 reported from pharmacovigilance were associated with general harms, but not serious harms or new onset
10
Page 276 of 637
280 diseases. The serious harms that were judged ‘definitely associated’ with POTS or CRPS by the blinded
281 physician were increased by the HPV vaccines, both for POTS (56 vs. 26, RR 1.92 [95% CI 1.21 to 3.07], NNH
282 1,073, P=0.006, I2=0%) and CRPS (95 vs. 57, RR 1.54 [95% CI 1.11 to 2.14], NNH 906, P=0.010, I2=0%). The
283 new onset diseases that were judged ‘definitely associated’ with POTS were also increased by the HPV
284 vaccines (3,675 vs. 3,352, RR 1.08 [95% CI 1.01 to 1.15], NNH 144, P=0.03, I2=29%) (see Table 9 and
285 Additional file 4).
286
287 Subgroup analyses
288 Younger HPV vaccinated participants were more protected against moderate HPV-related intraepithelial
289 neoplasia or worse than older participants (age 15 to 29: 784 vs. 1,079, RR 0.71 [95% CI 0.61 to 0.83]; age
290 21 to 72: 168 vs. 160, RR 1.04 [95% CI 0.84 to 1.29]; ratio of relative risk [RRR] 1.46 [1.12 to 1.91]) and also
291 experienced fewer fatal harms than older participants (age 15 to 27: 24 vs. 32, RR 0.77 [95% CI 0.45, 1.33];
292 age 21 to 72: 21 vs. 6, RR 3.13 [95% CI 1.29 to 7.61]; RRR 0.25 [95% CI 0.09 to 0.70]), but there were no
293 differences for serious nervous system disorders (age 10 to 35: 53 vs. 35, RR 1.46 [95% CI 0.95 to 2.25]; age
294 21 to 72: 19 vs. 11, RR 1.56 [95% CI 0.75 to 3.25]; RRR 0.93 [95% CI 0.40 to 2.19]), serious harms that were
295 judged ‘definitely associated’ with of CRPS (age 9 to 35: 76 vs. 48, RR 1.48 [95% CI 1.03 to 2.12]; age 21 to
296 72: 19 vs. 9, RR 2.11 [95% CI 0.67 to 6.69]; RRR 0.70 [95% CI 0.21 to 2.34]) or serious harms that were
297 judged ‘definitely associated’ with POTS (age 12 to 35: 43 vs. 21, RR 1.86 [95% CI 1.10, 3.15]; age 21 to 72:
298 13 vs. 5, RR 2.22 [95 CI 0.76 to 6.47]; RRR 0.84 [95% CI 0.25 to 2.76]) (see Additional file 4; note that the
299 sub-group analyses used overlapping age groups due to the age groups included in the trials).
300
301 Random effects vs. fixed effect
302 We found similar results with the fixed effect model but with narrower confidence intervals, as the
303 between-trial variance is not included in this model.
304
305 Discussion
306 Our systematic review of 24 clinical study reports with 95,670 participants showed that the HPV vaccines
307 within four years of follow-up decreased HPV-related carcinoma in situ, which have a high likelihood of
11
Page 277 of 637
308 progressing to cancer (1), and HPV-related treatment procedures, but the vaccines also increased serious
309 nervous system disorders and general harms. Younger participants who are those primarily intended to
310 receive HPV vaccination (1) were more protected against HPV-related neoplasia and had fewer fatal harms.
311
312 Strengths
313 Our review was based on study programmes, randomised trials reported in clinical study reports, clinically
314 important pre-specified outcomes, intention to treat analyses, absolute risk estimates and a conservative
315 statistical method based on the random effects model. There was no heterogeneity for serious nervous
316 system disorders or for the post hoc exploratory harm analyses of serious signs and symptoms judged
317 ‘definitely associated’ with POTS and CRPS by a blinded physician with clinical expertise.
318
319 Limitations
320 Insufficient trial data access, incomplete reporting, data fragmentation and limited trial follow-up periods
321 were major limitations. It took three years to obtain an incomplete subset of the eligible clinical study
322 reports; a process which we have documented in detail elsewhere (38). Our review is therefore limited by
323 reporting bias—the bias that we aimed to reduce (37). We did not obtain any periodical safety update
324 reports before our data lock. The inclusion of the remaining participants from the 26 studies with no
325 available clinical study reports included a fifth of the total eligible participants, which could have influenced
326 our review, as some of our results had p-values around our cut-off of 0.05 and confidence intervals that
327 were wide.
328 We performed multiple comparisons: 166 meta-analyses of which 31 (19%) showed statistical
329 significance for the total risk ratio estimate. With our p-value cut-off of 0.05, about eight (166*0.05) or a
330 fourth (8/31) of the significant results are likely to have occurred by chance. We did not use Bonferroni (or
331 similar) corrections (40), as one of our primary outcomes was serious harms, which were affected by
332 incomplete reporting (see Table 1) and lack of saline placebo controls.
333 The 24 included clinical study reports only included one Gardasil 9 trial (V503-006) that was small
334 and did not investigate histological outcomes. Many countries are currently implementing Gardasil 9 as a
335 two-dose regimen in their vaccination programme instead of Cervarix or Gardasil (1). Two doses of Gardasil
12
Page 278 of 637
336 9 may induce fewer harms than three doses, but Gardasil 9 may induce more harms than Gardasil. For
337 example, in the clinical study report that we obtained of the phase 3 multicentre trial V503-
338 001/NCT00543543 (not eligible for our systematic review) of 7,106 and 7,109 healthy females age 16-26
339 randomised to receive three doses Gardasil 9 and Gardasil, there were more serious harms (233 vs. 183, RR
340 1.27 [95% CI 1.05 to 1.54], P=0.010; reported from day 0 to 390) and general harms (‘systemic adverse
341 events’: 2,086 vs. 1,929, RR 1.08 [95% CI 1.03 to 1.14], P=0.003; reported 0-14 days post-vaccination) in the
342 Gardasil 9 group. A 0.5 ml dose of Gardasil 9 contains more virus-like particles (270 microgram vs. 100
343 microgram) and aluminium-containing adjuvant (500 microgram vs. 225 microgram) compared to a 0.5 ml
344 dose of Gardasil, which could explain the harm differences. Although Gardasil 9 targets five more HPV
345 types than Gardasil, Gardasil 9 did not decrease CIN2+ more than Gardasil during trial V503-001’s 42-month
346 follow-up (325 vs. 326, RR 1.00 [95% CI 0.86 to 1.16], P=0.97).
347 A substantial part of our results should be interpreted carefully due to high heterogeneity. We
348 expected the high heterogeneity for several results (e.g., for HPV-related carcinoma in situ), as the included
349 trials comprised 16 different subgroups—based on type of HPV vaccine, comparator, age and gender. All
350 meta-analyses were divided according to the 16 subgroups to provide heterogeneity measures (see
351 Additional file 4), but the nationality of the participants and regional practices of HPV-related screening and
352 treatment procedures may also have contributed to the heterogeneity.
353
354 Limitations of benefit assessment
355 Only 10 HPV-related cancers occurred in the follow-up periods. Extended follow-up was not possible for
356 75% of the comparator participants (36,344/48,595), as they were offered HPV vaccination at trial
357 completion.
358 We only included benefit results of intention to treat analyses, which also included participants
359 that were enrolled after they had been infected with HPV. The HPV vaccines have no documented effect on
360 HPV-related neoplasia caused by previous infections (1). Our benefit results may therefore be skewed
361 toward the null compared to real-life settings where mainly 12-year-old adolescents—that are expected to
362 not be previously HPV-infected—are HPV vaccinated. Getting vaccinated before sexual debut is likely to
13
Page 279 of 637
363 improve the HPV vaccines’ benefits, but no included trial investigated histological outcomes for participants
364 that were vaccinated under the age of 15.
365 Three trials—HPV-008, V501-013 and V501-015 that contained 38% (36,266/95,670) of the
366 analysed participants—were stopped early when HPV type 16/18-related cervical intraepithelial neoplasia
367 or worse (CIN2+) was significantly reduced for their HPV vaccine per-protocol populations. On average,
368 trials stopped early for benefits exaggerate effects by 29% compared to completed trials of the same
369 intervention (41). When the three trials were excluded from our CIN2+ meta-analysis, CIN2+ was not
370 significantly decreased (184 vs. 200, RR 0.85 [95% CI 0.54 to 1.33], P=0.47, I2=77%; see Additional file 4).
371 One clinical study report (HPV-015) only reported CIN2+, although there were three cases of HPV-
372 related cancers in the HPV vaccine group and one in the comparator group (see Additional file 4). These
373 cancers were listed as serious harms and were not mentioned elsewhere in the clinical study report. For
374 transparency, it would have been more appropriate to report each histological outcome (cancer, carcinoma
375 in situ, moderate intraepithelial neoplasia etc.) than only a composite surrogate outcome such as CIN2+.
376 No trial tested the HPV vaccines’ protection against cervical cancer without using cervical
377 screening. This may reduce external validity, as some studies show that HPV-vaccinated women may tend
378 to avoid cervical screening (42); although other studies have not shown a clear tendency (43). The trial
379 personnel often performed cervical screening together with colposcopy every six months and the included
380 participants were often women aged 15-26. In clinical practice, cervical screening is usually performed
381 every three to five years and recommended after age 25 (44), as most CIN2+ lesions in women under age 30
382 regress spontaneously, which may justify active surveillance rather than immediate intervention (45).
383 No trial used mandatory biopsies, which may reduce internal validity. For example, the precursor
384 lesion of cervical adenocarcinoma is difficult to detect on colposcopy, but easier to recognise on a biopsy
385 (46). The incidence of cervical adenocarcinoma is increasing and may more often be HPV negative
386 compared to cervical squamous carcinoma (46), but only 5% (40/857) of the reported cervical carcinoma
387 situ cases in the included studies were adenocarcinoma in situ (see Table 4).
388 We did not prespecify genital warts as an outcome, but the HPV vaccines reduced external genital
389 lesions and there is strong evidence that the HPV vaccines (especially Gardasil and Gardasil 9 that target
390 the HPV types 6 and 11) decrease genital warts (47).
14
Page 280 of 637
391
392 Limitations of harm assessment
393 Only Merck clinical study reports reported aggregate numbers for participants with MedDRA system organ
394 classified harms, and only for new onset diseases and general harms. The synthesis of MedDRA system
395 organ classes for all GlaxoSmithKline clinical study reports and for serious harms for Merck clinical study
396 reports may therefore include a participant more than once. As a result, we consider these analyses
397 exploratory.
398 Serious harms were incompletely reported for 72% of participants (68,610/95,670; see Table 1 and
399 Additional file 2). There were 2.8 times more serious harms reported in the clinical study reports that
400 reported serious harms for the whole trial period (1,838/27,493 vs. 923/38,356). As an example, trial HPV-
401 008 of Cervarix that had reported all serious harms during its 48-months follow-up reported ten times more
402 participants with serious harms compared to V501-015 of Gardasil that only reported serious harms 14 days
403 post-vaccination (1,664/18,644 vs. 102/12,167). In the cluster-randomised trial HPV-040, 88% (28,473 of
404 32,176) of the participants were not included for serious harms reporting (see Table 1 and Additional file 2).
405 The use of active comparators may have underestimated harms related to the HPV vaccines (38).
406 The aluminium-containing comparators were used, as they provided a similar appearance to that of the
407 HPV vaccines, which enhanced blinding and decreased the risk of performance and detection bias. A single
408 trial—V503-006, of Gardasil 9—used a saline placebo in 306 participants who had previously been
409 vaccinated with Gardasil. It is unlikely that those who had experienced harms following previous Gardasil
410 vaccination would have participated in the Gardasil 9 trial, so the trial’s harm results are not reliable. The
411 trial’s blinding procedure was adequate to ensure low risk of performance and detection bias and could
412 have been used in other trials.
413 Although the manufacturers consider the aluminium-containing comparators to be safe, 52% of the
414 participants (49,301/95,670) were only included in the trials if they had never received the aluminium-
415 containing comparators before. GlaxoSmithKline state that their aluminium-containing comparator induces
416 myalgia (“higher incidences of myalgia might namely be attributable to the higher content of aluminium in
417 the HPV vaccine [450 micrograms Al(OH)3] than the content of aluminium in the HAV [hepatitis A] vaccine
418 [225 micrograms Al(OH)3]” (48)), which we found was increased by the HPV vaccines (see Table 7).
15
Page 281 of 637
419 The clinical study reports, their informed consent forms and corresponding journal publications (for
420 example, V501-013 (49) and V501-015 (50)) often used the term placebo (which is a substance with no
421 active effect) to describe the active aluminium-based comparators.
422 Two thirds of the participants (63,468/95,670) were only included in the trials if they had no history
423 of immunological or nervous system disorders (see Additional file 2). Such disorders are not listed as
424 warnings or contraindications on the package inserts of the approved HPV vaccines (8–10). The degree of
425 harms might therefore be higher in clinical practice than in the trials. The HPV vaccines did not increase the
426 three largest HPV vaccine-related VigiBase® harms clusters for serious harms and new onset diseases (see
427 Methods, Table 9 and Additional file 4), which may reflect the differences between real-life and the trials’
428 settings and entry criteria.
429 The exploratory analyses of MedDRA system organ classes may have included a participant more
430 than once. For serious nervous system disorders, this is unlikely, as there were only 118 participants with
431 such disorders (reported as individual MedDRA preferred terms) for 61,331 participants (see Additional file
432 4). We note, however, that the serious nervous system disorders consisted of very heterogenous harms, for
433 example, ‘anoxic encephalopathy,’ ‘moyamoya disease’ and ‘vertebral artery dissection.’
434 The serious harm analyses of MedDRA preferred terms associated with POTS and CRPS may also
435 have included a participant more than once, although this is unlikely as there only were 82 participants
436 with a POTS sign/symptom for 60,058 participants and 152 participants with a CRPS sign/symptom for
437 60,915 participants. The selection of MedDRA preferred terms associated with POTS and CRPS was
438 subjective, not verified by other assessors and included some signs/symptoms that do not align well with
439 the diagnostic criteria of POTS or CRPS (51,52), for example, ‘constipation,’ ‘vision blurred’ and ‘vomiting.’
440 Other blinded assessors would possibly assign MedDRA preferred terms differently, as there were over
441 3,000 different included MedDRA preferred terms. The post hoc exploratory POTS and CRPS analyses were
442 based on randomised trial data where serious harms were underreported and likely underestimated, but
443 since no complete serious harm narratives or complete case report forms were available, the analyses
444 could not take symptom duration, symptom clustering or the diagnostic criteria into account. Therefore,
445 the analyses do not prove that the HPV vaccines cause POTS and CRPS, but they do provide an important
16
Page 282 of 637
446 signal, which makes it paramount to carry out independent analyses of POTS and CRPS based on the
447 complete data set with individual participant data.
448
449 Similar studies
450 In May 2018, a Cochrane review of the HPV vaccines that included 26 trials with 73,428 female participants
451 concluded that the HPV vaccines decrease precursors to cervical cancer and do not increase serious or
452 general harms (3). The Cochrane review had similar inclusion criteria to our review, but it was mainly based
453 on journal publications and only included phase II and III trials. In comparison, we identified 50 possibly
454 eligible studies for which we obtained clinical study reports for 22 trials and two follow-up studies and
455 included 30% more participants (95,670) than the Cochrane review. We found that the HPV vaccines
456 decrease precursors to HPV-related cancer and treatment procedures but increase serious nervous system
457 disorders and general harms. Another recent review on males (53) and most large epidemiological studies
458 have found no serious harms associated with the HPV vaccines (16–20).
459
460 Conclusion
461 At four years follow-up, the HPV vaccines decreased HPV-related precursors to cervical cancer and
462 treatment procedures but increased serious nervous system disorders and general harms. The extent to
463 which the benefits outweigh the harms is unclear, as the included trials were primarily designed to assess
464 benefits and not adequately designed to assess harms. Limited access to clinical study reports and trial data
465 with case report forms prevented a thorough assessment. If granted access to the complete data set with
466 individual participant data, we will update this systematic review. A large industry-independent multicentre
467 trial of two doses Gardasil 9 vs. saline placebo would likely be informative in identifying a more accurate
468 benefit-harm balance, but we recognise that such a trial will be considered unethical in most settings.
469
470 List of abbreviations
471 AIN Anal intraepithelial neoplasia
472 CIN Cervical intraepithelial neoplasia
473 CFS Chronic fatigue syndrome
17
Page 283 of 637
474 CRPS Chronic regional pain syndrome
475 EGL External genital lesions
476 EMA European Medicines Agency
477 FDA Food and Drug Administration
478 GBS Guillain-Barré syndrome
479 GSK GlaxoSmithKline
480 HPV Human papillomavirus
481 ICH International Council for Harmonisation of Technical Requirements for Pharmaceuticals for
482 Human Use
483 MedDRA Medical Dictionary for Regulatory Activities
484 Merck Merck & Co., Inc. or Merck Sharp & Dohme outside the United States and Canada
485 MSC Medically significant conditions
486 NMH New medical history
487 NNH Number needed to harm
488 NNV Number needed to vaccinate
489 PICO Patient, intervention, comparator and outcome
490 PIN Penile intraepithelial neoplasia
491 POF Premature ovarian failure
492 POTS Postural orthostatic tachycardia syndrome
493 PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses
494 PROSPERO International Prospective Register of Systematic Reviews
495 SGAE Solicited general adverse events
496 SYAE Systemic adverse events
497 UGAE Unsolicited general adverse events
498 VaIN Vaginal intraepithelial neoplasia
499 VIN Vulvar intraepithelial neoplasia
500 WHO World Health Organization
18
Page 284 of 637
501 Declarations
502 Ethics approval and consent to participate
503 Not applicable.
504
505 Consent for publication
506 Not applicable.
507
508 Availability of data and material
509 The datasets generated and analysed during the current systematic review are available from the
510 corresponding author (LJ) upon request.
511
512 Competing interests
513 All authors have completed the ICMJE uniform disclosure form. LJ declares no support from any
514 organisation for the submitted work, no financial relationships with any organisations that might have an
515 interest in the submitted work, no other relationships or activities that could appear to have influenced the
516 submitted work. PCG spoke by video link about the HPV vaccines at the IFICA conference in 2018 but
517 received no fee or reimbursement for this. PCG and TJ were co-signatories of a complaint to the European
518 Ombudsman on maladministration in relation to the EMA investigation of possible harms from HPV
519 vaccines. PCG does not regard this as a competing interest. TJ was a co-recipient of a UK National Institute
520 for Health Research grant (HTA – 10/80/01 Update and amalgamation of two Cochrane Reviews:
521 neuraminidase inhibitors for preventing and treating influenza in healthy adults and children—
522 https://www.journalslibrary.nihr.ac.uk/programmes/hta/108001#/). TJ is also in receipt of a Cochrane
523 Methods Innovations Fund grant to develop guidance on the use of regulatory data in Cochrane reviews. TJ
524 is occasionally interviewed by market research companies about phase I or II pharmaceutical products. In
525 2011-14, TJ acted as an expert witness in a litigation case related to the antiviral oseltamivir, in two
526 litigation cases on potential vaccine-related damage and in a labour case on influenza vaccines in
527 healthcare workers in Canada. He has acted as a consultant for Roche (1997-99), GSK (2001-2), Sanofi-
528 Synthelabo (2003) and IMS Health (2013). In 2014-16, TJ was a member of three advisory boards for
19
Page 285 of 637
529 Boehringer Ingelheim. TJ was a member of an independent data monitoring committee for a Sanofi Pasteur
530 clinical trial on an influenza vaccine.
531
532 Funding
533 This study was funded by the Nordic Cochrane Centre, which is funded by the Danish government.
534
535 Authors' contributions
536 LJ wrote the first draft. LJ and TJ contributed to the conception of the review, the design of the review, the
537 collection and assembly of data, the analysis and interpretation of the data, the drafting of the article, the
538 critical revision of the article for important intellectual content and the final approval of the article. PCG
539 contributed to the conception of the review, the critical revision of the article for important intellectual
540 content and the final approval of the article. All authors had full access to all the data in the study and takes
541 responsibility for the integrity of the data and the accuracy of the data analysis.
542
543 Acknowledgements
544 We would like to thank EMA and Louise Brinth for their assistance and Jeppe Bennekou Schroll and Karsten
545 Juhl Jørgensen for helpful comments.
546 References
547 1. World Health Organization. Human papillomavirus vaccines: WHO position paper. Wkly Epidemiol
548 Rec. 2017 May 12;92(19):241-68.
549 2. European Medicines Agency. Review under Article 20 of Regulation (EC) No 726/2004 Human
550 papillomavirus (HPV) vaccines. Available from:
551 http://www.ema.europa.eu/docs/en_GB/document_library/Referrals_document/HPV_vaccines_20/
552 Opinion_provided_by_Committee_for_Medicinal_Products_for_Human_Use/WC500197129.pdf
553 3. Arbyn M, Xu L, Simoens C, Martin-Hirsch PPL. Prophylactic vaccination against human
554 papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database Syst Rev. 2018
555 May 9;5:CD009069. doi: 10.1002/14651858.CD009069.pub3
20
Page 286 of 637
556 4. Kjaer SK, Nygård M, Dillner J, Brooke Marshall J, Radley D, Li M, et al. A 12-Year Follow-up on the
557 Long-Term Effectiveness of the Quadrivalent Human Papillomavirus Vaccine in 4 Nordic Countries.
558 Clin Infect Dis. 2018 Jan 18;66(3):339-345. doi: 10.1093/cid/cix797.
559 5. Guo F, Cofie LE, Berenson AB. Cervical Cancer Incidence in Young U.S. Females After Human
560 Papillomavirus Vaccine Introduction. Am J Prev Med. 2018 Aug;55(2):197-204. doi:
561 10.1016/j.amepre.2018.03.013.
562 6. Dehlendorff C, Sparén P, Baldur-Felskov B, Herweijer E, Arnheim-Dahlström L, Ploner A, et al.
563 Effectiveness of varying number of doses and timing between doses of quadrivalent HPV vaccine
564 against severe cervical lesions. Vaccine. 2018 Oct 15;36(43):6373-6378. doi:
565 10.1016/j.vaccine.2018.09.011.
566 7. Hall MT, Simms KT, Lew J-B, Smith MA, Brotherton JM, Saville M, et al. The projected timeframe until
567 cervical cancer elimination in Australia: a modelling study. Lancet Public Health. 2018 Oct 1. doi:
568 10.1016/S2468-2667(18)30183-X.
569 8. Research C for BE and. Approved Products - Cervarix. Available from:
570 https://www.fda.gov/BiologicsBloodVaccines/Vaccines/ApprovedProducts/ucm186957.htm
571 9. Research C for BE and. Approved Products - Gardasil. Available from:
572 https://www.fda.gov/BiologicsBloodVaccines/Vaccines/ApprovedProducts/UCM094042
573 10. Research C for BE and. Approved Products - Gardasil 9. Available from:
574 https://www.fda.gov/BiologicsBloodVaccines/Vaccines/ApprovedProducts/ucm426445.htm
575 11. Pagliusi SR, Teresa Aguado M. Efficacy and other milestones for human papillomavirus vaccine
576 introduction. Vaccine. 2004 Dec 16;23(5):569-78.
577 12. IARC HPV Working Group. Primary End-points for Prophylactic HPV Vaccine Trials. International
578 Agency for Research on Cancer; 2014.
579 13. de Sanjose S, Quint WG, Alemany L, Geraets DT, Klaustermeier JE, Lloveras B, et al. Human
580 papillomavirus genotype attribution in invasive cervical cancer: a retrospective cross-sectional
581 worldwide study. Lancet Oncol. 2010 Nov;11(11):1048-56. doi: 10.1016/S1470-2045(10)70230-8.
21
Page 287 of 637
582 14. Rodríguez-Carunchio L, Soveral I, Steenbergen RDM, Torné A, Martinez S, Fusté P, et al. HPV-negative
583 carcinoma of the uterine cervix: a distinct type of cervical cancer with poor prognosis. BJOG. 2015
584 Jan;122(1):119-27. doi: 10.1111/1471-0528.13071.
585 15. Interim Clinical Study Report for Study 580299/008 (HPV-008). Available from: https://www.gsk-
586 clinicalstudyregister.com/files2/gsk-580299-008-clinical-study-report-redact.pdf.
587 16. Feiring B, Laake I, Bakken IJ, Greve-Isdahl M, Wyller VB, Håberg SE, et al. HPV vaccination and risk of
588 chronic fatigue syndrome/myalgic encephalomyelitis: A nationwide register-based study from
589 Norway. Vaccine. 2017 Jul 24;35(33):4203-4212. doi: 10.1016/j.vaccine.2017.06.031.
590 17. Miranda S, Chaignot C, Collin C, Dray-Spira R, Weill A, Zureik M. Human papillomavirus vaccination
591 and risk of autoimmune diseases: A large cohort study of over 2 million young girls in France. Vaccine.
592 2017 Aug 24;35(36):4761-4768. doi: 10.1016/j.vaccine.2017.06.030.
593 18. Scheller NM, Pasternak B, Mølgaard-Nielsen D, Svanström H, Hviid A. Quadrivalent HPV Vaccination
594 and the Risk of Adverse Pregnancy Outcomes. N Engl J Med. 2017 Mar 30;376(13):1223-1233. doi:
595 10.1056/NEJMoa1612296
596 19. Phillips A, Patel C, Pillsbury A, Brotherton J, Macartney K. Safety of Human Papillomavirus Vaccines:
597 An Updated Review. Drug Saf. 2018 Apr;41(4):329-346. doi: 10.1007/s40264-017-0625-z.
598 20. Liu EY, Smith LM, Ellis AK, Whitaker H, Law B, Kwong JC, et al. Quadrivalent human papillomavirus
599 vaccination in girls and the risk of autoimmune disorders: the Ontario Grade 8 HPV Vaccine Cohort
600 Study. CMAJ. 2018 May 28;190(21):E648-E655. doi: 10.1503/cmaj.170871.
601 21. Golder S, Loke YK, Wright K, Norman G. Reporting of Adverse Events in Published and Unpublished
602 Studies of Health Care Interventions: A Systematic Review. PLoS Med. 2016 Sep 20;13(9):e1002127.
603 doi: 10.1371/journal.pmed.1002127.
604 22. Schroll JB, Penninga EI, Gøtzsche PC. Assessment of Adverse Events in Protocols, Clinical Study
605 Reports, and Published Papers of Trials of Orlistat: A Document Analysis. PLoS Med. 2016 Aug
606 16;13(8):e1002101. doi: 10.1371/journal.pmed.1002101.
607 23. Sharma T, Guski LS, Freund N, Gøtzsche PC. Suicidality and aggression during antidepressant
608 treatment: systematic review and meta-analyses based on clinical study reports. BMJ. 2016 Jan
609 27;352:i65. doi: 10.1136/bmj.i65.
22
Page 288 of 637
610 24. Maund E, Tendal B, Hróbjartsson A, Jørgensen KJ, Lundh A, Schroll J, et al. Benefits and harms in
611 clinical trials of duloxetine for treatment of major depressive disorder: comparison of clinical study
612 reports, trial registries, and publications. BMJ. 2014 Jun 4;348:g3510. doi: 10.1136/bmj.g3510.
613 25. Cochrane Handbook for Systematic Reviews of Interventions. Available from:
614 http://training.cochrane.org/handbook
615 26. Blitshteyn S, Brook J. Postural tachycardia syndrome (POTS) with anti-NMDA receptor antibodies after
616 human papillomavirus vaccination. Immunol Res. 2017 Feb;65(1):282-284. doi: 10.1007/s12026-016-
617 8855-1.
618 27. Brinth LS, Pors K, Theibel AC, Mehlsen J. Orthostatic intolerance and postural tachycardia syndrome
619 as suspected adverse effects of vaccination against human papilloma virus. Vaccine. 2015 May
620 21;33(22):2602-5. doi: 10.1016/j.vaccine.2015.03.098.
621 28. Kinoshita T, Abe R-T, Hineno A, Tsunekawa K, Nakane S, Ikeda S-I. Peripheral sympathetic nerve
622 dysfunction in adolescent Japanese girls following immunization with the human papillomavirus
623 vaccine. Intern Med. 2015;54(15):1955. doi: 10.2169/internalmedicine.54.4644.
624 29. Chandler RE, Juhlin K, Fransson J, Caster O, Edwards IR, Norén GN. Current Safety Concerns with
625 Human Papillomavirus Vaccine: A Cluster Analysis of Reports in VigiBase®. Drug Saf. 2017
626 Jan;40(1):81-90. doi: 10.1007/s40264-016-0456-3.
627 30. Jefferson T, Jørgensen L. Human papillomavirus vaccines, complex regional pain syndrome, postural
628 orthostatic tachycardia syndrome, and autonomic dysfunction - a review of the regulatory evidence
629 from the European Medicines Agency. Indian J Med Ethics. 2017 Jan;2(1):30-37. doi:
630 10.20529/IJME.2017.006.
631 31. Weber C, Andersen S. The company behind the HPV vaccine underestimated the extent of serious
632 side effects [Firma bag HPV-vaccinen underdrev omfanget af alvorlige bivirkninger]. 2015 Oct 26.
633 Available from: https://www.b.dk/content/item/105701.
634 32. Rapporteurs’ Day 150 Joint Response Assessment Report of Gardasil 9. 2014 Nov 25.
635 EMEA/H/C/3852.
23
Page 289 of 637
636 33. Palmieri B, Poddighe D, Vadalà M, Laurino C, Carnovale C, Clementi E. Severe somatoform and
637 dysautonomic syndromes after HPV vaccination: case series and review of literature. Immunol Res.
638 2017 Feb;65(1):106-116. doi: 10.1007/s12026-016-8820-z.
639 34. Ojha RP, Jackson BE, Tota JE, Offutt-Powell TN, Singh KP, Bae S. Guillain-Barre syndrome following
640 quadrivalent human papillomavirus vaccination among vaccine-eligible individuals in the United
641 States. Hum Vaccin Immunother. 2014;10(1):232-7. doi: 10.4161/hv.26292.
642 35. Little DT, Ward HRG. Premature ovarian failure 3 years after menarche in a 16-year-old girl following
643 human papillomavirus vaccination. BMJ Case Rep. 2012 Sep 30;2012. doi: 10.1136/bcr-2012-006879.
644 36. Jørgensen L, Gøtzsche PC, Jefferson T. Index of the human papillomavirus (HPV) vaccine industry
645 clinical study programmes and non-industry funded studies: a necessary basis to address reporting
646 bias in a systematic review. Syst Rev. 2018 Jan 18;7(1):8. doi: 10.1186/s13643-018-0675-z.
647 37. Jørgensen L, Gøtzsche PC, Jefferson T. Benefits and harms of the human papillomavirus vaccines:
648 systematic review of industry and non-industry study reports. PROSPERO. 2017 Jan. Available from:
650 38. Jørgensen L, Doshi P, Gøtzsche PC, Jefferson T. Challenges of independent assessment of potential
651 harms of HPV vaccine. BMJ. 2018 Sep 24;362:k3694. doi: 10.1136/bmj.k3694.
652 39. Jørgensen L, Gøtzsche PC, Jefferson T. Protocol amendment no. 1 and 2: Benefits and harms of the
653 human papillomavirus vaccines: systematic review of industry and non-industry study reports.
654 PROSPERO. 2017 Nov 16. Available from:
656 40. Polanin JR, Pigott TD. The use of meta-analytic statistical significance testing. Res Synth Methods.
657 2015 Mar;6(1):63-73. doi: 10.1002/jrsm.1124.
658 41. Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, et al. Stopping randomized trials early for
659 benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA.
660 2010 Mar 24;303(12):1180-7. doi: 10.1001/jama.2010.310.
661 42. Budd AC, Brotherton JML, Gertig DM, Chau T, Drennan KT, Saville M. Cervical screening rates for
662 women vaccinated against human papillomavirus. Med J Aust. 2014 Sep 1;201(5):279-82.
24
Page 290 of 637
663 43. Paynter CA, Van Treeck BJ, Verdenius I, Lau AWY, Dhawan T, Lash KA, et al. Adherence to cervical
664 cancer screening varies by human papillomavirus vaccination status in a high-risk population. Prev
665 Med Rep. 2015 Jul 31;2:711-6. doi: 10.1016/j.pmedr.2015.07.011.
666 44. Dickinson JA, Ogilvie G, Niekerk DV, Popadiuk C. Evidence that supports policies to delay cervical
667 screening until after age 25 years. CMAJ. 2017 Mar 13;189(10):E380-E381. doi: 10.1503/cmaj.160636.
668 45. Tainio K, Athanasiou A, Tikkinen KAO, Aaltonen R, Cárdenas J, Hernándes, et al. Clinical course of
669 untreated cervical intraepithelial neoplasia grade 2 under active surveillance: systematic review and
670 meta-analysis. BMJ. 2018 Feb 27;360:k499. doi: 10.1136/bmj.k499.
671 46. Serrano B, Brotons M, Bosch FX, Bruni L. Epidemiology and burden of HPV-related disease. Best Pract
672 Res Clin Obstet Gynaecol. 2018 Feb;47:14-26. doi: 10.1016/j.bpobgyn.2017.08.006.
673 47. Harper DM, DeMars LR. HPV vaccines - A review of the first decade. Gynecol Oncol. 2017
674 Jul;146(1):196-204. doi: 10.1016/j.ygyno.2017.04.004.
675 48. GSK Study Register - Study 104951. Available from: https://www.gsk-
676 clinicalstudyregister.com/study/104951?search=study&search_terms=104951#csr
677 49. Garland SM, Hernandez-Avila M, Wheeler CM, Perez G, Harper DM, Leodolter S, et al. Quadrivalent
678 Vaccine against Human Papillomavirus to Prevent Anogenital Diseases. N Engl J Med. 2007 May
679 10;356(19):1915-27. doi: 10.1056/NEJMoa061741.
680 50. FUTURE II Study Group. Quadrivalent vaccine against human papillomavirus to prevent high-grade
681 cervical lesions. N Engl J Med. 2007 May 10;356(19):1915-27. doi: 10.1056/NEJMoa061741
682 51. Grubb BP. Postural tachycardia syndrome. Circulation. 2008 May 27;117(21):2814-7. doi:
683 10.1161/CIRCULATIONAHA.107.761643.
684 52. Harden RN, Bruehl S, Perez RSGM, Birklein F, Marinus J, Maihofner C, et al. Validation of proposed
685 diagnostic criteria (the “Budapest Criteria”) for Complex Regional Pain Syndrome. Pain. 2010
686 Aug;150(2):268-74. doi: 10.1016/j.pain.2010.04.030.
687 53. Harder T, Wichmann O, Klug SJ, van der Sande MAB, Wiese-Posselt M. Efficacy, effectiveness and
688 safety of vaccination against human papillomavirus in males: a systematic review. BMC Med. 2018 Jul
689 18;16(1):110. doi: 10.1186/s12916-018-1098-3.
690
25
Page 291 of 637
691 Additional material
692 Figure 1. Word document: Benefits and harms of the HPV vaccines - flowchart
693 Figure 2. Word document: Benefits and harms of the HPV vaccines - risk of bias graph
694 Figure 3. Word document: Benefits and harms of the HPV vaccines - risk of bias summary
695 Additional file 1. Word document: Benefits and harms of the HPV vaccines - PRISMA 2009 checklist
696 Additional file 2. Word document: Benefits and harms of the HPV vaccines - characteristics of included
697 studies
698 Additional file 3. Word document: Benefits and harms of the HPV vaccines - list of excluded studies
699 Additional file 4. PDF document: Benefits and harms of the HPV vaccines - meta-analyses
26
Page 292 of 637
700 Table 1: Benefits and harms of the HPV vaccines: reporting of harms in included HPV vaccine studies
Reporting of harms1 Total GlaxoSmithKline Merck Sharp & Dohme
Studies Participants Studies Participants Studies Participants
(N=24) (N=96,855)1 (N=17) (N=66,235)1 (N=7) (N=30,620)
Fatal harms
Reported for the whole study period 23 64,679 (67%) 16 34,059 (51%) 7 30,620 (100%)
Reported for the whole study period for some2 1 32,176 (33%) 1 32,176 (49%) 0 0 (0%)
participants
Serious harms3
Reported for the whole study period 14 28,245 (30%) 14 28,245 (42%) 0 0 (0%)
- No breakdown into MedDRA preferred terms 3 (21%) 2,586 (9%) 3 (21%) 2,586 (9%) 0 (0%) 0 (0%)
Reported 0 to 14 days post-vaccination 7 30,620 (31%) 0 0 (0%) 7 30,620 (100%)
Reported for the 7-month vaccination period 2 5,814 (6%) 2 5,814 (9%) 0 0 (0%)
Reported for a subset or the serious harms judged vaccine- 1 32,176 (33%) 1 32,176 (49%) 0 0 (0%)
related by the trial investigators2
New onset diseases4
Reported as ‘medically significant conditions’ for the 15 65,741 (68%) 15 65,741 (99%) 0 0 (0%)
whole study period
- No breakdown into MedDRA preferred terms 2 (13%) 33,216 (51%) 2 (13%) 33,216 (51%) 0 (0%) 0 (0%)
Reported as ‘new medical history’ for the whole study 7 30,620 (31%) 0 0 (0%) 7 30,620 (100%)
period
Not reported/included in clinical study report 2 494 (1%) 2 494 (1%) 0 0 (0%)
General harms5
Reported as ‘solicited’ and ‘unsolicited’ general harms 7- 14 64,010 (66%) 14 64,010 (96%) 0 0 (0%)
and 30-days post-vaccination
- Reported for a subset of participants6 2 (14%) 7,791/50,820 2 (14%) 7,791/50,820 0 (0%) 0 (0%)
Reported as ‘systemic adverse events’ 14 days post- 7 30,620 (31%) 0 0 (0%) 7 30,620 (100%)
vaccination
- No breakdown into MedDRA preferred terms 3 (43%) 21,441 (70%) 0 (0%) 0 (0%) 3 (43%) 21,441 (70%)
Not reported/included in clinical study report 3 2,225 (3%) 3 2,225 (4%) 0 0 (0%)
701 1See Additional file 2 for details on the reporting of harms. Table 1 includes all 24 clinical study reports including the two follow-up studies HPV-023
702 (follow-up for trial HPV-001) of 433 participants and HPV-063 (follow-up for trial HPV-032) of 752 participants, i.e., 1,185 participants (433+752) are
703 included twice for the trials HPV-001 and HPV-032. The total denominator is 95,670 for the 22 included trials and 96,855 (95,670+1,185) for the 24
704 included clinical study reports.
705 2In one trial (HPV-040), 12% (3,703/32,176) of participants were included in a subset population for fatal and serious harms reporting.
706 31) GlaxoSmithKline defined serious harms as “any untoward medical occurrence that: a. resulted in death, b. was life-threatening, NOTE: The term
707 'life-threatening' in the definition of 'serious' refers to an event in which the subject was at risk of death at the time of the event. It did not refer to
708 an event, which hypothetically might have caused death, if it were more severe. c. required hospitalization or prolongation of existing
709 hospitalization, NOTE: In general, hospitalization signified that the subject had been detained (usually involving at least an overnight stay) at the
710 hospital or emergency ward for observation and/or treatment that would not have been appropriate in the physician's office or out-patient setting.
711 Complications that occurred during hospitalization were AEs [adverse events]. If a complication prolonged hospitalization or fulfilled any other
712 serious criteria, the event was serious. When in doubt as to whether "hospitalization" occurred or was necessary, the AE was to be considered
713 serious. Hospitalization for elective treatment of a pre-existing condition that did not worsen from baseline was not considered an AE. d. resulted in
714 disability/incapacity, NOTE: The term disability means a substantial disruption of a person's ability to conduct normal life functions. This definition
715 was not intended to include experiences of relatively minor medical significance such as uncomplicated headache, nausea, vomiting, diarrhoea,
716 influenza, and accidental trauma (e.g. sprained ankle) which may interfere or prevent everyday life functions but did not constitute a substantial
717 disruption. e. was a congenital anomaly/birth defect in the offspring of a study subject.” 2) Merck Sharp & Dohme defined serious harms as “any
718 adverse experience occurring at any dose that: Results in death; or that is life threatening (places the subject/patient, in the view of the
719 investigator, at immediate risk of death from the experience as it occurred. [Note: This does not include an adverse experience that, had it occurred
720 in a more severe form, might have caused death.]); or that results in a persistent or significant disability/incapacity (substantial disruption of one's
721 ability to conduct normal life functions); or that results in or prolongs an existing inpatient hospitalization (hospitalized is defined as an inpatient
722 admission, regardless of length of stay, even if the hospitalization is a precautionary measure for continued observation.); or ALSO: Other important
723 medical events that may not result in death, not be life threatening, or not require hospitalization may be considered a serious adverse experience
724 when, based upon appropriate medical judgment, the event may jeopardize the subject/patient and may require medical or surgical intervention to
725 prevent one of the outcomes listed above.”
726 41) GlaxoSmithKline defined 'medically significant conditions' as "Adverse events prompting emergency room or physician visits that are not (1)
727 related to common diseases or (2) routine visits for physical examination or vaccination, or SAEs [serious adverse events] that are not related to
728 common diseases. Serious adverse events related to common diseases were reported but are not classified as medically significant conditions for
729 analysis purposes. Common diseases include: upper respiratory infections, sinusitis, pharyngitis, gastroenteritis, urinary tract infections,
730 cervicovaginal yeast infections, menstrual cycle abnormalities and injury." 2) Merck Sharp & Dohme did not provide a formal definition for 'new
731 medical history' but described 'new medical history' as “all new reported diagnoses” in the clinical study report of trial V501-019.
732 51) GlaxoSmithKline defined 'solicited' general adverse events as "Adverse events to be recorded as endpoints in the clinical study [i.e., arthralgia,
733 fatigue, headache, myalgia, pyrexia, rash and urticaria]. The presence/occurrence/intensity of these events is actively solicited from the subject or
734 an observer during a specified post-vaccination follow-up period." 2) GlaxoSmithKline defined 'unsolicited' general adverse event as "Any AE
735 [adverse event] reported in addition to those solicited during the clinical study. Also, any "solicited" symptom with onset outside the specified
736 period of follow-up for solicited symptoms was reported as an unsolicited AE." 3) Merck Sharp & Dohme defined 'systemic adverse event' as "any
737 systemic clinical adverse event that developed on the day of vaccination or during the 14 days after vaccination was recorded on the VRC
738 [vaccination report card] along with the date it started and the last date it was present."
739 6The two trials HPV-008 and HPV-040 only reported general harms for 15% (7,791/50,820) of included participants.
740
27
Page 293 of 637
741 Table 2: Benefits and harms of the HPV vaccines: number of pages obtained of clinical study reports from
742 the European Medicines Agency and GlaxoSmithKline
HPV vaccine manufacturer Study programme ID Total pages obtained European Medicines Agency GlaxoSmithKline
1 GlaxoSmithKline HPV-001 5,813 5,813 0
2 GlaxoSmithKline HPV-003 799 0 799
3 GlaxoSmithKline HPV-008 11,456 4,263 7,193
4 GlaxoSmithKline HPV-013 8,323 382 7,941
18 Merck Sharp & Dohme V501-005 357 357 0
19 Merck Sharp & Dohme V501-013 1,797 1,797 0
Total pages obtained 58,412 20,717 37,695
743
28
Page 294 of 637
744 Table 3: Benefits and harms of the HPV vaccines: characteristics of included participants
Characteristics of Total HPV vaccine Comparator
included participants1 HPV vaccine Comparator Cervarix Gardasil Gardasil 9 HPV 16 vaccine Placebo Adjuvant2 Hepatitis vaccine3
(N=47,075) (N=48,595) (N=31,316) (N=13,937) (N=618) (N=1,204) (N=306) (N=18,789) (N=29,500)
Participation
Randomised 47,075 48,595 31,316 13,937 618 1,204 306 18,789 29,500
- Received one (1) dose 47,012 (99%) 48,556 (99%) 31,291 (99%) 13,927 (99%) 615 (99%) 1,193 (99%) 306 (100%) 18,750 (99%) 29,500 (100%)
- Received two (2) doses 46,105 (98%) 47,725 (98%) 30,788 (98%) 13,564 (97%) 604 (98%) 1,092 (91%) 304 (99%) 18,304 (97%) 29,117 (99%)
- Received three (3) 45,079 (96%) 46,726 (96%) 30,073 (96%) 13,286 (95%) 597 (97%) 1,019 (85%) 300 (98%) 17,906 (96%) 28,520 (97%)
doses
- Completed vaccination 44,202 (94%) 45,862 (94%) 29,331 (94%) 13,156 (94%) 595 (97%) 993 (82%) 300 (98%) 17,809 (95%) 27,753 (94%)
period
- Entered follow-up 18,540 (39%) 18,059 (37%) 4,090 (14%) 13,344 (96%) Not 1,126 (94%) Not 17,590 (94%) 469 (2%)
period applicable applicable
- Completed follow-up 15,826 (34%) 14,601 (30%) 2,929 (10%) 11,986 (86%) Not 835 (69%) Not 14,445 (77%) 156 (1%)
period applicable applicable
Gender
Female 42,036 (89%) 37,066 (76%) 28,876 (92%) 11,338 (81%) 618 (100%) 1,204 (100%) 306 (100%) 16,481 (88%) 20,279 (69%)
Age
Mean age in years 20.3 20.2 21.2 21.4 19.0 20.0 19.0 22.9 20.5
Age group range in years 9-72 8-68 9-72 9-45 12-26 16-25 12-26 9-68 8-46
Race
Asian 7,589 (16%) 7,295 (15%) 6,232 (20%) 1,248 (9%) 40 (6%) 69 (6%) 14 (5%) 2,678 (14%) 4,603 (16%)
Black 1,426 (3%) 1,492 (3%) 467 (2%) 862 (6%) 3 (1%) 94 (8%) 3 (1%) 1,108 (6%) 381 (1%)
Hispanic 4,492 (10%) 4,378 (9%) 1,787 (6%) 2,616 (19%) 0 (0%) 89 (7%) 0 (0%) 3,403 (18%) 975 (3%)
White 31,743 (67%) 33,558 (69%) 22,335 (70%) 7,998 (56%) 483 (78%) 918 (76%) 231 (75%) 9,960 (53%) 23,367 (79%)
Other 1,625 (3%) 1,576 (3%) 297 (1%) 1,202 (9%) 92 (15%) 34 (3%) 58 (19%) 1,343 (7%) 174 (1%)
Unknown 209 (1%) 296 (1%) 198 (1%) 11 (1%) 0 (0%) 0 (0%) 0 (0%) 297 (2%) 0 (0%)
745 1See Additional file 2 for details on the characteristics of included participants. Table 3 does not include data from the two follow-up studies HPV-
746 023 (follow-up for trial HPV-001) of 433 participants and HPV-063 (follow-up for trial HPV-032) of 752 participants.
747 2Adjuvant comparators included amorphous aluminium hydroxyphosphate sulphate (AAHS), aluminium hydroxide (Al[OH] ) and Gardasil’s carrier
3
748 solution (yeast protein, sodium chloride, L-histidine, polysorbate 80 and sodium borate).
749 3Hepatitis vaccines included Aimmugen™ (hepatitis A vaccine), Engerix-B™ (hepatitis B vaccine), Havrix™ (hepatitis A vaccine) and Twinrix
750 Paediatric™ (hepatitis A and B vaccine); see Additional file 2.
29
Page 295 of 637
751 Table 4: Benefits and harms of the HPV vaccines: summary of HPV-related outcomes
Summary of HPV-related outcomes1 HPV vaccine (N=47,075) Comparator (N=48,595) Risk ratio3 [95% CI]
Cancer mortality
Total 2 1 1.44 [0.23, 9.12]
- Cervical 1 (50%) 0 (0%) 2.99 [0.12, 73.33]
- Oropharyngeal 1 (50%) 1 (100%) 1.00 [0.10, 9.58]
Cancer incidence
Total 7 3 1.68 [0.51, 5.49]
- Anal 0 (0%) 0 (0%) Not applicable
- Cervical 3 (43%) 2 (67%) 1.41 [0.19, 10.21]
- Oropharyngeal 1 (14%) 1 (33%) 1.00 [0.10, 9.58]
- Penile Not reported Not reported Not applicable
- Vaginal 1 (14%) 0 (0%) 2.99 [0.12, 73.33]
- Vulvar 2 (29%) 0 (0%) 3.01 [0.31, 28.89]
Not HPV-related 20 23 0.90 [0.49, 1.63]
Carcinoma in situ incidence
Total 367 490 0.73 [0.53, 1.00]
- Anal (AIN3) Not reported Not reported Not applicable
- Cervical 367 (100%) 490 (100%) 0.73 [0.53, 1.00]
- Adenoid type (AIS) 9 (2%) 31 (6%) 0.32 [0.15, 0.66]
- Squamous type (CIN3) 358 (98%) 459 (94%) 0.85 [0.61, 1.17]
- Penile (PIN3) Not reported Not reported Not applicable
- Vaginal (VaIN3) Not reported Not reported Not applicable
- Vulvar (VIN3) Not reported Not reported Not applicable
Moderate intraepithelial neoplasia incidence
Total 538 763 0.81 [0.59, 1.11]
- Anal (AIN2) 0 (0%) 0 (0%) Not applicable
- Cervical (CIN2) 538 (100%) 763 (100%) 0.81 [0.59, 1.11]
- Penile (PIN2) Not reported Not reported Not applicable
- Vaginal (VaIN2) Not reported Not reported Not applicable
- Vulvar (VIN2) Not reported Not reported Not applicable
Carcinoma in situ or worse incidence
Total 372 498 0.79 [0.59, 1.05]
- Anal (AIN3+) Not reported Not reported Not applicable
- Cervical (CIN3+, AIS included) 372 (100%) 498 (100%) 0.79 [0.59, 1.05]
- Penile (PIN3+) Not reported Not reported Not applicable
- Vaginal (VaIN3+) Not reported Not reported Not applicable
- Vulvar (VIN3+) Not reported Not reported Not applicable
Moderate intraepithelial neoplasia or worse incidence
Total 952 1,239 0.78 [0.66, 0.91]
- Anal (AIN2+) 0 (0%) 0 (0%) Not applicable
- Cervical (CIN2+) 892 (93%) 1,144 (92%) 0.81 [0.68, 0.97]
- Penile (PIN2+) 3 (1%) 3 (1%) 1.00 [0.20, 4.95]
- Vaginal (VaIN2+) 17 (2%) 27 (2%) 0.64 [0.32, 1.27]
- Vulvar (VIN2+) 18 (2%) 36 (3%) 0.49 [0.18, 1.36]
- Vaginal or vulvar (VIN2+ or VaIN2+) 22 (2%) 29 (2%) 0.76 [0.44, 1.32]
External genital lesion (EGL) incidence
Total 289 582 0.56 [0.39, 0.82]
HPV-related referral procedures2
Any 1,941 2,264 0.86 [0.81, 0.90]
Biopsy 2,449 3,021 0.74 [0.62, 0.88]
Endoscopy 4,354 4,965 0.88 [0.85, 0.91]
Treatment (surgical and non-surgical) 1,018 1,416 0.71 [0.63, 0.80]
752 1See Additional file 4 section 1 to 8 for meta-analyses of HPV-related outcomes. It was not feasible to present this summary table for the 16
753 subgroups (based on age group, gender, type of HPV vaccine and comparator) of the 24 included clinical study reports.
754 2Two trials (V501-013 and V501-015) reported ‘any’ procedure, while other trials reported individual outcomes, for example, ‘biopsy.’
755 3Risk ratios were calculated with the random effects inverse variance method.
30
Page 296 of 637
756 Table 5: Benefits and harms of the HPV vaccines: summary of fatal and serious harms
Summary of fatal and serious harms1 HPV vaccine (N=47,075) Comparator (N=48,595) Risk ratio5 [95% CI]
Fatal harms
Participants with fatal harms2 45 38 1.19 [0.65, 2.19]
Number of MedDRA classified fatal harms2 79 51 Not applicable
- Number of fatal harms judged HPV vaccine-related 0 (0%) 0 (0%) Not applicable
Most common fatal harms (MedDRA preferred terms: n=participants)
Cardiorespiratory arrest 3 2 0.99 [0.13, 7.65]
Completed suicide 4 8 0.58 [0.15, 2.19]
Gunshot wound 2 3 0.74 [0.09, 5.85]
Homicide 2 2 0.95 [0.14, 6.50]
Road traffic accident 5 7 0.77 [0.24, 2.46]
Fatal harms most increased by the HPV vaccines (MedDRA preferred
terms: n=participants)
Cardiac arrest 2 0 3.00 [0.31, 28.82]
Metastases to lung 2 0 3.00 [0.31, 28.82]
Renal failure acute 2 0 3.00 [0.31, 28.82]
Systemic lupus erythematosus 2 0 3.00 [0.31, 28.82]
Traumatic intracranial haemorrhage 2 0 3.00 [0.31, 28.82]
Fatal harms most decreased by the HPV vaccines (MedDRA preferred
terms: n=participants)3
Completed suicide 4 8 0.58 [0.15, 2.19]
Gunshot wound 2 3 0.74 [0.09, 5.85]
Road traffic accident 5 7 0.77 [0.24, 2.46]
Serious harms
Participants with serious harms4 1,404 1,357 1.01 [0.94, 1.08]
- Participants that withdrew due to a serious harm 54 (4%) 49 (4%) 1.08 [0.72, 1.61]
Number of MedDRA classified serious harms4 1,741 1,628 Not applicable
- Number of serious harms judged HPV vaccine-related 46 (3%) 44 (3%) Not applicable
Most common serious harms (MedDRA preferred terms: n=participants)
Abortion missed 33 41 0.81 [0.51, 1.27]
Abortion spontaneous 89 78 1.14 [0.84, 1.55]
Abortion spontaneous complete 63 62 1.01 [0.71, 1.44]
Abortion spontaneous incomplete 73 54 1.35 [0.95, 1.92]
Appendicitis 72 82 0.85 [0.62, 1.17]
Serious harms most increased by the HPV vaccines (MedDRA preferred
Abortion spontaneous 89 78 1.14 [0.84, 1.55]
Abortion spontaneous incomplete 73 54 1.35 [0.95, 1.92]
Pneumonia 26 15 1.64 [0.87, 3.09]
Pyelonephritis 31 17 1.70 [0.93, 3.10]
Tonsillitis 18 9 1.59 [0.72, 3.49]
Serious harms most decreased by the HPV vaccines (MedDRA preferred
Abortion missed 33 41 0.81 [0.51, 1.27]
Appendicitis 72 82 0.85 [0.62, 1.17]
Ligament rupture 5 12 0.44 [0.15, 1.29]
Ovarian cyst rupture 6 13 0.46 [0.18, 1.21]
Overdose 22 31 0.72 [0.42, 1.23]
757 1See Additional file 4 section 9 and 10 for fatal and serious harm meta-analyses. The applied harm categories are MedDRA preferred terms. It was
758 not feasible to present this summary table for the 16 subgroups (based on age group, gender, type of HPV vaccine and comparator) of the 24
759 included clinical study reports.
760 2The clinical study reports reported 130 individual MedDRA classified fatal harms for 83 participants.
761 3There were 20 different MedDRA preferred term categories of fatal harms with the same non-significant difference, i.e., no fatal harm in the HPV
762 vaccine group and one fatal harm in the comparator group.
763 4The clinical study reports reported 3,369 individual MedDRA classified serious harms for 2,761 participants, i.e., 1.2 serious harms per participant.
764 Each MedDRA classified serious harm was reported as the number of participants with a MedDRA classified serious harm over the total number of
765 participants.
31
Page 297 of 637
767 Table 6: Benefits and harms of the HPV vaccines: summary of new onset diseases
Summary of new onset diseases1 HPV vaccine Comparator Risk ratio6 total Risk ratio6 MSC Risk ratio6 NMH
total (N=47,075) total (N=48,595) [95% CI] [95% CI] [95% CI]
Total
Participants with new onset diseases2 14,258 14,014 0.99 [0.97, 1.02] 0.98 [0.90, 1.06] 1.00 [0.97, 1.03]
- Follow-up3 2,296 2,365 0.98 [0.94, 1.01] Not applicable 0.98 [0.94, 1.01]
Number of MedDRA classified new onset 47,474 46,662 Not applicable Not applicable Not applicable
diseases2
- Medically significant conditions (MSC)4 7,882 (17%) 7,994 (17%) Not applicable Not applicable Not applicable
- New medical history (NMH)5 39,592 (83%) 38,668 (83%) Not applicable Not applicable Not applicable
Most common new onset diseases (MedDRA
preferred terms: n=participants)
MSC Depression 443 432 1.02 [0.89, 1.16] 1.02 [0.85, 1.23] 1.01 [0.84, 1.22]
Genitourinary tract gonococcal 149 162 0.92 [0.74, 1.15] 0.91 [0.73, 1.14] 1.15 [0.37, 3.52]
infection
Gynaecological chlamydia infection 1,409 1,512 0.93 [0.87, 1.00] 0.95 [0.88, 1.03] 0.87 [0.76, 1.00]
NMH Vaginal candidiasis 1,297 1,359 0.95 [0.89, 1.02] Not applicable 0.95 [0.89, 1.02]
Vaginitis bacterial 1,185 1,204 0.98 [0.91, 1.06] Not applicable 0.98 [0.91, 1.06]
Urinary tract infection 1,023 1,086 0.93 [0.86, 1.01] 0.33 [0.01, 8.19] 0.93 [0.86, 1.02]
New onset diseases most increased by the HPV
vaccines (MedDRA preferred terms:
n=participants)
MSC Abdominal pain 433 374 1.21 [0.98, 1.50] 1.38 [1.00, 1.92] 1.17 [0.87, 1.57]
Back pain 397 336 1.15 [1.00, 1.33] 1.40 [1.05, 1.86] 1.08 [0.91, 1.28]
Headache 771 693 1.06 [0.92, 1.22] 1.29 [0.75, 2.24] 1.04 [0.93, 1.15]
NMH Amenorrhoea 394 359 1.09 [0.87, 1.37] 0.66 [0.38, 1.15] 1.17 [0.93, 1.48]
Headache 771 693 1.06 [0.92, 1.22] 1.29 [0.75, 2.24] 1.04 [0.93, 1.15]
Joint sprain 113 83 1.18 [0.80, 1.75] 0.60 [0.29, 1.22] 1.45 [0.94, 2.24]
New onset diseases most decreased by the
HPV vaccines (MedDRA preferred terms:
n=participants)
MSC Cystitis 480 502 0.93 [0.77, 1.09] 0.65 [0.44, 0.96] 0.99 [0.87, 1.13]
Gynaecological chlamydia infection 1,409 1,512 0.93 [0.87, 1.00] 0.95 [0.88, 1.03] 0.87 [0.76, 1.00]
Type 2 diabetes mellitus 31 47 0.89 [0.38, 2.09] 0.62 [0.32, 1.20] 3.00 [0.47, 19.02]
NMH Urinary tract infection 1,023 1,086 0.93 [0.86, 1.01] 0.33 [0.01, 8.19] 0.93 [0.86, 1.02]
Vaginal candidiasis 1,297 1,359 0.95 [0.89, 1.02] Not applicable 0.95 [0.89, 1.02]
Vaginal infection 369 420 0.87 [0.76, 1.00] Not applicable 0.87 [0.76, 1.00]
768 1See Additional file 4 section 11 for meta-analyses of new onset diseases. The applied harm categories are MedDRA preferred terms. New onset
769 diseases consist of ‘medically significant conditions’ (MSC) and ‘new medical history’ (NMH). Numbers for ‘HPV vaccine’ and ‘comparator’ are the
770 total of MSC and NMH. We divided new onset diseases for MSC and NMH, since the definitions for MSC and NMH differed (see Table 1). It was not
771 feasible to present this summary table for the 16 subgroups (based on age group, gender, type of HPV vaccine and comparator) of the 24 included
772 clinical study reports.
773 2The clinical study reports reported 94,136 individual MedDRA preferred term classified new onset diseases for 28,272 participants, i.e., 3.3 new
774 onset diseases per participant. New onset diseases were reported as the number of participants over the total number of participants.
775 3‘Follow-up’ represents the trials V501-005, V501-019 and V501-020 that had dichotomized reporting of new medical history (NMH) into the
776 vaccination period (day 0 to month 7) and follow-up period (from month 7 to the last day of follow-up). We included the vaccination periods for
777 these trials in ‘participants with new onset diseases’ and included the follow-up periods in ‘follow-up.’
778 4GlaxoSmithKline defined 'medically significant conditions' as "Adverse events prompting emergency room or physician visits that are not (1) related
779 to common diseases or (2) routine visits for physical examination or vaccination, or SAEs [serious adverse events] that are not related to common
780 diseases. Serious adverse events related to common diseases were reported but are not classified as medically significant conditions for analysis
781 purposes. Common diseases include: upper respiratory infections, sinusitis, pharyngitis, gastroenteritis, urinary tract infections, cervicovaginal yeast
782 infections, menstrual cycle abnormalities and injury."
783 5Merck Sharp & Dohme did not provide a formal definition for 'new medical history' but described 'new medical history' as “all new reported
784 diagnoses” in the clinical study report of trial V501-019.

32
Page 298 of 637
786 Table 7: Benefits and harms of the HPV vaccines: summary of general harms
Summary of general harms1 HPV vaccine total Comparator Risk ratio6 total Risk ratio6 SGAE Risk ratio6 UGAE Risk ratio6 SYAE
(N=47,075) total [95% CI] and [95% CI] [95% CI] [95% CI]
(N=48,595)
Total
Participants with general harms2 13,248 12,394 1.07 [1.03, 1.11] 1.11 [1.06, 1.16]7 1.11 [1.06, 1.16]7 1.01 [0.98, 1.03]
Number of MedDRA classified general 37,999 31,916 Not applicable Not applicable Not applicable Not applicable
harms2
- Solicited general adverse events 30,408 (80%) 25,300 (79%) Not applicable Not applicable Not applicable Not applicable
(SGAE)3
- Unsolicited general adverse events 3,197 (8%) 3,136 (10%) Not applicable Not applicable Not applicable Not applicable
(UGAE)4
- Systemic adverse events (SYAE)5 4,394 (12%) 3,480 (11%) Not applicable Not applicable Not applicable Not applicable
Most common general harms (MedDRA
SGAE Fatigue 4,933 4,489 1.13 [1.08, 1.18] 1.14 [1.09, 1.19] 1.00 [0.15, 6.53] 0.92 [0.70, 1.20]
and Headache 5,561 5,246 1.06 [1.02, 1.11] 1.08 [1.03, 1.14] 1.76 [1.26, 2.47] 0.98 [0.90, 1.07]
UGAE
Myalgia 3,989 3,047 1.41 [1.24, 1.60] 1.42 [1.24, 1.63] 1.15 [0.24, 5.57] 1.33 [0.95, 1.85]
SYAE Headache 5,561 5,246 1.06 [1.02, 1.11] 1.08 [1.03, 1.14] 1.76 [1.26, 2.47] 0.98 [0.90, 1.07]
Pyrexia 1,599 1,386 1.12 [1.02, 1.22] 1.15 [1.06, 1.25] 1.47 [0.93, 2.34] 1.05 [0.80, 1.36]
Nasopharyngitis 339 277 1.17 [0.91, 1.50] Not applicable 1.40 [0.94, 2.09] 0.95 [0.78, 1.16]
General harms most increased by the
n=participants)
SGAE Fatigue 4,933 4,489 1.13 [1.08, 1.18] 1.14 [1.09, 1.19] 1.00 [0.15, 6.53] 0.92 [0.70, 1.20]
and Headache 5,561 5,246 1.06 [1.02, 1.11] 1.08 [1.03, 1.14] 1.76 [1.26, 2.47] 0.98 [0.90, 1.07]
UGAE
Myalgia 3,989 3,047 1.41 [1.24, 1.60] 1.42 [1.24, 1.63] 1.15 [0.24, 5.57] 1.33 [0.95, 1.85]
SYAE Myalgia 3,989 3,047 1.41 [1.24, 1.60] 1.42 [1.24, 1.63] 1.15 [0.24, 5.57] 1.33 [0.95, 1.85]
Nausea 213 148 1.21 [0.89, 1.65] Not applicable 1.32 [0.35, 4.98] 1.25 [0.84, 1.86]
Pyrexia 1,599 1,386 1.12 [1.02, 1.22] 1.15 [1.06, 1.25] 1.47 [0.93, 2.34] 1.05 [0.80, 1.36]
General harms most decreased by the
n=participants)
SGAE Influenza 119 120 0.91 [0.61, 1.36] Not applicable 0.88 [0.39, 1.97] 0.94 [0.56, 1.58]
and Cough 86 87 0.89 [0.65, 1.21] Not applicable 0.83 [0.46, 1.49] 0.90 [0.60, 1.37]
UGAE
Oropharyngeal pain 111 97 1.10 [0.80, 1.50] Not applicable 0.91 [0.58, 1.43] 1.29 [0.75, 2.22]
SYAE Fungal infection 4 11 0.78 [0.09, 6.43] Not applicable 3.01 [0.31, 28.83] 0.18 [0.04, 0.82]
Sinus headache 9 15 0.49 [0.21, 1.14] Not applicable Not applicable 0.49 [0.21, 1.14]
Joint injury 2 5 0.47 [0.11, 2.01] Not applicable 3.01 [0.31, 28.83] 0.15 [0.03, 0.88]
787 1See Additional file 4 section 12 for meta-analyses of general harms for the 16 subgroups (based on age group, type of HPV vaccine and
788 comparator) of the 24 included clinical study reports. The applied harm categories are MedDRA preferred terms. The table contains general harms
789 of ‘solicited general adverse events’ (SGAE), ‘unsolicited general adverse events’ (UGAE) and ‘systemic adverse events’ (SYAE). Numbers for ‘HPV
790 vaccine’ and ‘comparator’ are the total of SGAE, UGAE and SYAE, but to avoid double counting of participants, UGAE (that accounted for less than
791 10% of the general harms) were dismissed from the total risk ratio for studies that reported SGAE and UGAE separately (SGAE and UGAE were not
792 reported as pooled estimates for individual general harms classified with MedDRA preferred terms; see Additional file 4). It was not feasible to
793 present this summary table for the 16 subgroups (based on age group, type of HPV vaccine and comparator) of the 24 included clinical study
794 reports.
795 2The clinical study reports reported 69,915 individual MedDRA classified general harms for 25,642 participants, i.e., 2.7 general harms per
796 participant. General harms were reported as the number of participants with a MedDRA classified general harm over the total number of
797 participants.
798 3GlaxoSmithKline defined 'solicited general adverse events’ (SGAE) as "Adverse events to be recorded [from day 0 to day 6 after each vaccination]
799 as endpoints [arthralgia, fatigue, headache, myalgia, pyrexia, rash and urticaria] in the clinical study."
800 4GlaxoSmithKline defined 'unsolicited general adverse events’ (UGAE) as "Any AE [adverse event] reported in addition to those solicited during the
801 clinical study. Also, any "solicited" symptom with onset outside the specified period of follow-up for solicited symptoms was reported as an
802 unsolicited AE."
803 5Merck Sharp & Dohme defined 'systemic adverse events' (SYAE) as "any systemic clinical adverse event that developed on the day of vaccination or
804 during the 14 days after vaccination was recorded on the VRC [vaccination report card]."
806 7The total numbers of participants with general harms in Cervarix studies were reported as ‘solicited [SGAE] and unsolicited [UGAE],’ i.e., the risk
807 ratio is similar for SGAE and UGAE.
808
33
Page 299 of 637
809 Table 8: Benefits and harms of the HPV vaccines: summary of exploratory harm analyses by MedDRA
810 system organ class
Summary of Fatal harms Serious harms New onset diseases2 General harms3
exploratory harm HPV HPV Comparato Comparato
analyses by MedDRA Comparator Risk ratio4 Comparator Risk ratio4 HPV vaccine Risk ratio4 HPV vaccine Risk ratio4
vaccine vaccine r r
(N=48,595) [95% CI] (N=48,595) [95% CI] (N=47,075) [95% CI] (N=47,075) [95% CI]
system organ class1 (N=47,075) (N=47,075) (N=48,595) (N=48,595)
Blood and lymphatic Not 0.97 1.00 1.33
0 0 11 12 424 421 17 10
system disorders applicable [0.44, 2.18] [0.88, 1.15] [0.56, 3.15]
2.45 1.12 1.09 0.89
Cardiac disorders 13 5
[0.87, 6.87]
28 21
[0.49, 2.55]
108 86
[0.67, 1.78]
6 5
[0.26, 2.99]
Congenital, familial and Not 0.32 0.74 0.55
0 0 1 9 50 67 0 1
genetic disorders applicable [0.09, 1.13] [0.51, 1.06] [0.02, 13.40]
Ear and labyrinth Not 0.68 1.04 1.10

0 0 4 6 154 139 54 40
disorders applicable [0.19, 2.45] [0.78, 1.39] [0.71, 1.69]
2.83 1.60 0.91 0.72
Endocrine disorders 1 0
[0.12, 69,36]
7 4
[0.48, 5.35]
189 207
[0.74, 1.10]
1 1
[0.04, 11.85]
Not 1.51 0.89 0.84
Eye disorders 0 0
applicable
6 4
[0.42, 5.37]
296 326
[0.76, 1.04]
40 39
[0.53, 1.33]
Gastrointestinal 2.83 1.12 1.05 1.09
1 0 107 92 2,226 2,141 4,085 3,735
disorders [0.12, 69,36] [0.85, 1.48] [0.95, 1.16] [1.01, 1.17]
Not 1.45 1.14 1.10
General disorders 0 0
applicable
13 9
[0.60, 3.50]
563 483
[1.01, 1.28]
6,401 5,842
[1.04, 1.17]
0.94 1.03 0.86 2.00
Hepatobiliary disorders 1 1
[0.06, 15.06]
52 48
[0.70, 1.52]
126 143
[0.61, 1.22]
2 1
[0.18, 22.00]
Immune system 2.83 0.68 0.92 1.29
1 0 9 14 371 398 35 24
disorders [0.12, 69,36] [0.30, 1.58] [0.78, 1.08] [0.76, 2.18]
Infections and 1.51 1.04 0.98 1.14

8 5 400 367 8,970 9,025 1,141 965
infestations [0.49, 4.61] [0.80, 1.37] [0.96, 1.00] [1.00, 1.30]
Injury, poisoning and

0.84 0.85 0.94 1.03
procedural 16 18
[0.43, 1.64]
199 224
[0.64, 1.13]
1,205 1,209
[0.82, 1.06]
142 122
[0.81, 1.33]
complications
Not 0.69 1.00 1.34
Investigations 0 0
applicable
1 2
[0.11, 4.40]
1,438 1,429
[0.94, 1.07]
7 4
[0.41, 4.41]
Metabolism and 0.94 1.44 0.95 0.98
1 1 19 11 342 355 17 15
nutrition disorders [0.06, 15.06] [0.72, 2.89] [0.82, 1.10] [0.49, 1.96]
Musculoskeletal and
4.71 1.07 1.02 1.34
connective tissue 2 0
[0.23, 98.09]
35 31
[0.66, 1.74]
1,263 1,213
[0.94, 1.10]
6,005 4,683
[1.21, 1.49]
disorders
Neoplasms benign,
3.06 1.20 1.09 0.71
malignant and 13 4
[1.00, 9.39]
68 56
[0.84, 1.71]
466 421
[0.96, 1.25]
2 2
[0.14, 3.51]
unspecified
Nervous system 3.77 1.49 1.06 1.09
4 1 72 46 1,410 1,305 5,967 5,422
disorders [0.42, 33.71] [1.02, 2.16] [0.97, 1.16] [1.04, 1.14]
Pregnancy, puerperium Not 1.08 1.02 2.91

0 0 458 426 726 714 1 0
and perinatal conditions applicable [0.94, 1.23] [0.92, 1.12] [0.12, 68.66]
0.47 0.92 0.99 0.90
Psychiatric disorders 4 8
[0.14, 1.56]
80 87
[0.68, 1.25]
961 959
[0.91, 1.08]
49 53
[0.61, 1.33]
Renal and urinary 1.88 1.07 1.01 1.32
2 1 19 17 404 395 12 6
disorders [0.17, 20.77] [0.57, 2.01] [0.88, 1.16] [0.48, 3.59]
Reproductive system 4.71 0.90 0.99 1.13

2 0 72 78 3,458 3,463 213 158
and breast disorders [0.23, 98.09] [0.65, 1.24] [0.95, 1.04] [0.92, 1.38]
Respiratory, thoracic
1.41 1.26 1.00 1.02
and mediastinal 3 2
[0.24, 8.45]
44 34
[0.81, 1.97]
795 771
[0.91, 1.11]
464 394
[0.89, 1.18]
disorders
Skin and subcutaneous 2.83 1.48 0.99 1.21
1 0 12 7 1,173 1,176 997 771
tissue disorders [0.12, 69,36] [0.60, 3.63] [0.88, 1.12] [1.03, 1.44]
0.94 0.95 0.95 1.00
Social circumstances 2 2
[0.13, 6.69]
2 2
[0.14, 6.50]
46 42
[0.62, 1.47]
1 1
[0.10, 9.60]
Not
Surgical and medical Not 1.15 0.97
0 0 6 5 1,318 1,340 0 0 applicabl
procedures applicable [0.36, 3.67] [0.90, 1.04]
e
1.26 0.96 0.80 0.68
Vascular disorders 4 3
[0.28, 5.61]
16 16
[0.47, 1.99]
234 294
[0.67, 0.94]
11 12
[0.28, 1.63]
811 1See Additional file 4 sections 9 to 12 for meta-analyses of MedDRA system organ classes. Only Merck clinical study reports aggregated data for
812 MedDRA system organ classes per participant and only for new onset diseases (‘new medical history’) and general harms (‘systemic adverse
813 events’). A participant may have been counted more than once in the MedDRA system organ class analyses of GlaxoSmithKline clinical study reports
814 and in analyses of fatal and serious harms; we therefore consider these analyses exploratory. Risk ratios for GlaxoSmithKline and Merck studies are
815 provided separately in Additional file 4. It was not feasible to present this summary table for the 16 subgroups (based on age group, gender, type of
816 HPV vaccine and comparator) of the 24 included clinical study reports. To avoid double counting of participants in the total risk ratio estimate we
817 only included the new onset diseases reported in the vaccination period for the trials V501-005, V501-019 and V501-020, and we only included
818 solicited adverse events when studies also reported unsolicited adverse events (see Additional file 4).
819 2New onset diseases was compiled of the harm categories ‘medically significant conditions’ (for Cervarix) and ‘new medical history’ (for Gardasil,
820 Gardasil 9 and the HPV 16 vaccine).

821 3General harms was compiled of the harm categories ‘solicited general adverse events,’ ‘unsolicited general adverse events’ (for Cervarix) and
822 ‘systemic adverse events’ (for Gardasil, Gardasil 9 and the HPV 16 vaccine). To avoid double counting of participants, ‘unsolicited general adverse
823 events’ were dismissed from the total risk ratio for studies that reported SGAE and UGAE separately.
825
34
Page 300 of 637
826 Table 9: Benefits and harms of the HPV vaccines: summary of harms of special interest and post hoc
827 exploratory harm analyses
Summary of harms of special Serious harms New onset diseases4 General harms5
interest and post hoc exploratory HPV vaccine Comparator Riskratio6 HPV vaccine Comparator Riskratio6 HPV vaccine Comparator Risk ratio6
harm analyses1 (N=47,075) (N=48,595) [95% CI] (N=47,075) (N=48,595) [95% CI] (N=47,075) (N=48,595) [95% CI]
Harms of special interest (MedDRA

Anaphylaxis 2 4 0.59 [0.13, 2.82] 11 8 1.18 [0.48, 2.91] 0 0 Not applicable
Chronic fatigue syndrome (CFS) 0 0 Not applicable 0 0 Not applicable 0 0 Not applicable
Chronic regional pain syndrome 0 0 Not applicable 0 0 Not applicable 0 0 Not applicable
(CRPS)
Guillain-Barré syndrome (GBS) 0 0 Not applicable 0 0 Not applicable 0 0 Not applicable
Postural orthostatic tachycardia 0 0 Not applicable 0 0 Not applicable 0 0 Not applicable

syndrome (POTS)
Premature ovarian failure (POF) 0 0 Not applicable 1 0 3.00 [0.12, 73.48] 0 0 Not applicable
Syncope 4 3 0.94 [0.23, 3.81] 62 60 1.03 [0.58, 1.84] 7 7 0.77 [0.25, 2.34]
Post hoc exploratory analyses of

VigiBase® harm clusters2
Expected systemic reactions 25 11 1.96 [0.96, 3.98] 1,465 1,358 1.03 [0.93, 1.14] 10,926 9,948 Not applicable7
Allergic/hypersensitivity reactions 2 2 0.96 [0.14, 6.52] 284 279 1.05 [0.82, 1.35] 1,912 1,469 1.30 [1.18, 1.45]
Vasovagal reactions 9 5 1.31 [0.50, 3.46] 232 212 1.06 [0.78, 1.44] 173 123 1.20 [0.93, 1.55]
Post hoc exploratory analyses of

CRPS and POTS3
Harms judged as ‘definitely 95 57 1.54 [1.11, 2.14] 5,079 4,790 1.04 [0.98, 1.10] 27,899 23,223 Not applicable7
associated’ with CRPS
Harms judged as ‘definitely 56 26 1.92 [1.21, 3.07] 3,675 3,352 1.08 [1.01, 1.15] 18,207 16,288 Not applicable7
associated’ with POTS
828 1See Additional file 4 sections 13 and 14 for meta-analyses of the harms of special interest and post hoc exploratory harm analyses. There was no
829 applicable fatal harm of special interest. It was not feasible to present this summary table for the 16 subgroups (based on age group, gender, type
830 of HPV vaccine and comparator) of the 24 included clinical study reports. As we did not obtain complete case report forms or individual participant
831 data, we could not assign harms to individual participants.
832 2As the included studies’ harm assessments were at risk of low internal and external validity (see Table 1 and Discussion), we compared the three
833 largest harm clusters reported from pharmacovigilance up to 1 January 2015 to the World Health Organization’s (WHO) VigiBase with the clinical
834 study report data. We did this to see if the pharmacovigilance data were similar to the study data. VigiBase’s largest HPV vaccine harm cluster
835 (expected systemic reactions) consists of ‘headache, nausea, pyrexia, dizziness and vomiting.’ VigiBase’s 2nd largest HPV vaccine harm cluster
836 (allergic/hypersensitivity reactions) consists of ‘pruritis, urticaria, rash and erythema.’ VigiBase’s 3rd largest HPV vaccine harm cluster (vasovagal
837 reactions) consists of ‘syncope, dizziness, loss of consciousness, pallor and seizure.’ As we synthesised individual MedDRA preferred term classified
838 harms, our post hoc exploratory analyses of VigiBase harm clusters may therefore include a participant more than once in each separate analysis.
839 3We asked a physician with clinical expertise in POTS and CRPS to assess the reported MedDRA terms as ‘definitely,’ ‘probably,’ ‘probably not’ or
840 ‘definitely not’ associated with the syndromes. The physician was blinded to the allocation groups and outcome data. The data was synthesized for
841 those MedDRA preferred terms that the physician judged ‘definitely’ associated with POTS or CRPS. As we synthesised individual MedDRA preferred
842 terms, our post hoc exploratory analyses of CRPS and POTS may include a participant more than once in each separate analysis.
843 4New onset diseases were compiled of the harm categories ‘medically significant conditions’ (for Cervarix) and ‘new medical history’ (for the HPV 16
844 vaccine, Gardasil and Gardasil 9).
845 5General harms were compiled of the harm categories ‘solicited general adverse events,’ ‘unsolicited general adverse events’ (for Cervarix) and
846 ‘systemic adverse events’ (for Gardasil, Gardasil 9 and the HPV 16 vaccine).
848 7Some numerators exceeded the denominators making the result nonsensical. Therefore, we did not perform meta-analyses.
849
850
35
Page 301 of 637
Figure 1: Benefits and harms of the HPV vaccines: flowchart of the inclusion of clinical study
reports
1For details on the correspondence and searches conducted in steps 1 to 6, see Jørgensen et al. “Index of the HPV vaccine industry clinical
study programmes and non-industry funded studies” (https://doi.org/10.1186/s13643-018-0675-z): Appendices 1 and 2.

2N=the number of studies/entries evaluated.
3206 studies were identified according to our inclusion and classification criteria, see Jørgensen et al. “Index of the HPV vaccine industry
clinical study programmes and non-industry funded studies” (https://doi.org/10.1186/s13643-018-0675-z): Methods.
Page 302 of 637

Figure 2: Benefits and harms of the HPV vaccines: risk of bias graph
Page 303 of 637

Figure 3: Benefits and harms of the HPV vaccines: risk of bias summary1
1Each study is noted as: “manufacturer ID: type of HPV vaccine vs. type of comparator (included gender, age group; months of follow-up),”
e.g., “HPV-001: Cervarix vs. Al(OH)3 (f, 15-26; 27).”
Page 304 of 637

Additional file 1: Benefits and harms of the HPV vaccines: PRISMA 2009 checklist
Reported on
page #
TITLE
Title 1 Identify the report as a systematic review, meta-analysis or both. 1
ABSTRACT
Structured summary 2 Provide a structured summary including, as applicable: background; objectives; 2-3
INTRODUCTION
Rationale 3 Describe the rationale for the review in the context of what is already known. 3-4
Objectives 4 Provide an explicit statement of questions being addressed with reference to 4
METHODS
Protocol and 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web 5
registration address), and, if available, provide registration information including registration
number.
Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report 5
Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact 5
with study authors to identify additional studies) in the search and date last
searched.
Search 8 Present full electronic search strategy for at least one database, including any 5
limits used, such that it couldbe repeated.
Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in 5
systematic review, and, if applicable, included in the meta-analysis).
Data collection 10 Describe method of data extraction from reports (e.g., piloted forms, 5
process independently, in duplicate) and any processes for obtaining and confirming data
from investigators.
Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding 5-7
Risk of bias in 12 Describe methods used for assessing risk of bias of individual studies (including 5
Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, 8
including measures of consistency (e.g., I2) for each meta-analysis.
Section/topic # Checklist item Reported on page #
Page 305 of 637

Additional file 1: Benefits and harms of the HPV vaccines: PRISMA 2009 checklist
Risk of bias across 15 Specify any assessment of risk of bias that may affect the cumulative 5
studies evidence (e.g., publication bias, selective reporting within studies).
Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup 7

RESULTS
Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in 8
the review, with reasons for exclusions at each stage, ideally with a flow
diagram.
Study characteristics 18 For each study, present characteristics for which data were extracted 8
(e.g., study size, PICOS, follow-up period) and provide the citations.
Risk of bias within 19 Present data on risk of bias of each study and, if available, any outcome 8-9
studies level assessment (see item 12).
Results of individual 20 For all outcomes considered (benefits or harms), present, for each study: 9
studies (a) simple summary data for each intervention group (b) effect estimates
and confidence intervals, ideally with a forest plot.
Synthesis of results 21 Present results of each meta-analysis done, including confidence Tables
intervals and measures of consistency.
Risk of bias across 22 Present results of any assessment of risk of bias across studies (see Item 8
studies 15).
Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup 11
DISCUSSION
Summary of 24 Summarize the main findings including the strength of evidence for each 11-12
evidence main outcome; consider their relevance to key groups (e.g., healthcare
providers, users, and policy makers).
Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at 11-15
review-level (e.g., incomplete retrieval of identified research, reporting
bias).
FUNDING
Funding 27 Describe sources of funding for the systematic review and other support 20
(e.g., supply of data); role of funders for the systematic review.
Page 306 of 637

Additional file 2
Benefits and harms of the HPV vaccines:

characteristics of included studies with supporting statements for risk of bias judgements
Summary table
HPV vaccine Study N=months N=participants HPV vaccine Comparator Note

manufacturer programme ID follow-up
1 GlaxoSmithKline HPV-001 27 1,113 Cervarix Al(OH)3
2 GlaxoSmithKline HPV-003 12 61 Cervarix Al(OH)3
3 GlaxoSmithKline HPV-008 48 18,644 Cervarix Havrix
4 GlaxoSmithKline HPV-013 12 2,067 Cervarix Havrix
5 GlaxoSmithKline HPV-015 36 5,753 Cervarix Al(OH)3
6 GlaxoSmithKline HPV-023 36 (433*) Cervarix Al(OH)3 HPV-001 follow-up
7 GlaxoSmithKline HPV-029 12 541 Cervarix Twinrix
8 GlaxoSmithKline HPV-030 12 493 Cervarix Engerix-B
10 GlaxoSmithKline HPV-032 7 1,040 Cervarix Aimmugen
11 GlaxoSmithKline HPV-033 7 321 Cervarix Havrix
14 GlaxoSmithKline HPV-040 72 32,176 Cervarix Engerix-B
16 GlaxoSmithKline HPV-063 12 (752*) Cervarix Aimmugen HPV-032 follow-up
17 GlaxoSmithKline HPV-069 12 1,212 Cervarix Engerix-B
18 Merck Sharp & Dohme V501-005 48 2,409 HPV 16 vaccine AAHS
19 Merck Sharp & Dohme V501-013 45 5,455 Gardasil AAHS
21 Merck Sharp & Dohme V501-018 18 1,781 Gardasil Carrier solution
24 Merck Sharp & Dohme V503-006 7 924 Gardasil 9 Saline placebo
*Studies HPV-023 and HPV-063 were follow-up studies of HPV-001 and HPV-032.
Page 307 of 637

HPV-001 (NCT00689741)
Title: “A double blind, placebo-comparator led, randomized, pilot phase IIB study of the efficacy of a human
papillomavirus (HPV) HPV-16/18 AS04 vaccine in the prevention of HPV-16 and/or HPV-18 cervical infection in healthy
adolescent and young adult women in North America and Brazil.”
Characteristics:
Methods Randomized, parallel group, double-blind trial that was 27 months long.
Participants 1,113 healthy females (560 in the HPV vaccine group and 553 in the comparator group), age 15-26,
from USA, Canada and Brazil. Participants were excluded from the trial if they previously had
received any of the adjuvants that were part of the HPV vaccine or comparator or had a history of
any neurological disease.
Interventions Cervarix (0.5 millilitre) vs. aluminium hydroxide (Al[OH]3, 0.5 milligram in 0.5 millilitre saline) given
intramuscularly at 0, 1 and 6 months. The batch numbers of the HPV vaccine and comparator were
redacted.
Outcomes All-cause mortality, fatal and serious harms, new onset diseases (reported as ‘medically significant
conditions’ for the whole study period) and general harms (reported as ‘solicited’ and ‘unsolicited’
general harms 7- and 30-days post-vaccination, respectively). No benefit outcomes were eligible.
Serious harms were not reported for individual MedDRA categories.
Notes The maximum blinding time was 27 months and there was an optional extension phase of the trial.
Risk of Bias assessment:

Author’s
Domain Supporting statement
judgment
Random sequence Low risk of bias "Randomization was performed via a SAS program written by GSK
generation (selection Biologicals using a procedure 'Ranuni' and using an Internet-based
bias) central randomization system (SBIR). The randomization was stratified
by age and region."
Allocation concealment Low risk of bias “…the randomisation system determined the vaccine number to be
(selection bias) used for the subject. The vaccine number was used as patient
identification number (PID) for all data collected on the subject under
the study. The investigator (or designate) used the eCRF of the RDE
system to document the patient identification number (PID).”
Blinding of participants Low risk of bias "This was a double-blind study. The investigator and associated study
and personnel personnel were unaware of the treatment assignments for the
(performance bias) subjects. The study statistician at GSK Biologicals, Rixensart, had access
to the randomization schedule. Once the final subject was randomized,
the study statistician filed all randomization schedule documents in a
sealed envelope. The study remains blinded and all analyses have been
performed by an external statistician."
Blinding of outcome Low risk of bias "This was a double-blind study. The investigator and associated study
assessment (detection personnel were unaware of the treatment assignments for the
bias) subjects. The study statistician at GSK Biologicals, Rixensart, had access
to the randomization schedule. Once the final subject was randomized,
the study statistician filed all randomization schedule documents in a
sealed envelope. The study remains blinded and all analyses have been
performed by an external statistician."
Incomplete outcome High risk of bias 958 of 1,113 participants completed follow-up (86%). Less than half of
data (attrition bias) the case report forms were included.
Selective reporting High risk of bias The outcome data contained redactions.
(reporting bias)
Other High risk of bias The trial was funded by GlaxoSmithKline and used inadequate design
and reporting.
Page 308 of 637

HPV-003 (NCT not available)
Title: “A phase I/II study to evaluate the safety and immunogenicity of MEDI-517, a virus-like participle vaccine against
human papillomavirus (HPV) types 16 and 18 [i.e., Cervarix], in healthy adult female volunteers who are HPV-16 or
HPV-18 DNA positive.”
Characteristics:
Participants 61 healthy females (31 in the HPV vaccine group and 30 in the comparator group) age 18-30
allocated to 27 different centres in the United States. Participants were excluded from the trial if
they previously had received any of the adjuvants that were part of the HPV vaccine or comparator.
intramuscularly at 0, 1 and 6 months.
Outcomes All-cause mortality, fatal and serious harms and general harms (reported as ‘solicited’ and
‘unsolicited’ general harms 7- and 30-days post-vaccination, respectively). No benefit outcomes
were eligible and new onset diseases were not reported.
Notes None.

Author’s
Risk of bias domain Supporting statement
judgment
Random sequence Low risk of bias Computer generated randomization list.
generation (selection
bias)
Allocation concealment Low risk of bias “The investigator assigned a participant identification number”. The
(selection bias) number was computer generated. The study pharmacist passed the
number to the investigator “but did not reveal treatment assignment
to anyone.”
Blinding of participants Low risk of bias "This was a double-blind study. The investigator, all clinic staff (other
and personnel than the study pharmacist), the volunteers and the sponsor's staff
(performance bias) (other than the study statistician) were unaware of the treatment
assignments of the volunteers."
Blinding of outcome Low risk of bias "This was a double-blind study. The investigator, all clinic staff (other
assessment (detection than the study pharmacist), the volunteers and the sponsor's staff
bias) (other than the study statistician) were unaware of the treatment
assignments of the volunteers."
Incomplete outcome High risk of bias 47 of 61 participants completed follow-up (77%). The individual
data (attrition bias) participant data was not included.
Selective reporting High risk of bias The outcome data contained redactions. The trial only reported
(reporting bias) serious harms for the seven-month vaccination period and not for the
five-month follow-up period. New onset diseases were not reported.
and reporting.
Page 309 of 637

HPV-008 (NCT00122681)
Title: “A Phase III, Double-blind, Randomized, Controlled, Multi-centre Study to Evaluate the Efficacy of
GlaxoSmithKline. Biologicals. HPV-16/18 VLP AS04 Vaccine Compared to Hepatitis A Vaccine as Control in Prevention
of Persistent HPV-16 or HPV-18 Cervical Infection and Cervical Neoplasia, Administered Intramuscularly According to a
0, 1, 6 Month Schedule in Healthy Females 15-25 Years of Age.”
Characteristics:
Participants 18,644 healthy females (9,319 in the HPV vaccine group and 9,325 in the comparator group) age
15-25 allocated to 133 different centres in Australia, Belgium, Brazil, Canada, Finland, Germany,
Italy, Mexico, Philippines, Spain, Taiwan, Thailand, the United Kingdom and the United States.
Participants were excluded from the trial if they previously had received any of the adjuvants that
were part of the HPV vaccine or comparator or had a history of any neurological disease.
Interventions Cervarix (0.5 millilitre) vs. Havrix (hepatitis A vaccine, 0.5 millilitre) given intramuscularly at 0, 1 and
6 months.
Outcomes All-cause mortality, mortality from and incidence of HPV-related cancers irrespective of HPV-type,
incidence of histologically confirmed carcinoma in situ and moderate abnormal histology
irrespective of HPV-type, HPV-related referral procedures, fatal and serious harms, new onset
diseases (reported as ‘medically significant conditions’ for the whole study period) and general
harms (reported as ‘solicited’ and ‘unsolicited’ general harms 7- and 30-days post-vaccination,
respectively, for a subset of participants: 6,159).
Notes The control group was vaccinated with the HPV vaccine at the end of follow-up (“The study IDMC
recommended that unblinding and cross-over immunization of both treatment and comparator
recipients with the HPV vaccine or licensed Havrix, as appropriate, be offered to subjects after
completion of their end-of-study activities”).

Author’s
judgment
Random sequence Low risk of bias "A randomization list was generated at GSK Biologicals, Rixensart,
generation (selection using a standard SAS (Statistical Analysis System) program and was
bias) used to number the vaccines. A randomization blocking scheme (1: 1
ratio) was used to ensure that balance between treatments was
maintained: a treatment number uniquely identified the vaccine doses
to be administered to the same subject.”
Allocation concealment Low risk of bias “the treatment allocation was performed at the investigator sites using
(selection bias) a central randomization system on Internet (SBIR) ... the system
allocated a unique treatment number using a minimization algorithm
taking into account the study site and the subject age range.”
Blinding of participants Low risk of bias "This study was performed in a double-blinded manner. Blinding was
and personnel maintained for all subjects and investigators and their study staff
(performance bias) participating in this study with regard to the individual subject
treatment assignments allocated in this study (vaccine or comparator)
and the HPV DNA PCR and serological results until all subjects had
completed the study, except if the subjects had requested the non-
emergency unblinding."
Blinding of outcome Low risk of bias "This study was performed in a double-blinded manner. Blinding was
assessment (detection maintained for all subjects and investigators and their study staff
bias) participating in this study with regard to the individual subject
treatment assignments allocated in this study (vaccine or comparator)
and the HPV DNA PCR and serological results until all subjects had
completed the study, except if the subjects had requested the non-
emergency unblinding."
Incomplete outcome High risk of bias 15,609 of 18,644 participants completed follow-up (84%). The
data (attrition bias) individual participant data was not included. Reasons for selecting
safety diary card subset population by centre were not stated.
(reporting bias)
and reporting.
Page 310 of 637

HPV-013 (NCT00196924)
Title: “A Phase III, double-blind, randomized, comparator led study to evaluate the safety and immunogenicity of
GlaxoSmithKline. Biologicals' HPV-16/18 AS04 vaccine administered intramuscularly according to a 0, 1, 6-month
schedule in healthy female subjects aged 10-14 years.”
Characteristics:
Methods Randomized, parallel group, observer-blind trial that was 12 months long.
Participants 2,067 healthy females (1,035 in the HPV vaccine group and 1,032 in the comparator group) age 10-
14 allocated to 57 different centres in Australia, Colombia, Czech Republic, France, Germany,
Honduras, Korea, Norway, Panama, Spain, Sweden and Taiwan. Participants were excluded from
the trial if they previously had received any of the adjuvants that were part of the HPV vaccine or
comparator, had a history of any neurological or immunological disorder.
6 months.
Notes None.

Author’s
judgment
Random sequence Low risk of bias "A randomized list was generated at GSK Biologicals, Rixensart,
generation (selection Belgium, using a standard SAS® (Statistical Analysis system) program
bias) and was used to number the vaccines. A randomization blocking
scheme (1: 1 ratio) was used to ensure that balance between
treatments was maintained.”
Allocation concealment Low risk of bias “…the randomization system [SBIR] determined the treatment number
(selection bias) to be allocated to the subject.”
Blinding of participants High risk of bias “…the different visual presentations of the HPV -16/18 vaccine and the
and personnel HAV comparator vaccine (both vaccines contain different quantities of
(performance bias) aluminium, which results in a different turbidity of the vaccines and
may allow identification)…all subjects and study personnel not
involved in preparation and administration of study vaccines were
blinded to the individual subject treatment.”
Blinding of outcome Low risk of bias "Study HPV-013 was conducted as observer-blinded.”
assessment (detection
bias)
Incomplete outcome High risk of bias 2,027 of 2,067 participants completed follow-up (98%). The individual
(reporting bias)
and reporting.
Page 311 of 637

HPV-015 (NCT00294047)
Title: “A phase III, double-blind, randomized, comparator-led study to evaluate the safety, immunogenicity and
efficacy of GlaxoSmithKline Biologicals' HPV-16/18 AS04 vaccine administered intramuscularly according to a three-
dose schedule (0, 1, 6 month) in healthy adult female subjects aged 26 years and above.”
Characteristics:
72 allocated to 76 different centres in Australia, Canada, Mexico, the Netherlands, Peru,
Philippines, Portugal, Russia, Singapore, Thailand, the United Kingdom and the United States.
Participants were excluded from the trial if they previously had received any of the adjuvants that
were part of the HPV vaccine or comparator or had a history of any neurological or immunological
disorder.
Outcomes All-cause mortality, HPV-related cancer and referral procedures, fatal and serious harms, new onset
diseases (reported as ‘medically significant conditions’ for the whole study period) and general
harms (reported as ‘solicited’ and ‘unsolicited’ general harms 7- and 30-days post-vaccination,
respectively). HPV-015 only reported combined surrogate outcomes (e.g., CIN2+) for benefits.
Notes None.

Author’s
judgment
generation (selection using a standard SAS® (Statistical Analysis System) program and was
bias) used to number the vaccines. A randomization blocking scheme (1:1
maintained."
Allocation concealment Low risk of bias “The treatment allocation at the investigator site was performed using
(selection bias) a central randomisation system on internet (SBIR)…When a subject
was eligible and informed consent was obtained, the person in charge
of the vaccination accessed SBIR…the randomisation system
determined the treatment number to be used for the subject.”
Blinding of participants Low risk of bias "This study was performed in a double-blinded manner: all subjects,
and personnel investigators and study staff participating in this study were blinded to
(performance bias) the individual subject treatment (HPV vaccine or comparator) and HPV
DNA PCR results. GSK personnel directly involved in the conduct of the
study were also blinded to the individual subject treatment. The study
blind was maintained until the end of the study."
Blinding of outcome Low risk of bias "This study was performed in a double-blinded manner: all subjects,
assessment (detection investigators and study staff participating in this study were blinded to
bias) the individual subject treatment (HPV vaccine or comparator) and HPV
DNA PCR results. GSK personnel directly involved in the conduct of the
study were also blinded to the individual subject treatment. The study
blind was maintained until the end of the study."
(reporting bias) investigator judged serious harms in the follow-up (i.e., "From Visit 12
to Visit 17, only SAEs [serious harms] related to vaccination, other GSK
medications or study procedures, any fatal SAE and AEs/SAEs leading
to premature discontinuation of the study"). The pathologists
categorized abnormal histology as, “…CIN2, CIN3, AIS, MALIGN [i.e.,
cancer],” but the trial only reported CIN2+, although there were three
cases of HPV-related cancers in the HPV vaccine group and one in the
comparator group. The cancers were tabulated as serious harms and
not mentioned elsewhere in the trial report.
and reporting.
Page 312 of 637

HPV-023 (NCT00518336)
Title: “A blinded long-term follow-up study of the efficacy of candidate HPV-16/18 L1 VLP AS04 vaccine in young adult
women in Brazil vaccinated in the phase IIb, double-blind, multicentre primary study HPV-001 and having participated
in the follow-up study HPV-007.”
Characteristics:
allocated to five different centres in Brazil. Participants were excluded from the trial if they
previously had received any of the adjuvants that were part of the HPV vaccine or comparator or
had a history of any neurological disease.
Outcomes All-cause mortality and fatal and serious harms. No benefit outcomes were eligible or happened
(e.g., “During the first year of follow-up in study HPV-023, there were no new cases of VIN or VaIN
associated with oncogenic types") during the follow-up. Serious harms were not reported for
individual MedDRA categories. New onset diseases and general harms were not reported.
Notes None.

Author’s
judgment
Random sequence Low risk of bias "No treatment was given in study HPV-023. The randomization that
generation (selection occurred in primary study HPV-001 was maintained in follow-up study
bias) HPV-007 and the current follow-up study HPV-023."
Allocation concealment Low risk of bias “…the randomisation system determined the vaccine number to be
(selection bias) used for the subject. The vaccine number was used as patient
identification number (PID) for all data collected on the subject under
the study. The investigator (or designate) used the eCRF of the RDE
system to document the patient identification number (PID).”
Blinding of participants Low risk of bias "Blinding was to be maintained for all subjects and investigators and
and personnel their study staff participating in this study with regard to the individual
(performance bias) subject treatment (vaccine or placebo) assignments allocated in study
HPV-001. GSK personnel directly involved in the conduct of this study
(e.g. site monitors, medical monitors, laboratory personnel, etc.) were
also blinded to the subjects’ treatment assignments. The site monitors,
medical monitors, as well as laboratory and data validation personnel
did not have access to any individual unblinded subject data listings
from study HPV-001 or HPV-007 from this cohort."
Blinding of outcome Low risk of bias "Blinding was to be maintained for all subjects and investigators and
assessment (detection their study staff participating in this study with regard to the individual
bias) subject treatment (vaccine or placebo) assignments allocated in study
HPV-001. GSK personnel directly involved in the conduct of this study
(e.g. site monitors, medical monitors, laboratory personnel, etc.) were
also blinded to the subjects’ treatment assignments. The site monitors,
medical monitors, as well as laboratory and data validation personnel
did not have access to any individual unblinded subject data listings
from study HPV-001 or HPV-007 from this cohort."
Selective reporting High risk of bias The outcome data contained redactions. The trial reported serious
(reporting bias) harms for the follow-up period, but these were not included for
individual MedDRA categories. New onset diseases and general harms
were not reported.
and reporting.
Page 313 of 637

HPV-029 (NCT00578227)
Title: “A phase IIIb, randomized, open, multicentre study to evaluate the immunogenicity and safety of
GlaxoSmithKline Biologicals’ HPV-16/18 L1 VLP AS04 vaccine co-administered with GlaxoSmithKline Biologicals’
inactivated hepatitis A and hepatitis B vaccine adsorbed (Twinrix® Paediatric) in healthy female subjects aged 9 - 15
years.”
Characteristics:
Methods Randomized, parallel group, open trial that was 12 months long.
allocated to 21 different centres in Canada, Denmark, Hungary and Sweden. Participants were
excluded from the trial if they previously had received any of the adjuvants that were part of the
HPV vaccine or comparator or had a history of any immunological disorder.
Interventions Cervarix (0.5 millilitre) vs. Twinrix (hepatitis A and hepatitis B vaccine, 0.5 millilitre) given
Notes The control group was vaccinated with the HPV vaccine at the end of follow-up ("After completion
of the safety follow-up and outside the study protocol, vaccination with Twinrix® Paediatric will be
offered to the HPV group and vaccination with GSK Biologicals’ HPV-16/18 L1 VLP AS04 vaccine will
be offered to the HAB group, if commercially available in the participating country”).

Author’s
judgment
generation (selection using a standard SAS (Statistical Analysis System) program, to number
bias) the vaccines. A randomization blocking scheme (1:1:1 ratio) was used
to ensure that balance between treatments was maintained: a
treatment number identified uniquely the vaccine doses administered
to the same subject."
(selection bias) a central randomization system on Internet (SBIR) ... The person in
charge of the vaccination accessed the randomization system on
Internet.”
Blinding of participants High risk of bias "This was an open study and therefore the subject and investigator
and personnel were aware of the group allocated and the treatment given."
(performance bias)
Blinding of outcome High risk of bias "This was an open study and therefore the subject and investigator
assessment (detection were aware of the group allocated and the treatment given."
bias)
(reporting bias)
and reporting.
Page 314 of 637

HPV-030 (NCT00652938)
Title: “A phase IIIb, randomized, open, multicentre study to evaluate the immunogenicity and safety of
GlaxoSmithKline Biologicals’ HPV-16/18 L1 VLP AS04 vaccine (Cervarix™) co-administrated with GlaxoSmithKline
Biologicals’ Hepatitis B vaccine (Engerix-B™) in healthy female subjects aged 9 - 15 years.”
Characteristics:
Methods Randomized, parallel group, open trial that was 12 months long.
allocated to seven different centres in the Netherlands and Sweden. Participants were excluded
from the trial if they previously had received any of the adjuvants that were part of the HPV vaccine
or comparator or had a history of any neurological or immunological disorder.
Interventions Cervarix (0.5 millilitre) vs. Engerix (hepatitis B vaccine, 0.5 millilitre) given intramuscularly at 0, 1
and 6 months.
Outcomes All-cause mortality, fatal harms, serious harms, new onset diseases (reported as ‘medically
significant conditions’ for the whole study period), general harms (reported as ‘solicited’ and
were eligible.
Notes The control group was vaccinated with the HPV vaccine at the end of follow-up ("After completion
of the safety follow-up and outside the study protocol, Engerix-B will be offered to the HPV group
and Cervarix will be offered to the HepB [Engerix-B] group, if commercially available in the
participating country”).

Author’s
judgment
bias) used to number the vaccines. A randomization blocking scheme (1:1:1
maintained: a treatment number was to identify uniquely the vaccine
doses to be administered to the same subject."
Allocation concealment Low risk of bias “The treatment allocation at the investigator site was to be performed
(selection bias) using a central randomization system on Internet (SBIR) ... The person
in charge of the vaccination was to access the randomization system
on Internet.”
Blinding of participants High risk of bias "This was an open study and therefore the subject and investigator
and personnel were aware of the group allocated and the treatment given."
(performance bias)
Blinding of outcome High risk of bias "This was an open study and therefore the subject and investigator
assessment (detection were aware of the group allocated and the treatment given."
bias)
(reporting bias)
and reporting.
Page 315 of 637

HPV-031 (NCT00344032)
Title: “A phase IIIb, double-blind, randomized, comparator-led study to evaluate the immunogenicity and safety of
GlaxoSmithKline (GSK) Biologicals’ HPV-16/18 L1 VLP AS04 vaccine administered intramuscularly according to a 0, 1, 6
months schedule in healthy Indian female subjects aged 18 – 35 years.”
Characteristics:
allocated to four different centres in India. Participants were excluded from the trial if they
had a history of any immunological disorder.
were eligible.
Notes None.

Author’s
judgment
Random sequence Low risk of bias "The randomization was performed at GSK Biologicals, Rixensart, using
generation (selection a standard SASâ (Statistical Analysis System) program. A
bias) randomization blocking scheme (1:1 ratio) was used to ensure that the
balance between treatments was maintained: throughout the study, a
single treatment number identified uniquely the vaccine/placebo
doses to be administered to the same subject."
Allocation concealment Low risk of bias This was not explicitly described, but the trial was allegedly “double
(selection bias) blind,” and we therefore assumed that the allocation concealment
probably was adequate.
Blinding of participants Low risk of bias "This study was conducted in a double-blind manner. All subjects, the
and personnel investigator and all study personnel involved in clinical evaluation of
(performance bias) subjects were blinded to the individual subject treatment allocation
(HPV-16/18 L1 VLP AS04 vaccine or placebo). GSK personnel directly
involved in the conduct of the study were also blinded to the
individual subject treatment. Blinding was maintained for the entire
study period until the database was frozen."
Blinding of outcome Low risk of bias "This study was conducted in a double-blind manner. All subjects, the
assessment (detection investigator and all study personnel involved in clinical evaluation of
bias) subjects were blinded to the individual subject treatment allocation
(HPV-16/18 L1 VLP AS04 vaccine or placebo). GSK personnel directly
involved in the conduct of the study were also blinded to the
individual subject treatment. Blinding was maintained for the entire
study period until the database was frozen. "
(reporting bias)
and reporting.
Page 316 of 637

HPV-032 (NCT00316693)
Title: “A double-blind (observer-blind), randomized, comparator-led, phase II study to assess the efficacy,
immunogenicity and safety of GlaxoSmithKline Biologicals HPV-16/18 L1 VLP AS04 vaccine administered
intramuscularly according to a 0, 1, 6-month schedule in healthy Japanese female subjects aged 20-25 years.”
Characteristics:
Participants 1,040 healthy females (519 in the HPV vaccine group and 521 in the comparator group) age 19-25
allocated to 13 different centres in Japan. Participants were excluded from the trial if they
had a history of any neurological or immunological disorder.
Interventions Cervarix (0.5 millilitre) vs. Aimmugen (hepatitis A vaccine, 0.5 millilitre) given intramuscularly at 0, 1
and 6 months.
Outcomes All-cause mortality and fatal harms. No benefit outcomes were eligible. Serious harms were not
reported for individual MedDRA categories. New onset diseases and general harms were not
reported.
Notes None.

Author’s
judgment
Random sequence Low risk of bias "Randomization of supplies A randomization list was generated at GSK
generation (selection Biologicals, Rixensart, using a standard SAS (Statistical Analysis System)
bias) program and was used to number the vaccines. A randomization
blocking scheme (1:1 ratio) was used to ensure that balance between
treatments was maintained: a treatment number identified uniquely
the vaccine doses to be administered to the same subject."
Allocation concealment Low risk of bias “The treatment allocation at the investigator/co-investigator site was
(selection bias) performed using a central randomization system on Internet (SBIR) ...
When a subject was eligible and informed consent had been obtained,
the person in charge of randomization accessed SBIR. Upon providing
the subject number, the randomization system used the minimization
algorithm to determine the treatment number used for the subject.”
Blinding of participants High risk of bias "The study was double-blind (observer-blind) such that subjects and all
and personnel study personnel involved in clinical evaluation of subjects were blinded
(performance bias) to treatment allocation. Study personnel involved in vaccine
administration were aware of treatment assignment but only
participated in vaccine/comparator administration during the study. It
was an observer blind study as the vaccines have different
presentations. All GSK personnel, investigators/co-investigators and
their study staff as well as study subjects remained blinded to the
individual subject assignments and will remain such until the
conclusion of the study (Month 24)."
Blinding of outcome High risk of bias "The study was double-blind (observer-blind) such that subjects and all
assessment (detection study personnel involved in clinical evaluation of subjects were blinded
bias) to treatment allocation. Study personnel involved in vaccine
administration were aware of treatment assignment but only
participated in vaccine/comparator administration during the study. It
was an observer blind study as the vaccines have different
presentations. All GSK personnel, investigators/co-investigators and
their study staff as well as study subjects remained blinded to the
individual subject assignments and will remain such until the
conclusion of the study (Month 24)."
Incomplete outcome High risk of bias 941 of 1,040 participants completed follow-up (90%) The individual
Selective reporting High risk of bias The outcome data contained redactions. New onset diseases and
(reporting bias) general harms were not reported.
and reporting.
HPV-033 (NCT00290277)
Page 317 of 637

Title: “A phase III, double-blind, randomized, comparator-led study to evaluate the immunogenicity and safety of GSK
Biologicals’ HPV-16/18 L1 VLP AS04 vaccine administered intramuscularly according to a 1, 1, 6 month schedule in
healthy female subjects aged 10 – 14 years.”
Characteristics:
allocated to 8 different centres in South Korea. Participants were excluded from the trial if they
6 months. The batch numbers of the HPV vaccine and comparator were redacted.
were eligible.
Notes None.

Author’s
judgment
Random sequence Low risk of bias Central randomization system.
bias)
Allocation concealment Low risk of bias “The treatment allocation was performed using a central
(selection bias) randomization system on Internet (SBIR) ... When a subject was eligible
and informed consent had been obtained, the person in charge of
randomization accessed SBIR.”
Blinding of participants High risk of bias “All subjects and study personnel involved in study conduct including
and personnel safety assessments and not involved in preparation and administration
(performance bias) of study vaccines were blinded to the individual subject
treatment…This was due to differences in the appearance of the HPV-
16/18 vaccine and HAV comparator vaccine.”
Blinding of outcome Low risk of bias “ ... study personnel involved in study conduct including safety
assessment (detection assessments and not involved in preparation and administration of
bias) study vaccines were blinded to the individual subject treatment.”
(reporting bias)
and reporting.
Page 318 of 637

HPV-035 (NCT00306241)
Title: “A Phase III, Double-blind, Randomized, Controlled Study to Evaluate Immunogenicity & Safety of GSK
Biologicals' HPV-16/18 L1 VLP AS04 Vaccine, Administered Intramuscularly (0, 1, 6 Month Schedule) in Healthy
Females Aged 18 - 35 Years.”
Characteristics:
allocated in one centre in Hong Kong. Participants were excluded from the trial if they previously
had received any of the adjuvants that were part of the HPV vaccine or comparator or had a history
of any neurological or immunological disorder.
intramuscularly at 0, 1 and 6 months. The batch numbers of the HPV vaccine and comparator were
redacted.
were eligible.
Notes The control group was vaccinated with the HPV vaccine at the end of follow-up.

Author’s
judgment
Random sequence Low risk of bias Central randomization list.
bias)
Allocation concealment Low risk of bias “allocation at the investigator site was performed using a central
(selection bias) randomisation system on Internet (SBIR).”
Blinding of participants Low risk of bias This was not explicitly described, but the trial was allegedly “double
and personnel blind,” and we therefore assumed that the blinding of participants and
(performance bias) personnel probably was adequate.
Blinding of outcome Low risk of bias This was not explicitly described, but the trial was allegedly “double
assessment (detection blind,” and we therefore assumed that the blinding of outcome
bias) assessors probably was adequate.
Incomplete outcome High risk of bias 294 of 300 participants completed follow-up (98%) The individual
(reporting bias)
and reporting.
Page 319 of 637

HPV-038 (NCT00485732)
Title: “A phase IIIb, double-blind, randomized, comparatorled study to evaluate the immunogenicity and safety of
GlaxoSmithKline (GSK) Biologicals HPV-16/18 L1 VLP AS04 vaccine, administered intramuscularly according to a 0, 1, 6
month schedule in healthy female subjects aged 15 – 25 years.”
Characteristics:
allocated to six different centres in South Korea. Participants were excluded from the trial if they
were eligible.
Notes None.

Author’s
judgment
maintained: throughout the study a single treatment number uniquely
identified the vaccine doses to be administered to the same subject."
(selection bias) a central randomisation system on Internet (SBIR). The randomisation
algorithm used a minimisation procedure. The person in charge of the
vaccination accessed the randomisation system on Internet. Upon
providing a subject number and the age for the subject, the
randomisation system used the minimisation algorithm accounting for
centre and age.”
Blinding of participants Low risk of bias "The study was performed in a double-blind manner. All subjects and
and personnel study personnel were blinded to the individual subject treatment (HPV
(performance bias) vaccine or placebo). GSK Biologicals’ personnel directly involved in the
conduct of the study were also blinded to the individual subject
treatment. Blinding was maintained for the whole study period (until
the last subject enrolled completed the last visit at Month 7) and until
the database was frozen."
Blinding of outcome Low risk of bias "The study was performed in a double-blind manner. All subjects and
assessment (detection study personnel were blinded to the individual subject treatment (HPV
bias) vaccine or placebo). GSK Biologicals’ personnel directly involved in the
conduct of the study were also blinded to the individual subject
treatment. Blinding was maintained for the whole study period (until
the last subject enrolled completed the last visit at Month 7) and until
the database was frozen."
(reporting bias)
and reporting.
Page 320 of 637

HPV-040 (NCT00534638)
Title: “A phase III/IV, community-randomized, comparator-led study to evaluate the effectiveness of two vaccination
strategies using GlaxoSmithKline Biologicals’ HPV-16/18 L1 VLP AS04 vaccine in reducing the prevalence of HPV-16/18
infection when administered intramuscularly according to a 0, 1, 6-month schedule in healthy female and male study
participants aged 12 – 15 years.”
Characteristics:
Methods Cluster-randomized, community stratified, open trial that was 72 months long.
Participants 32,176 healthy males and females (14,838 in the HPV vaccine group and 17,338 in the comparator
group) age 12-16 allocated to 250 different clusters in 33 communities in Finland.
Interventions Cervarix (0.5 millilitre) vs. Engerix-B (hepatitis B vaccine, 0.5 millilitre) given intramuscularly at 0, 1
and 6 months.
Outcomes All-cause mortality, fatal and serious harms, general harms (reported as ‘solicited’ and ‘unsolicited’
New onset diseases were reported as ‘medically significant conditions,’ but the results were
redacted.
Notes The control group was vaccinated with the HPV vaccine at the end of follow-up ("Cross-over
vaccination will be offered to all female study participants immunized in the trial, when invited for
the effectiveness evaluation phase (i.e., subjects that had received HPV vaccine will be offered
vaccination with Hepatitis B vaccine and subjects that received Hepatitis B vaccine will be offered
HPV vaccine). Similarly, males who received HPV vaccine during the Immunization phase will be
invited to receive cross-over vaccination with Hepatitis B vaccine. After study completion, GSK’s
HPV vaccine will be offered to male study participants who have received the Hepatitis B vaccine
during the immunization phase if the results from the study demonstrate that male vaccination
provides a meaningful benefit on the prevalence of HPV infection in females and if the vaccine is
licensed for use in males”).

Author’s
judgment
Random sequence Low risk of bias “A randomization list was generated at GSK Biologicals, Rixensart,
generation (selection using a standard SAS® (Statistical Analysis System) program and was
maintained: a treatment number identified uniquely the vaccine doses
to be administered to the same study participant. The vaccine doses
were distributed to each study centre, respecting the randomization
block size.”
Allocation concealment Low risk of bias The treatment allocation at the investigator site was performed using
(selection bias) SBIR. The randomization algorithm used a minimization procedure. The
person in charge of the vaccination was to access the randomization
system on Internet.
Blinding of participants High risk of bias "All study participants know in which intervention arm their
and personnel community has been assigned."
(performance bias)
Blinding of outcome High risk of bias This was not explicitly described, but the trial was allegedly “open,”
assessment (detection and we therefore assumed that the blinding of outcome assessors
bias) probably was inadequate.
Incomplete outcome High risk of bias 31,901 of 32,176 participants completed follow-up (99%). The
data (attrition bias) individual participant data was not included.
Selective reporting High risk of bias The outcome data contained redactions. Only a limited and varied
(reporting bias) number of participants were included in the harm analyses, e.g.,
serious harms for 3,703 participants or the serious harms that were
judged vaccine-related by investigators were reported ("Any SAEs
reported to the investigator and considered by the investigator as
possibly related to vaccination were to be reported to GSK
Biologicals”) and only general harms for 1,628 participants.
and reporting.
Page 321 of 637

HPV-058 (NCT00996125)
Title: “A phase III, double-blind, randomized, comparator-led study to evaluate the immunogenicity and safety of GSK
Biologicals’ HPV-16/18 L1 VLP AS04 vaccine administered intramuscularly according to a 0, 1, 6-month schedule in
healthy Chinese female subjects aged 9-17 years.”
Characteristics:
allocated to one centre in China. Participants were excluded from the trial if they previously had
any neurological or immunological disorder.
conditions’ for the whole study period), general harms (reported as ‘solicited’ and ‘unsolicited’
Notes None.

Author’s
judgment
Random sequence Low risk of bias Central MATEX randomization.
bias)
(selection bias) a central randomisation system on Internet (SBIR) … the study staff
accessed the randomisation system on Internet.”
Blinding of participants Low risk of bias "Blinding was maintained for all subjects and investigators and their
and personnel study staff"
(performance bias)
Blinding of outcome Low risk of bias "Blinding was maintained for all subjects and investigators and their
assessment (detection study staff"
bias)
(reporting bias)
and reporting.
Page 322 of 637

HPV-063 (NCT00929526)
Title: “An open, multi-centre, long term extension study to the primary vaccination phase II study HPV-032 to assess
the efficacy of HPV-16/18 L1 VLP AS04 vaccine in the prevention of HPV-16 and/or HPV-18 associated cervical
intraepithelial neoplasia (CIN) and cervical cancer, to assess the immunogenicity of HPV-16/18 L1 VLP AS04 vaccine
and to assess safety up to 48 months after administration of the first dose of HPV-16/18 L1 VLP AS04 vaccine to
healthy Japanese women vaccinated at 20 - 25 years of age.”
Characteristics:
Methods Randomized, parallel group, open trial that was 12 months long (follow-up to trial HPV-032).
Participants 752 healthy females (375 in the HPV vaccine group and 377 in the comparator group) age 22-29 (at
the time of this follow-up) allocated to 13 different centres in Japan. Participants had been
excluded from the trial if they previously had received any of the adjuvants that were part of the
HPV vaccine or comparator or had a history of any neurological or immunological disorder.
Interventions Cervarix (0.5 millilitre) vs. Aimmugen (hepatitis A vaccine, 0.5 millilitre) given intramuscularly at 0, 1
and 6 months.
significant conditions’ for the whole study period). No benefit outcomes were eligible except for
combined cervical surrogate outcomes (i.e., CIN2+ and CIN3+). General harms were reported in trial
HPV-032.
Notes None.

Author’s
judgment
Random sequence Low risk of bias "Randomization of supplies A randomization list was generated at GSK
generation (selection Biologicals, Rixensart, using a standard SAS (Statistical Analysis System)
bias) program and was used to number the vaccines. A randomization
blocking scheme (1:1 ratio) was used to ensure that balance between
treatments was maintained: a treatment number identified uniquely
the vaccine doses to be administered to the same subject."
Allocation concealment Low risk of bias “The treatment allocation at the investigator/co-investigator site was
(selection bias) performed using a central randomization system on Internet (SBIR) ...
When a subject was eligible and informed consent had been obtained,
the person in charge of randomization accessed SBIR. Upon providing
the subject number, the randomization system used the minimization
algorithm to determine the treatment number used for the subject.”
Blinding of participants High risk of bias "The study was open such that the subjects, the investigator/co-
and personnel investigator and all study staff involved in clinical evaluation of
(performance bias) subjects were aware of the previous treatment allocation in study
HPV-032. However, to maintain the scientific integrity of the study, the
laboratories in charge of the laboratory testing remained blinded to
the treatment."
Blinding of outcome High risk of bias "The study was open such that the subjects, the investigator/co-
assessment (detection investigator and all study staff involved in clinical evaluation of
bias) subjects were aware of the previous treatment allocation in study
HPV-032. However, to maintain the scientific integrity of the study, the
laboratories in charge of the laboratory testing remained blinded to
the treatment."
(reporting bias)
and reporting.
Page 323 of 637

HPV-069 (NCT01277042)
Title: “A phase III, observer-blind, randomized, comparator-led study to evaluate the immunogenicity and safety of
GlaxoSmithKline (GSK) Biologicals’ HPV-16/ 18 L1 VLP AS04 vaccine administered intramuscularly according to a 0, 1,
6-month schedule in healthy adult Chinese female subjects aged 26-45 years.”
Characteristics:
Participants 1,212 healthy females (606 in the HPV vaccine group and 606 in the comparator group) age 26-46
allocated to one centre in China. Participants were excluded from the trial if they previously had
any immunological disorder.
Interventions Gardasil (0.5 millilitre) vs. Engerix-B (hepatitis B vaccine, 0.5 millilitre) given intramuscularly at 0, 2
and 6 months.
were eligible.
Notes None.

Author’s
judgment
Random sequence Low risk of bias "The randomization was performed at GSK Biologicals, Rixensart, using
generation (selection MATEX, a program developed for use in Statistical Analysis System
bias) (SAS®) (Cary, NC, USA) by GSK Biologicals. The vaccine doses were
distributed to the study centre while respecting the randomization
block size."
(selection bias) a central randomisation system on internet (SBIR) … the study staff in
charge of the vaccination accessed SBIR.”
Blinding of participants High risk of bias "Because of the difference in volume and appearance between the
and personnel HPV vaccine and Engerix-B, the vaccines were prepared and
(performance bias) administrated by qualified medical personnel not otherwise involved in
the conduct of the study or in the assessment of symptoms. Study staff
involved in the assessment of subjects was blinded."
Blinding of outcome Low risk of bias "Data was collected in an observer-blind manner. By observer-blind, it
assessment (detection was meant that during the course of the study, the vaccine recipient
bias) and those responsible for the evaluation of any study endpoint (e.g.
safety, reactogenicity and immunogenicity) were all unaware of which
vaccine was administered. To do so, vaccine preparation and
administration was done by authorised medical personnel who did not
participate in any of the study clinical evaluation assays. Blinding was
maintained for all subjects and investigators and their study staff
participating in this study with regard to the individual subject
treatment (HPV vaccine or comparator vaccine) assignments allocated
in this study during the entire study period. GSK personnel directly
involved in the conduct of this study (e.g. site monitors, medical
monitors, laboratory personnel, etc.) were also blinded to the subject’s
treatment assignments during the entire study period. The GSK
statistician and authorised staff remained blinded until the complete
freezing of the database."
(reporting bias)
and reporting.
Page 324 of 637

V501-005 (NCT00365378)
Title: “Study of Pilot Manufacturing Lot of HPV 16 Virus-Like Particle (VLP) Vaccine in the Prevention of HPV 16
Infection in 16- to 23-Year-Old Females.”
Characteristics:
25 allocated to 16 different centres in the United States. Participants were excluded from the trial if
they previously had received any of the adjuvants that were part of the HPV vaccine or comparator
or had a history of any immunological disorder.
Interventions HPV 16 vaccine (0.5 millilitre) vs. amorphous aluminium hydroxyphosphate sulphate (AAHS, 0.225
milligram in 0.5 millilitre saline) given intramuscularly at 0, 2 and 6 months. The batch numbers of
the HPV vaccine and comparator were redacted.
Outcomes All-cause mortality, fatal and serious harms (reported 14 days post-vaccination), new onset
diseases (reported as ‘new medical history’ for the whole study period), general harms (reported as
‘systemic clinical adverse evens’ 14 days post-vaccination). No benefit outcomes were eligible in
the acquired clinical study report, but according to the table of contents eligible benefit outcomes
were reported.
Notes None.

Author’s
judgment
Random sequence Low risk of bias "A randomization schedule was generated by the Clinical Biostatistics
generation (selection department of MRL"
bias)
Allocation concealment Low risk of bias “… computer-generated allocation schedule … study subjects, study
(selection bias) staff, and all Merck personnel were blinded to individual treatment
allocation throughout the duration of the study.”
Blinding of participants Low risk of bias "The study was conducted under double-blind (with in house blinding)
and personnel conditions. Thus, study subjects, study staff and all Merck Sharp &
(performance bias) Dohme personnel were blinded to individual treatment allocation
throughout the duration of the study."
Blinding of outcome Low risk of bias "The study was conducted under double-blind (with in house blinding)
assessment (detection conditions. Thus, study subjects, study staff and all Merck Sharp &
bias) Dohme personnel were blinded to individual treatment allocation
throughout the duration of the study."
(reporting bias) serious harms "…within 14 days following any vaccination visit.”
Other High risk of bias The trial was funded by Merck Sharp & Dohme and used inadequate
design and reporting.
Page 325 of 637

V501-013 (NCT00092521)
Title: “A Study to Evaluate the Efficacy of Quadrivalent HPV Vaccine in Reducing the Incidence of HPV 6-, 11-, 16- and
18-Related CIN, AIS and Cervical Cancer and HPV 6-, 11-, 16- and 18-Related External Genital Warts, Vulvar
Intraepithelial Neoplasia Vaginal Intraepithelial Neoplasia, Vulvar Cancer and Vaginal Cancer in 16- to 23-Year-Old
Women.”
Characteristics:
45 allocated to 62 different centres in Australia, Brazil, Canada, Colombia, Czech Republic,
Germany, Hong Kong, Italy, Mexico, New Zealand, Peru, Puerto Rico, Russia, Thailand, the United
Kingdom and the United States. Participants were excluded from the trial if they previously had
received any of the adjuvants that were part of the HPV vaccine or comparator, had a history of any
neurological disease or had a history of any immunological disorder.
Interventions Gardasil (0.5 millilitre) vs. amorphous aluminium hydroxyphosphate sulphate (AAHS, 0.225
irrespective of HPV-type, external genital lesions, HPV-related referral procedures, fatal and serious
harms (reported 14 days post-vaccination) and new onset diseases (reported as ‘new medical
history’ for the whole study period). The trial did not report general harms.
Notes The trial shortened the study period from 48 to 45 months. The control group was vaccinated with
the HPV vaccine at the end of follow-up.

Author’s
judgment
Random sequence Low risk of bias Described as “randomized” but not how the randomization was performed
generation (selection (e.g., computer generated).
bias)
Allocation concealment Low risk of bias “Only the team responsible for the immunogenicity and safety analyses had
(selection bias) access to the allocation schedule and unblinded database … individual
treatment allocations were not revealed … Allocation schedules were
generated by the Clinical Biostatistics department of MRL … The clinical, data
management and statistics personnel at the Sponsor remained blinded to
individual vaccination allocation through the completion of data review”
Blinding of participants Low risk of bias "Protocol 013 was double-blind, operating under in-house blinding
and personnel procedures. In addition to the subject and the investigator, all laboratory
(performance bias) personnel conducting the clinical assays, the clinical, statistical and data
management data review team and the Pathology Panel members were
blinded. This being an ongoing study, the investigator, including other study
personnel, subjects, laboratory personnel and Pathology Panel members are
to remain blinded until all subjects have completed the study at the Month 48
visit and the data are screened for completeness and accuracy."
Blinding of outcome Low risk of bias "Protocol 013 was double-blind, operating under in-house blinding
assessment (detection procedures. In addition to the subject and the investigator, all laboratory
bias) personnel conducting the clinical assays, the clinical, statistical and data
management data review team and the Pathology Panel members were
blinded.”
Selective reporting High risk of bias The outcome data contained redactions. The trial only reported serious harms
(reporting bias) 14 days post-vaccination ("Investigators were instructed to report any serious
adverse experience, including death due to any cause, occurring in any subject
from the time the consent form was signed through 14 days following the first
vaccination and from the time of any subsequent vaccinations through 14 days
thereafter, whether or not related to the investigational product").
Other High risk of bias The trial was funded by Merck Sharp & Dohme and used inadequate design
and reporting.
Page 326 of 637

V501-015 (NCT00092534)
Title: “A Randomized Worldwide, Placebo-Controlled, Double-Blind Study to Investigate the Safety, Immunogenicity
and Efficacy on the Incidence of HPV 16-/18-Related CIN 2/3 or Worse of the Quadrivalent HPV (Types 6, 11, 16, 18] Ll
Virus-Like Particle (VLP) Vaccine in 16- to 23-Year-Old Women- The FUTURE II Study (Females United to Unilaterally
Reduce Endo/Ectocervical Disease).”
Characteristics:
Participants 12,167 healthy females (6,087 in the HPV vaccine group and 6,080 in the comparator group) age
15-26 allocated to 90 different centres in Brazil, Colombia, Denmark, Finland, Iceland, Mexico,
Norway, Peru, Poland, Singapore, Sweden, the United Kingdom and the United States. Participants
were excluded from the trial if they had a history of any immunological disorder.
history’ for the whole study period). V501-015 did not report individual general harms.
Notes The trial shortened the study period from 48 to 36 months. The control group was vaccinated with
the HPV vaccine at the end of follow-up.

Author’s
judgment
Random sequence Low risk of bias Described as “randomized” but not how the randomization was performed
generation (selection (e.g., computer generated).
bias)
Allocation concealment Low risk of bias “Allocation schedules were generated by the Clinical Biostatistics department
(selection bias) … An Interactive Voice Response System (IVRS) was used to allocate subjects.
At the enrollment visit, the IVRS assigned the subject an AN [allocation
number] from those allocated to the study site [16.1.7.1]. The IVRS
subsequently assigned the appropriate vial identification number for HPV
vaccine/placebo that corresponded to the subject's AN and vaccination
group.”
Blinding of participants Low risk of bias "Protocol 015 was conducted under double-blind (with in-house blinding)
and personnel procedures. In addition to the subject and the investigator, all laboratory
(performance bias) personnel conducting the clinical assays and the Pathology Panel members
were blinded. The investigator and his/her staff, subjects, laboratory
personnel and Pathology Panel members were to remain blinded until all
subjects completed the study, the data were screened for completeness and
accuracy and all protocol violators were identified."
Blinding of outcome Low risk of bias "Protocol 015 was conducted under double-blind (with in-house blinding)
assessment (detection procedures. In addition to the subject and the investigator, all laboratory
bias) personnel conducting the clinical assays and the Pathology Panel members
were blinded. The investigator and his/her staff, subjects, laboratory
personnel and Pathology Panel members were to remain blinded until all
subjects completed the study, the data were screened for completeness and
accuracy and all protocol violators were identified."
Selective reporting High risk of bias The outcome data contained redactions. The trial only reported serious harms
(reporting bias) 14 days post-vaccination ("All investigators were instructed to report any
serious adverse experience, including death due to any cause, occurring in any
subject from the time the consent form was signed through 14 days following
the first vaccination and from the time of any subsequent vaccinations
through 14 days thereafter, whether or not related to the investigational
product”).
Other High risk of bias The trial was funded by Merck Sharp & Dohme and used inadequate design
and reporting.
Page 327 of 637

V501-018 (NCT00092547)
Title: “A Safety and Immunogenicity Study of Quadrivalent HPV (Types 6, 11, 16, 18) L1 Virus-Like Particle (VLP)
Vaccine in Preadolescents and Adolescents (Base Study). A Long Term Immunogenicity, Safety and Effectiveness Study
of GARDASIL (Human Papillomavirus [Types 6, 11, 16, 18] Recombinant Vaccine) Among Adolescents Who Received
GARDASIL at 9-18 Years of Age (Extension Study).”
Characteristics:
Methods Randomized, parallel group, observer-blind trial where we obtained data for the first 18 of 96
months.
Participants 1,781 healthy males and females (1,184 in the HPV vaccine group and 597 in the comparator
group) age 9-16 allocated to 47 different centres in Colombia, Denmark, Mexico, Norway, Portugal,
Spain, Taiwan, Thailand, the United Kingdom and the United States. Participants were excluded
from the trial if they had a history of any immunological disorder.
Interventions Gardasil (0.5 millilitre) vs. carrier solution (yeast protein, sodium chloride, L-histidine, polysorbate
80 and sodium borate in 0.5 millilitre saline) given intramuscularly at 0, 2 and 6 months. The batch
numbers of the HPV vaccine and comparator were redacted.
diseases (reported as ‘new medical history’ for the whole study period), general harms (reported as
‘systemic clinical adverse evens’ 14 days post-vaccination). No benefit outcomes were eligible.
Notes None.

Author’s
judgment
Random sequence Low risk of bias Described as “randomized” but not how the randomization was performed (e.g.,
computer generated).
bias)
Allocation concealment Low risk of bias “An Interactive Voice Response System (IVRS) was used to allocate study subjects … At
the first visit, study personnel accessed the IVRS after a subject's parent/legal guardian
(selection bias)
had signed informed consent … The IVRS assigned the subject an allocation number (AN)
and a unique vial identification number for the vial of clinical material that the subject
should have received at that visit.”
Blinding of participants Low risk of bias "Because of the differences in the appearance of the quadrivalent HPV (Types 6, 11, 16,
18) LI VLP vaccine and placebo, administration of study material required both unblinded
and personnel
and blinded personnel to minimize bias. The subjects were seen first by the blinded
(performance bias) personnel, who provided subjects with an informational brochure and obtained eligible
subjects' consent/assent. The blinded and unblinded study personnel accessed IVRS,
which assigned the subject with an AN and a unique vial identification number for the
vial of clinical material that the subject should have received at that visit … The
unblinded study personnel were considered unblinded during the course of the study
because of their responsibilities in preparation and administration of the clinical
material. As a result, they were NOT involved with subject management. Subjects were
monitored by the blinded study personnel after vaccination was completed ... The
unblinded study personnel were responsible for obtaining the subject's AN from the
blinded study personnel, selecting the appropriate vial from the refrigerator,
withdrawing and verifying the volume and contents of the syringe. The unblinded
personnel recorded the subject's AN, date and their own initials onto the appropriate
worksheet. Only the unblinded personnel had the responsibility for documentation that
dealt with vaccine accountability. The unblinded study personnel wrapped the syringe
with the non-transparent label provided by the Sponsor to mask the slight difference in
appearance between quadrivalent HPV (Types 6, 11, 16, 18) L 1 VLP vaccine and placebo.
After completing administration of the study material, the unblinded study personnel
left the examination room immediately and had no further contact with the subject or
parent/legal guardian during the remainder of the visit or during the 14-day follow-up
period.”
Blinding of outcome Low risk of bias “Review of medical history and the physical examination was conducted by the blinded
personnel; the demographic information needed for vaccine/placebo preparation
assessment (detection
including the body weight, was provided to the unblinded personnel."
bias)
Incomplete outcome High risk of bias 1,680 of 1,781 participants completed follow-up until 18 months (94%). The individual
participant data was not included.
data (attrition bias)
Selective reporting High risk of bias The outcome data contained redactions. The trial only reported serious harms 14 days
post-vaccination ("Investigators were instructed to report any serious adverse
(reporting bias)
experience, including death due to any cause, occurring in any subject from the time the
consent form was signed through 14 days following the first vaccination and from the
time of any subsequent vaccinations through 14 days thereafter").
Other High risk of bias The trial was funded by Merck Sharp & Dohme and used inadequate design and
reporting.
Page 328 of 637

V501-019 (NCT00090220)
Title: “Safety, Immunogenicity and Efficacy of Gardasil (V501 (Human Papilloma Virus [Types 6, 11, 16, 18]
Recombinant Vaccine) in Mid-Adult Women - The FUTURE III (Females United to Unilaterally Reduce
Endo/Ectocervical Cancer) Study.”
Characteristics:
46 allocated to 38 different centres in Colombia, France, Germany, Philippines, Spain, Thailand and
the United States. Participants were excluded from the trial if they previously had received any of
the adjuvants that were part of the HPV vaccine or comparator or had a history of any neurological
disorder.
history’ for the whole study period). The trial did not report general harms.
Notes None.

Author’s
judgment
Random sequence Low risk of bias Described as “randomized” but not how the randomization was
generation (selection performed (e.g., computer generated).
bias)
Allocation concealment Low risk of bias “The Clinical Biostatistics department of MRL generated the allocation
(selection bias) schedule for randomizing study participants to the 2 treatment groups.
Throughout the study and across all study sites, there was no
repetition of an allocation number. Subjects were assigned an
allocation number at randomization.”
Blinding of participants Low risk of bias "…study personnel, study subjects, laboratory personnel (including the
and personnel Sponsor's laboratory staff) and Pathology Panel members remain
(performance bias) blinded until all subjects complete the study (through Month 48), the
data is screened for completeness and accuracy and all protocol
violators are identified."
Blinding of outcome Low risk of bias "This study was double-blind, operating under in-house blinding
assessment (detection procedures. In addition to the subject and the investigator, the
bias) laboratory personnel conducting the clinical assays, the clinical,
statistical and data management data review team and the Pathology
Panel members were blinded."
(reporting bias) serious harms 14 days post-vaccination ("Any SAE, including death due
to any cause, which occurred to any subject in the study from the time
the consent was signed through 14 days following the first vaccination
and from the time of any subsequent vaccinations through 14 days
thereafter").
Page 329 of 637

V501-020 (NCT00090285)
Title: “A Study to Evaluate the Efficacy of GARDASIL in Reducing the Incidence of HPV 6-, 11-, 16- and 18-Related
External Genital Warts, PIN, Penile, Perianal and Perineal Cancer and the Incidence of HPV 6-, 11-, 16- and 18-Related
Genital Infection in Young Men.”
Characteristics:
Participants 4,065 healthy males (2,032 in the HPV vaccine group and 2,033 in the comparator group) age 15-27
allocated to 71 different centres in Australia, Brazil, Canada, Costa Rica, Croatia, Finland, Germany,
Mexico, the Netherlands, Norway, Peru, Philippines, Portugal, South Africa, Spain, Sweden, Taiwan
and the United States. Participants were excluded from the trial if they had a history of any
immunological disorder.
Outcomes All-cause mortality, external genital lesions, HPV-related referral procedures, fatal and serious
harms (reported 14 days post-vaccination), new onset diseases (reported as ‘new medical history’
for the whole study period) and general harms (reported as ‘systemic clinical adverse evens’ 14
days post-vaccination). The only eligible primary histological benefit outcome was the combined
surrogate of PIN2+.
Notes None.

Author’s
judgment
Random sequence Low risk of bias Described as “randomized” but not how the randomization was
generation (selection performed (e.g., computer generated).
bias)
Allocation concealment Low risk of bias "The Clinical Biostatistics department of Merck Sharp & Dohme
(selection bias) Research Laboratories (MRL) generated the allocation schedule for
randomizing study participants."
Blinding of participants Low risk of bias "This study was double-blind, operating under in-house blinding
and personnel procedures. In addition to the subject and the investigator, the
(performance bias) laboratory personnel conducting the clinical assays, the clinical,
Panel members were blinded. Every attempt was to be made to
contact one of the individuals listed on the Sponsor Contact
Information page of the protocol prior to unblinding the vaccination
group for a specific case. If unblinding did occur (e.g., accidental
unblinding, emergency unblinding due to a serious adverse
experience), the investigator was instructed to promptly document the
circumstances and immediately notify the MRL clinical monitor listed
on the Sponsor Contact Information page of the protocol."
Blinding of outcome Low risk of bias "This study was double-blind, operating under in-house blinding
assessment (detection procedures. In addition to the subject and the investigator, the
bias) laboratory personnel conducting the clinical assays, the clinical,
Panel members were blinded."
(reporting bias) serious harms 14 days post-vaccination ("Investigators were instructed
to report any serious clinical adverse experience, including death due
to any cause, occurring in any subject from the time the consent was
signed through 14 days following the first vaccination and from the
time of any subsequent vaccinations through 14 days thereafter,
whether or not related to the investigational product”).
Page 330 of 637

V503-006 (NCT01047345)
Title: “A Phase III Randomized, International, Placebo-Controlled, Double-Blind Clinical Trial to Study the Tolerability
and Immunogenicity of V 503, a Multivalent Human Papillomavirus (HPV) L1 Virus-Like Particle (VLP) Vaccine, Given to
Females 12-26 Years of Age Who Have Previously Received GARDASIL™ (Protocol 006) 1.”
Characteristics:
allocated to 32 different centres in Australia, Canada, Colombia, Denmark, Hong Kong, Mexico,
Sweden and the United States. Participants were excluded from the trial if they previously had
any immunological disorder.
Interventions Gardasil 9 (0.5 millilitre) vs. saline placebo (0.5 millilitre) given intramuscularly at 0, 2 and 6 months.
diseases (reported as ‘new medical history’ for the whole study period) and general harms
(reported as ‘systemic clinical adverse evens’ 14 days post-vaccination). No benefit outcomes were
eligible.
Notes All participants were vaccinated with three doses of Gardasil more than 12 months before entering
the study.

Author’s
judgment
Random sequence Low risk of bias “The study used central randomization.”
bias)
Allocation concealment Low risk of bias “An Interactive Voice Response System (IVRS) was used to allocate
(selection bias) study subjects and assist with the vaccine supply management at the
study site. At the first visit, study personnel accessed the IVRS after the
subject had signed informed consent (or for minors, after a subject's
parent/legal guardian had signed informed consent and the subject
had signed assent) and after the subject had met all inclusion and none
of the exclusion criteria. The IVRS assigned the subject allocation
number (AN) and a unique vial identification number for the vial of
clinical material that the subject was to receive at that visit.”
Blinding of participants Low risk of bias "The subjects, investigators (and his/her staff), laboratory staff and
and personnel Sponsor remained blinded to subject vaccine allocation for the
(performance bias) duration of the study. Because the 9vHPV vaccine and normal saline
placebo can be visibly distinguished, the vaccine/placebo in this study
had to be prepared by an unblinded third party who was otherwise not
involved in the conduct of the study. The unblinded third party was
responsible for all procedures involving clinical supplies."
Blinding of outcome Low risk of bias "The subjects, investigators (and his/her staff), laboratory staff and
assessment (detection Sponsor remained blinded to subject vaccine allocation for the
bias) duration of the study. Because the 9vHPV vaccine and normal saline
placebo can be visibly distinguished, the vaccine/placebo in this study
had to be prepared by an unblinded third party who was otherwise not
involved in the conduct of the study. The unblinded third party was
responsible for all procedures involving clinical supplies."
(reporting bias) serious harms 14 days post-vaccination ("…serious adverse events
within 14 days following any vaccination visit, vaccine-related serious
adverse events observed during any time in the study…”).
Page 331 of 637

Additional file 3
Benefits and harms of the HPV vaccines: list of excluded studies
List of excluded industry funded studies

No. Reason for exclusion Manufacturer ID Study ID NCT ID Title
1 Ineligible comparator HPV-002 580299/002 Not identified Phase I, Open-label Study to Evaluate the Safety, Tolerability and
Immunogenicity of a Fourth Dose of Human Papillomavirus (HPV) DNA
Plasmid (VGX-3100) + Electroporation (EP) in Adult Females Previously
Immunized With VGX-3100
2 Ineligible comparator HPV-004 580299/004 NCT00693615 Study to Evaluate the Safety and Immunogenicity of MEDI-517, a Virus-
Like Particle Vaccine Against Human Papillomavirus Types 16 and 18,
When Formulated With Aluminum Hydroxide, AS04, or Without Adjuvant,
in Healthy Adult Female Volunteers
3 Ineligible comparator HPV-010 108933 NCT00423046 Phase IIIb, Observer-blind Study to Compare Immunogenicity of GSK
Biologicals' HPV-16/18 L1/AS04 Vaccine Versus Gardasil® [Quadrivalent
Human Papillomavirus (HPV-6,11,16,18 L1 VLP) Recombinant Vaccine
Merck & Co., Inc.]
4 Ineligible comparator HPV-012 107479 NCT00337818 A Long-term, Open, Follow-up of the Immunogenicity and Safety of
GlaxoSmithKline Biologicals' HPV-16/18 L1/AS04 Vaccine in Healthy
Female Subjects Vaccinated Either Pre- or Post-menarche in the Primary
Study
5 Ineligible comparator HPV-016 104772 NCT00250276 Assess Lot-to-lot Consistency of GSK Biologicals' HPV-16/18 L1/AS04
Vaccine Following Manufacturing Adjustments Administered
Intramuscularly According to a 0,1,6-mth Schedule in Healthy Female
Subjects (18-25 y)
6 Ineligible comparator HPV-018 107682 NCT00369824 A Randomized, Open Study to Evaluate the Safety and Immunogenicity of
GlaxoSmithKline Biologicals' HPV Vaccine Co-administered Intramuscularly
With Boostrix® and/or Menactra™ in Healthy Female Subjects Aged 11-18
Years
7 Ineligible comparator HPV-019 109823 NCT01031069 Safety and Immunogenicity of Cervarix™ in Human Immunodeficiency
Virus Infected Females
8 Ineligible comparator HPV-024 109628 NCT00546078 Safety and Immunogenicity Study of an Additional Dose of HPV Vaccine
(580299) in Young, Adult Women in North America.
9 Ineligible comparator HPV-026 111567 NCT00637195 Evaluation of the Immunogenicity and Safety of a Commercially Available
Vaccine When Co-administered With GlaxoSmithKline Biologicals' HPV
Vaccine (580299) in Healthy Female Subjects.
10 Ineligible comparator HPV-042 108464 NCT00426361 A Multicentre Study to Evaluate the Immunogenicity and Safety of GSK
Biologicals' HPV Vaccine (580299) Co-administered With Boostrix Polio
(dTpa-IPV) in Healthy Female Subjects Aged 10-18 Years
11 Ineligible comparator HPV-044 109179 NCT00552279 Immunogenicity and Safety Study of GlaxoSmithKline Biologicals' HPV
Vaccine GSK580299 Administered According to an Alternative Dosing
Schedule as Compared to the Standard Dosing Schedule in Young Female
Subjects Aged 15-25 Years
12 Ineligible comparator HPV-049 Not identified Not identified No title.
13 Ineligible comparator HPV-051 102115 NCT00231413 A Dose-range Study to Assess the Safety and Immunogenicity of a Novel
HPV Vaccine When Administered Intramuscularly According to a 3-dose
Schedule (0,1,6-month) in Healthy Adult Females (18-25 Years of Age)
14 Ineligible comparator HPV-051 108052 NCT00359619 A Long-term, Follow-up of the Immunogenicity and Safety of
follow-up GlaxoSmithKline Biologicals' Novel HPV Vaccine in Healthy Female Subjects
Vaccinated in the Primary Study
15 Ineligible comparator HPV-055 111758 NCT00849381 No title.
16 Ineligible comparator HPV-060 112772 NCT00947115 Follow-up Study to Evaluate the Long-term Immunogenicity and Safety of
GlaxoSmithKline Biologicals' HPV (580299) Vaccine in Healthy Female
Subjects
17 Ineligible comparator HPV-070 114700 NCT01381575 Immunogenicity and Safety Study of GlaxoSmithKline Biologicals' HPV-
16/18 L1 AS04 Vaccine When Administered According to Alternative 2-
dose Schedules in 9 - 14-Year-Old Females
18 Ineligible comparator HPV-071 115411 NCT01462357 Immunogenicity and Safety Study of GlaxoSmithKline Biologicals' HPV-
16/18 L1 AS04 Vaccine and Merck's Gardasil® Vaccine When Administered
According to Alternative 2-dose Schedules in 9-14-Year-Old Females
19 Ineligible comparator HPV-071-PRI 2011-002035- Not identified A Phase IIIb observer-blind, randomized, multicentre primary
26 immunization study to evaluate the immunogenicity and safety of GSK
Page 332 of 637

Biologicals’ HPV-16/18 L1 VLP AS04 vaccine and Merck's Quadrivalent
Human Papillomavirus (Types 6, 11, 16, 18) Recombinant Vaccine, when
administered intramuscularly according to alternative 2-dose schedules in
9-14 year old healthy females
20 Ineligible comparator V501-016 2004_083 NCT00092495 A Study to Demonstrate Immunogenicity and Tolerability of Gardasil
(V501) Quadrivalent HPV (Types 6, 11, 16, 18) L1 Virus-Like Particle (VLP)
Vaccine in Preadolescents, and To Determine End-Expiry Specifications for
the Vaccine
21 Ineligible comparator V501-024 2005_093 NCT00337428 An Open-Label, Randomized, Multicenter Study of the Safety, Tolerability,
and Immunogenicity of Gardasil (V501) Given Concomitantly With
REPEVAX™ in Healthy Adolescents 11-17 Years of Age
22 Ineligible comparator V501-025 2005_092 NCT00325130 An Open-Label, Randomized, Multicentre Study of the Safety, Tolerability,
and Immunogenicity of V501 Given Concomitantly With Menactra™ and
ADACEL™ in Healthy Adolescents 11-17 Years of Age
23 Ineligible comparator V502-001 2005_086 NCT00260039 A Randomized, International, Double-Blinded (With In-House Blinding),
GARDASIL-Controlled, Dose-Ranging Study of Octavalent Human
Papillomavirus (HPV) L1 Virus-like Particle (VLP) Vaccine Administered to
16- to 23- Year-Old Women
24 Ineligible comparator V503-001 2007_538 NCT00543543 A Randomized, International, Double-Blinded (With In-House Blinding),
Controlled With GARDASIL, Dose-Ranging, Tolerability, Immunogenicity,
and Efficacy Study of a Multivalent Human Papillomavirus (HPV) L1 Virus-
Like Particle (VLP) Vaccine Administered to 16- to 26- Year-Old Women
25 Ineligible comparator V503-001-04 PER-090-13 Not identified A RANDOMIZED, INTERNATIONAL, DOUBLE-BLINDED (WITH IN-HOUSE
BLINDING), CONTROLLED WITH GARDASIL™, DOSE-RANGING,
TOLERABILITY, IMMUNOGENICITY, AND EFFICACY STUDY OF A
MULTIVALENT HUMAN PAPILLOMAVIRUS (HPV) L1 VIRUS-LIKE PARTICLE
(VLP) VACCINE ADMINISTERED TO 16- TO 26-YEAR-OLD WOMEN
26 Ineligible comparator V503-002 Not identified NCT00943722 Immunogenicity and Safety of a 9-Valent HPV Vaccine
(immune-bridging
study)
27 Ineligible comparator V503-002 2009_611 NCT00943722 A Phase III Clinical Trial to Study the Immunogenicity, Tolerability, and
(lot consistency Manufacturing Consistency of V503 (A Multivalent Human Papillomavirus
study) [HPV] L1 Virus-Like Particle [VLP] Vaccine) in Preadolescents and
Adolescents (9 to 15 Year Olds) With a Comparison to Young Women (16
to 26 Year Olds)
28 Ineligible comparator V503-002 follow- PER-120-12 Not identified A PHASE III CLINICAL TRIAL TO STUDY THE IMMUNOGENICITY,
up TOLERABILITY, AND MANUFACTURING CONSISTENCY OF V503 (A
MULTIVALENT HUMAN PAPILLOMAVIRUS [HPV] L1 VIRUS-LIKE PRTICLE
[VLP] VACCINE) IN PREADOLESCENTS AND ADOLESCENTS (9 TO 15 YEAR
OLDS) WITH A COMPARISON TO YOUNG WOMEN (16 TO 26 YEAR OLDS)
29 Ineligible comparator V503-003 2012-002758- NCT01651949 A Phase III Clinical Trial to Study the Tolerability and Immunogenicity of
22 9vHPV (V503), a Multivalent Human Papillomavirus (HPV) L1 Virus-Like
Particle (VLP) Vaccine, in 16- to 26-Year-Old Men and 16- to 26-Year-Old
Women
30 Ineligible comparator V503-005 Not identified NCT00988884 A Phase III Open-Label Clinical Trial to Study the Immunogenicity and
Tolerability of V503 (A Multivalent Human Papillomavirus [HPV] L1 Virus-
Like Particle [VLP] Vaccine) Given Concomitantly With Menactra™ and
Adacel™ in Preadolescents and Adolescents (11 to 15 Year Olds)
31 Ineligible comparator V503-007 2010_512 NCT01073293 A Phase III Open-Label Clinical Trial to Study the Immunogenicity and
Tolerability of V503, a Multivalent Human Papillomavirus (HPV) L1 Virus-
Like Particle (VLP) Vaccine, Given Concomitantly With REPEVAX™ in
Preadolescents and Adolescents (11 to 15 Year Olds)
32 Ineligible comparator V503-009 GDS01C NCT01304498 A Randomized, Double-Blind, Phase III Study of the Immunogenicity and
Safety of a 9-Valent Human Papillomavirus L1 Virus-Like Particle Vaccine
(V503) Versus Gardasil® in 9-15-Year-Old Girls. Vesikari T, Brodszki N, van
Damme P, et al. Pediatr Infect Dis J. 2015; 34(9):992-998.
33 Ineligible comparator V503-010 2013-001314- NCT01984697 A Phase III Clinical Trial to Study the Tolerability and Immunogenicity of a
15 2-dose Regimen of V503, a Multivalent Human Papillomavirus (HPV) L1
Virus-Like Particle (VLP) Vaccine, Administered in Preadolescents and
Adolescents (9 to 14 Year Olds) With a Comparison to Young Women (16
to 26 Year Olds)
34 Ineligible comparator V503-020 GDS07C NCT02114385 A Randomized, Double-Blinded, Controlled With GARDASIL (Human
Papillomavirus Vaccine [HPV] [Types 6, 11, 16, 18] (Recombinant,
Adsorbed)), Phase 3 Clinical Trial to Study the Immunogenicity and
Tolerability of V503 (9-Valent Human Papillomavirus L1 Virus-Like Particle
[VLP] Vaccine) in 16- to 26-year-old Men
35 Ineligible comparator V503-021 Not identified NCT02653118 A Registry-Based Extension of Protocol V503-001 in Countries With
Centralized Cervical Cancer Screening Infrastructures to Evaluate the Long-
Term Effectiveness, Immunogenicity, and Safety of Multivalent Human
Page 333 of 637

Papillomavirus (HPV) L1 Virus- Like Particle (VLP) Vaccine as Administered
to 16- to 26- Year- Old Women
36 Ineligible comparator V504-001 2007_566 NCT00551187 A Randomized, Double-Blinded, Tolerability and Immunogenicity Study of
a Multivalent Human Papillomavirus (HPV) L1 Virus-Like Particle (VLP)
Vaccine Administered Concomitantly With GARDASIL to 16- to 26- Year-
Old Women
37 Ineligible comparator Not identified 2016-001963- NCT02199691 Immunogenicity and Safety of an Investigational Quadrivalent
35 Meningococcal Conjugate Vaccine in Healthy Adolescents
38 Ineligible comparator Not identified Not identified NCT02993757 Immunogenicity and Safety of a Tetravalent Dengue Vaccine Administered
Concomitantly or Sequentially With Gardasil® in Healthy Subjects Aged 9
to 13 Years in Malaysia
39 Ineligible comparator Not identified Not identified NCT02979535 Immunogenicity and Safety of a Tetravalent Dengue Vaccine Administered
Concomitantly or Sequentially With Cervarix® in Healthy Female Subjects
Aged 9 to 14 Years in Mexico
40 Ineligible comparator Not identified Merck IISP NCT00925288 Acceptability and Feasibility of a Modified HPV Vaccine Schedule in Brothel
35706 Based Female Sex Workers in Peru
IRB00001625
41 Ineligible comparator Not identified MERCK 08 NCT02968420 Long Term Immune Memory Responses to Human Papillomavirus (HPV)
Vaccination Following 2 Verses 3 Doses of Quadrivalent HPV Vaccine
(Merck08)
42 Ineligible comparator Not identified 32090 NCT00572832 Randomized Trial of Alternative Quadrivalent Human Papilloma Virus
(HPV) Vaccination Schedules in a University Setting
43 Ineligible participants HPV-005 580299/005 NCT00693966 A Phase II Double-Blind, Randomized, Dose-Comparison Study to Evaluate
the Safety and Immunogenicity of MEDI-517, a Virus-Like Particle Vaccine
Against Human Papillomavirus Types 16 and 18, in Healthy Adult Female
Volunteers
44 Ineligible participants HPV-020 107863 NCT00586339 Evaluation of the Safety and Immunogenicity of GlaxoSmithKline
Biologicals' HPV Vaccine 580299 (Cervarix TM) in Adult Human
Immunodeficiency Virus (HIV) Infected Female Subjects
45 Ineligible participants V501-011 2007_576 NCT00517309 Immunogenicity and Safety of Quadrivalent HPV L1 Virus-Like Particle
(VLP) Vaccine in 16- to 23-Year-Old Women When Administered Alone or
Concomitantly With Hepatitis B Vaccine (Recombinant)--the F.U.T.U.R.E.
Study (Females United to Unilaterally Reduce Endo/Ectocervical Disease
46 Ineligible participants V501-012 2004_080 NCT00092482 Immunogenicity and Safety of Gardasil (V501) Quadrivalent HPV (Types 6,
11, 16, 18) L1 Virus-Like Particle (VLP) Vaccine in Consistency Lots for 16-
to 23-Year-Old Women With and Additional Immunogenicity Bridge to the
Monovalent HPV 16 Vaccine Pilot Manufacturing Lot Study-The
F.U.T.U.R.E. Study (Females United to Unilaterally Reduce
Endo/Ectocervical Disease) [participants were included in V501-013]
47 Ineligible participants Not identified QHPV-RTC NCT01928225 A Randomized, Placebo-controlled Trial of Pre-treatment HPV Vaccination
on Outcomes to LEEP Treatment of Cervical High Grade Squamous
Intraepithelial Lesions in HIV-infected Women.
48 Not comparative HPV-023 114379 NCT01418937 Safety Study of GSK Biologicals' Human Papillomavirus Vaccine (GSK-
follow-up 580299) in Healthy Female Control Subjects From the GSK HPV-023 Study
49 Not comparative HPV-025 111375 NCT00877877 Follow-up Study to Evaluate the Long-term Immunogenicity and Safety of
a HPV Vaccine (GSK 580299) in Healthy Female Subjects
50 Not comparative HPV-052 112024 NCT00937950 Gynaecological follow-up of a subset of 580299/008 study subjects
580299) in Healthy Female Subjects.
52 Not comparative HPV-057 111955 NCT00799825 Safety Study of GSK Biologicals' Human Papillomavirus Vaccine
(GSK580299) in Female American and Canadian Subjects Who Had
Received Control Vaccine in Study 580299/008
53 Not comparative HPV-062 113617 NCT01190176 Gynaecological Follow-up of a Subset of HPV-015 Study Subjects
580299) in Healthy Female Control Subjects From the Primary
NCT00294047 Study
580299) in Healthy Female Control Subjects From the Primary
NCT00294047 Study
56 Not comparative V501-030-1 Not identified NCT01427777 The Thirty-six-Month Immunogenicity Evaluation of Quadrivalent HPV
follow up (Types 6, 11, 16, 18) L1 Virus-Like Particle (VLP) Vaccine in Chinese Female
Subjects Aged 9 to 45 Years and Male Subjects Aged 9 to 15 Years
57 Not completed/ongoing as 311-HPV-1001 Not identified NCT03085381 A Randomized, Double-Blind and Placebo-Controlled Phase I Study to
of 1 July 2017 Evaluate the Safety and Primary Immunogenicity of the Quadrivalent
Human Papillomavirus (Types 6, 11, 16, 18) Recombinant Vaccine
(Hansenula Polymorpha) in Chinese Female Subjects Aged 9-45 Years
Page 334 of 637

58 Not completed/ongoing as 311-HPV-1002 Not identified NCT02740790 Immunogenicity and Safety of Recombinant Human Papillomavirus
of 1 July 2017 Bivalent(Type 16 and 18) Vaccine (Yeast) in Healthy Females
59 Not completed/ongoing as 311-HPV-1003 Not identified NCT02733068 A Phase III Double Blinded, Randomized Controlled Study to Evaluate
of 1 July 2017 Efficacy of Protection Against HPV-16 and 18 Related Diseases,
Immunogenicity and Safety of HPV-16/18 Vaccine in Healthy Females Aged
18-30 Years
60 Not completed/ongoing as 311-HPV-1004 Not identified NCT02740777 Immunogenicity Study of a 2-dose Immunization Schedule of Recombinant
of 1 July 2017 Human Papillomavirus Virus-like Particle Vaccine (Type 16 and 18 L1
Proteins, Yeast) in Adolescent Females Aged 9 to 14 Years
61 Not completed/ongoing as HPV-023-EPI 109624 NCT00518336 Brazilian cohort of women vaccinated in the phase IIb, blinded, primary
of 1 July 2017 study 580299/001
62 Not completed/ongoing as HPV-027 115006 NCT01393470 A Long-term Follow-up Registry-based Cohort Study of HPV Vaccine
of 1 July 2017 Efficacy Against Cervical Pre-cancerous Lesions and Cervical Cancers in a
Cohort of Females Previously Enrolled From Finland in Study HPV-008, as
Compared to a Non-intervention Population -Based Reference Cohort of
Females From Finland
63 Not completed/ongoing as HPV-111103-EPI 111103 Not identified A phase IV, randomized, open-label, controlled, post-licensure study to
of 1 July 2017 evaluate the safety of GlaxoSmithKline Biologicals’ HPV-16/18 L1 VLP AS04
vaccine (Cervarix®) when administered intramuscularly according to a 0, 1,
6-month schedule in females aged 18-25 years. - EPI-HPV-111103
64 Not completed/ongoing as HPV-PRO-003 Not identified NCT01735006 A Phase III Multicentre, Randomized, Double-Blind, Placebo(Hepatitis E
of 1 July 2017 Vaccine) Controlled Study to Evaluate the Efficacy, Immunogenicity and
Safety of a Recombinant (E.coli) Human Papillomavirus Bivalent Vaccine in
Healthy Women
65 Not completed/ongoing as HPV-PRO-005 Not identified NCT02710851 A Phase Ⅱ Randomized, Double-Blinded, Placebo Controlled Study to
of 1 July 2017 Evaluate the Immunogenicity of the Recombinant (E.coli) Human
Papillomavirus Type 6/11 Bivalent Vaccine in Healthy Volunteers Aged 18-
55 Years
66 Not completed/ongoing as V501-122 132237 NCT01862874 A Phase III Placebo-controlled Clinical Trial to Study the Tolerability,
of 1 July 2017 Immunogenicity and Efficacy of V501 in 16- to 26-year-old Japanese Men
67 Not randomised HPV-014 103514 NCT00196937 Phase 3, Open, Age-stratified Study to Assess Immunogenicity and Safety
of GSK Biologicals' HPV-16/18 Vaccine Administered Intramuscularly
According to 3-dose Schedule (0,1,6 Months) in Healthy Female Subjects
Aged 15 - 55 Years and Long-Term Follow-up
68 Not randomised HPV-018-EPI 114101 NCT01905462 Post-marketing Safety Study to Assess the Risk of Spontaneous Abortions
in Women Exposed to Cervarix in the United Kingdom
69 Not randomised HPV-040-EPI 116239 NCT01953821 An Observational Cohort Study to Assess the Risk of Autoimmune Diseases
in Adolescent and Young Adult Women Aged 9 to 25 Years Exposed to
Cervarix® in the United Kingdom
70 Not randomised HPV-PRO-006 Not identified NCT02562508 Immunogenicity and Safety Study of a Bivalent Human Papillomavirus
(Type 16, 18) Recombinant Vaccine (E.coli) in Healthy Female Subjects
Aged 9 to 17 Years
71 Not randomised HPV-PRO-006-2 Not identified NCT03206255 Immuno-persistence Study of a Bivalent Human Papillomavirus (Type 16,
18) Recombinant Vaccine (E.coli) in Healthy Female Subjects Aged 9 to 17
Years
72 Not randomised V501-031 2010_019 NCT01078220 A Post-licensure Surveillance Program for the Safety of GARDASIL™ in a
Managed Care Organization Setting
73 Not randomised V501-033 2010_018 NCT01077856 GARDASIL™ Vaccine Impact in Population Study
74 Not randomised V501-070 EP08014.070 NCT01567813 Post-Licensure Observational Study of the Safety of GARDASIL™ in Males
75 Not randomised V503-004 Not identified NCT03158220 An Open-Label Phase III Clinical Study to Study the Immunogenicity and
Tolerability of GARDASIL®9 (A Multivalent Human Papillomavirus [HPV] L1
Virus-Like Particle [VLP] Vaccine) in Adult Women (27- to 45-Year-Olds)
Compared to Young Adult Women (16 to 26 Year Olds)
76 Not randomised V503-008 Not identified NCT01254643 A Phase III Open-label, Safety, Tolerability and Immunogenicity Study of a
9-Valent Human Papillomavirus (HPV) L1 Virus-Like Particle (VLP) Vaccine
Administered to 9- to 15-Year-Old Japanese Preadolescent and Adolescent
Girls
77 Not randomised Not identified 112677 NCT01498627 Cervarix Long-term Safety Surveillance Using the PGRx Information System
(PGRx Study)
78 Not randomised Not identified 113522 NCT01153906 Post-marketing Safety Study of Autoimmune Diseases Following Cervarix®
Vaccination in Females Aged 9-25 Years in the US
79 Not randomised Not identified 07-09-0344 NCT00727636 Pilot Study of Immunogenicity and Tolerability to the Quadrivalent Human
Papillomavirus Virus-like Particle (VLP) Vaccine (Gardasil) Among IBD
Patients on Immunosuppressive Therapy Compared to Healthy Children
and Youth Adult Females
Page 335 of 637

80 Not randomised Not identified 35384 NCT00944879 Preparing for Adolescent HIV Vaccine Trials in South Africa: a Multi-centre
CT.2006.33111. Study to Evaluate Acceptability of the HPV Vaccine in Adolescents
004
81 Not randomised Not identified 0807-02 NCT00767897 The Impact of the Human Papilloma Virus in Paediatric Chronic Kidney
Disease, Dialysis, and Transplant Patients
82 Not randomised Not identified MERCK_33610 NCT00806676 Antibody Response to Human Papillomavirus Recombinant Vaccine
(Gardasil®) in Girls and Young Women with Chronic Kidney Disease
83 Not randomised Not identified 1000037024 NCT02624349 Immunogenicity and Safety of Human Papilloma Virus Vaccine in Solid
Organ Transplant Recipients
84 Not randomised Not identified 091-2014 NCT02382900 Evaluation of a Two-dose Schedule of Quadrivalent Human Papilloma Virus
(Types 6, 11, 16, 18) Recombinant Vaccine in 11-year-old Boys in Mexico
City (Gardasil® Merck and Co.).
85 Not randomised Not identified Gardasil in JIA NCT00573651 Pilot Study of the Safety and Efficacy of Quadrivalent Human
Papillomavirus Vaccine (Gardasil®) in Female Subjects with Juvenile
Idiopathic Arthritis (JIA)/ Seronegative Arthritis
86 Not randomised Not identified GINI NCT01924754 Gardasil Immunogenicity with Needle-Free Injection
87 Phase 1 study HPV-048 110659 NCT00541970 Evaluation of the Safety and Immunogenicity of GSK Biologicals' HPV
Vaccine 580299 When Administered in Healthy Females Aged 9 - 25 Years
Using an Alternative Schedule and an Alternative Dosing as Compared to
the Standard Schedule and Dosing
88 Phase 1 study HPV-PRO-004 Not identified NCT02405520 A Phase I Randomized, Double-Blinded, Placebo Controlled Study to
Evaluate the Safety and Immunogenicity of the Recombinant (E.coli)
Human Papillomavirus Type 6/11 Bivalent Vaccine in Healthy Volunteers
Aged 18-55 Years
89 Phase 1 study V501-001 Not identified Not identified No title.
90 Phase 1 study V502-002 2009_552 NCT00851643 A Randomized, Double-Blind, Multicentre, Biphasic, Controlled With
GARDASIL™ Dose-Escalation Study of Octavalent Human Papillomavirus
(HPV) L1 Virus-Like Particle (VLP) Vaccine Adjuvanted With Amorphous
Aluminium Hydroxyphosphate Sulphate (AAHS) and ISCOMATRIX™ (IMX)
91 Phase 1 study Not identified 109836 NCT00478621 A Multicentre Study to Evaluate the Safety and Immunogenicity of GSK
Biologicals' HPV Vaccine (GSK1674330A) in Healthy Female Subjects Aged
18-25 Years.
92 Phase 1 study Not identified KCT0000604 Not identified Assessment of the safety, tolerance, and immunogenicity of EG-HPV
(human papillomavirus vaccine) in healthy male adult volunteers: A
double-blinded, randomized, adjuvant vehicle-controlled trial
93 Prematurely terminated HPV-020-EPI 114176 NCT01290393 Post-marketing Safety Study to Assess the Risk of Spontaneous Abortion
Following Administration of CERVARIX in the United States and Canada
94 Prematurely terminated HPV-078 117099 NCT02082639 Immunogenicity and Safety Study of GSK Biologicals' Human
Papillomavirus (HPV) Vaccine (Cervarix™) (GSK-580299) When Co-
administered With GSK Biologicals' Hepatitis A Vaccine (Havrix®) (GSK-
208109) in Healthy Female Adolescents Aged 9-14 Years
95 Prematurely terminated HPV-081 (HPV- 200255 Not identified A phase IIIb, open-label, non-randomised, multicentre study to assess the
048 follow-up) immunogenicity and safety of GSK Biologicals’ HPV-16/18 L1 VLP AS04
vaccine when administered intramuscularly according to a 2-dose
schedule in healthy female adolescents or intramuscularly according to a
3-dose schedule in healthy female adults, 6.5 years after first vaccine dose
in study HPV-048 PRI (110659). - HPV-081 EXT:048 Y6.5
96 Prematurely terminated V502-003 2006_503 NCT00365443 No title.
97 Prematurely terminated Not identified 113763 Not identified Post-marketing surveillance (PMS) of GlaxoSmithKline (GSK) Biologicals’
human papillomavirus (HPV) -16/18 vaccine, Cervarix™ when administered
to healthy females according to the Prescribing Information in Sri Lanka
98 No clinical study report V501-002 Not identified Not identified MRL Clinical Study Report (Synopsis): Safety tolerability and
obtained Immunogenicity of a Research Lot of HPV 16 Virus-Like Particle (VLP)
Vaccine in College-Age Women (Protocol 002).
99 No clinical study report V501-004 Not identified Not identified No title.
obtained
100 No clinical study report V501-028 2006_052 NCT00411749 V501 Phase II Immunogenicity Study in Females Aged 9 to 17 Years
obtained
obtained
obtained
103 No clinical study report V505-001 2007_567 NCT00520598 A Phase IIa Randomized, Double-Blind Controlled with Gardasil, Clinical
obtained Trial to Study the tolerability and Immunogenicity of V505 (a Multivalent
Human Papilloma Virus [HPV] L1 Virus Like Particle [VLP] Vaccine) in
Page 336 of 637

Healthy 16 to 26-Year-Old Women
104 No clinical study report MENACWY-TT- 113823 NCT01755689 Immunogenicity and Safety Study of GSK Biologicals' Meningococcal
obtained 054 Vaccine 134612 With or Without Co-administration of Cervarix and
Boostrix in Female Adolescents and Young Adults at 9-25 Years of Age
105 No clinical study report HPV-073 115887 NCT01627561 Safety and Immunogenicity of GSK Biologicals' HPV-16/18 L1 VLP AS04
obtained Vaccine (GSK-580299) in Healthy Female Children 4-6 Years Old
106 No clinical study report HPV-007 580299/007 NCT00120848 Study of the Efficacy of Candidate HPV 16/18 VLP Vaccine in the
obtained Prevention of HPV-16 and/or HPV-18 Cervical Infection in Adolescent &
Young Adult Women in North America and Brazil Vaccinated in Primary
Study 580299/001
107 No clinical study report HPV-009 580299/009 NCT00128661 Efficacy of the HPV-16/18 Vaccine: Final according to protocol results from
obtained the blinded phase of the randomized Costa Rica HPV-16/18 Vaccine Trial
108 No clinical study report HPV-011 580299/011 NCT00309166 An Observer-blind, Randomized, Controlled Study to Assess the
obtained Immunogenicity and Safety of GlaxoSmithKline Biologicals' HPV Vaccine
Administered Intramuscularly According to a 0, 1, 6 Month Schedule in
Healthy Male Subjects Aged 10-18 Years
109 No clinical study report HPV-013 104896 NCT00316706 A Long-term, Open Follow-up of the Immunogenicity and Safety of GSK
obtained follow-up Biologicals' HPV Vaccine (580299) in Healthy Female Subjects Vaccinated
in Study HPV-013
110 No clinical study report HPV-021 106069 NCT00481767 Study to Assess the Immunogenicity and Safety of GlaxoSmithKline
obtained Biologicals' HPV Vaccine GSK580299 in Healthy Female Subjects Aged 10-
25 Years
111 No clinical study report HPV-036 105926 NCT00345878 Phase IIIb, Double-blind, Randomized, Controlled Study to Evaluate the
obtained Immunogenicity & Safety of GSK Biologicals' HPV-16/18 L1 VLP AS04
Vaccine Administered Intramuscularly (0, 1, 6 Month Schedule) in Healthy
Women from Malaysia.
112 No clinical study report HPV-039 107638 NCT00779766 Efficacy, Immunogenicity and Safety of GlaxoSmithKline Biologicals' HPV
obtained GSK 580299 Vaccine in Healthy Chinese Female Subjects
113 No clinical study report HPV-PRO-002 Not identified NCT01356823 A Randomized, Double-Blinded, Placebo-Controlled, Dose-Ranging Study
obtained of Recombinant Human Papillomavirus Virus 16/18 Bivalent Vaccine
(E.coli) in Healthy Female Subjects Aged 18 to 25 Years
114 No clinical study report V501-005 PMC2749988 Not identified Longer-term efficacy of a prophylactic monovalent human papillomavirus
obtained follow-up type 16 vaccine
115 No clinical study report V501-007 2006_516 NCT00365716 A Placebo-Controlled, Dose-Ranging Study of Quadrivalent HPV Virus-Like
obtained Particle (VLP) Vaccine in 16- to 23-Year-Old Women
116 No clinical study report V501-023 2005_066 NCT00157950 An Immunogenicity and Safety Study of Gardasil (V501 (Human
obtained Papillomavirus [Types 6, 11, 16, 18] Recombinant Vaccine)) in Females 9 to
23 Years of Age in Korea
117 No clinical study report V501-027 2006_032 NCT00378560 V501 Phase II Efficacy Study in Women Aged 18 to 26
obtained
118 No clinical study report V501-030 2007_021 NCT00496626 An Immunogenicity and Safety Study of Quadrivalent HPV (Types 6, 11, 16,
obtained 18) Virus-Like Particle (VLP) Vaccine in Chinese Female Subjects Aged 9 to
45 Years and Male Subjects Aged 9 to 15 Years
119 No clinical study report V501-041 2009_532 NCT00834106 A Randomized, Placebo-Controlled, Double-Blind Study of Quadrivalent
obtained HPV (Types 6, 11, 16, 18) L1 Virus-Like Particle (VLP) Vaccine to Investigate
the Safety, and Efficacy in Chinese 20 - to 45-Years-Old Women
120 No clinical study report V501-046 Not identified NCT01245764 Evaluation of Safety and Immunogenicity of GARDASIL™ in Healthy
obtained Females Between 9 and 26 Years of Age in Sub Saharan Africa
121 No clinical study report Not identified PMC4378717 NCT01489527 High HIV, HPV, and STI prevalence among young Western Cape, South
obtained African women: EVRI HIV prevention preparedness trial
Page 337 of 637

List of excluded non-industry funded studies

No. Reason for exclusion Study ID NCT ID Title
1 Ineligible comparator 122.05.01 9427-L1802/1-21C NCT01456715 Immunogenicity and Safety of Gardasil and Twinrix Vaccines Co-
administered or Administered a Month Apart, according to the 0, 6
Months Schedule and the Effect of a Third Dose of Gardasil or Cervarix
Administered 42 Months Later.
2 Ineligible comparator EUCTR2011-001871-37-DK Not identified Comparison of vaccine effects of two different vaccines against Human
Papillomavirus in HIV infected people
3 Ineligible comparator 2013/422 2013-002340-90 NCT01914367 Study of the Molecular Mechanisms Underlying the Cross-neutralizing
Capacity of AS04-adjuvanted HPV Vaccine (Cervarix®) in Comparison With
the Aluminium hydroxyphosphate Sulphate Adjuvanted HPV Vaccine
(Gardasil®)
4 Ineligible comparator ACTRN12608000339358 Not identified A pilot non-inferiority immunogenicity single blind randomised study of
two human papillomavirus vaccines administered intradermally to females
aged 18 to 26 years to bridge previous efficacy findings of intramuscularly
administered vaccine
5 Ineligible comparator CTRI/2013/11/004140 Not identified Comparison of Post-licensure safety surveillance of Bivalent and
Quadrivalent Human Papillomavirus vaccines in Healthy Women.
6 Ineligible comparator ESCUDDO NCT03180034 A Scientific Evaluation of One or Two Doses of Vaccine Against Human
Papillomavirus: the ESCUDDO Study
7 Ineligible comparator FASTER-Tialpan Study NCT03105856 HPV Vaccination Impact on Cervical Cancer Screening Program: FASTER-
Tialpan Study in Mexico
8 Ineligible comparator H07-00928 NCT00501137 A Controlled Trial to Assess the Immunogenicity of a Proposed Paediatric
Dosing Schedule of Human Papillomavirus Vaccine
9 Ineligible comparator HLS04/2011 NCT01512784 Long Term Immunogenicity of Quadrivalent Human Papillomavirus Vaccine
(Gardasil®)in HIV-infected Adolescents and Young Adults vs. Healthy
Adolescents and Young Adults: Non-randomized Controlled Clinical Trial
10 Ineligible comparator HPV 2355 NCT02567955 Immunogenicity and Safety of Gardasil-9 and Cervarix When Administered
to 9-10-year-old Subjects According to 0-6 Month Schedule
11 Ineligible comparator HPV CSP01 NCT00956553 A Phase IV, Randomised Study to Evaluate the Immune Responses of UK
Adolescent Girls Receiving CervarixTM or GardasilTM Human Papillomavirus
Vaccines
12 Ineligible comparator HPV01 NCT00524745 Comparison of the Immunogenicity and Reactogenicity of Alternative
Schedules of Gardasil Vaccine to Prevent HPV Infection
13 Ineligible comparator ICI-VPH-1 NCT02009800 ICI-VPH: Impact Des Calendriers d'Immunisation Contre Les HPV
14 Ineligible comparator IRB00046117 NCT01505049 Immune Memory After Papillomavirus Vaccination (IMAP-I) Study. DMID
10-0014
15 Ineligible comparator ISRCTN98283094 NCT00923702 Randomised Trial of Two Versus Three Doses of Human Papillomavirus
(HPV) Vaccine in India
16 Ineligible comparator LTN0001 NCT01386164 Immune Response to Bivalent and Tetravalent Human Papillomavirus
Vaccine in HIV Infected Adults
17 Ineligible comparator MITU-001 NCT01173900 Delivery, Uptake and Acceptability of HPV Vaccination in Tanzanian Girls
18 Ineligible comparator MITU-002 NCT02834637 A Dose Reduction Immuno-bridging and Safety Study of Two HPV Vaccines
in Tanzanian Girls
19 Ineligible comparator https://doi.org/10.1089/jwh.2009.1 Not identified Randomized trial of an alternate human papillomavirus vaccine
753 administration schedule in college-aged women.
20 Ineligible comparator http://dx.doi.org/10.1080/2164551 Not identified Reactogenicity of Cervarix and Gardasil human papillomavirus (HPV)
5.2016.1277846 vaccines in a randomized single blind trial in healthy UK adolescent
females
21 Ineligible comparator PMID: 23770335 DOI: Not identified A pilot randomized study to assess immunogenicity, reactogenicity, safety
10.1016/j.vaccine.2013.06.034 and tolerability of two human papillomavirus vaccines administered
intramuscularly and intradermally to females aged 18-26 years.
22 Ineligible participants A5298 11798 NCT01461096 A Randomized, Double-Blinded, Placebo-Controlled, Phase III Trial of the
Quadrivalent HPV Vaccine to Prevent Anal Human Papillomavirus Infection
in HIV-Infected Men and Women
23 Ineligible participants HRPO-07-0648 NCT00941889 The Effect of Human Papillomavirus Vaccination on Recurrence Rates in
HIV Positive Patients Treated for Anal Condylomata
24 Ineligible participants ISRCTN14732216 Not identified Effectiveness of the quadrivalent human papillomavirus (qHPV) vaccine in
HIV-positive Spanish men having sex with men (MSM): Double-bind
randomised Clinical Trial
25 Ineligible participants ISRCTN14732216 follow-up Not identified No title.
Page 338 of 637

26 Ineligible participants P1047 NCT00339040 Phase II Safety and Immunogenicity Study of Quadrivalent Human
Papillomavirus (Types 6, 11, 16, 18) L1 Virus-Like Particle (VLP) Vaccine in
HIV Infected Children 7 to 12 Years of Age
27 Not comparative 342/2009 NCT02296255 Effective Surveillance and Impact of HPV Vaccination on Screening for
Cervical Cancer in Tuscany
28 Not completed/ongoing as 1603017415 NCT02864147 Treatment of High-Grade Pre-Neoplastic Cervical Lesions (CIN 2/3) Using a
of 1 July 2017 Novel "Prime and Pull" Strategy
29 Not completed/ongoing as EUCTR2012-004007-13-DE Not identified A randomized, placebo-controlled, phase IIIb HPV vaccination trial with
of 1 July 2017 Gardasil® in patients with recurrent condylomata acuminata
30 Not completed/ongoing as EUCTR2013-002009-70-NL Not identified Quadrivalent HPV vaccination after effective treatment of Anal
of 1 July 2017 Intraepithelial Neoplasia in HIV+ men - VACCAIN-P
31 Not completed/ongoing as 2016-002083-13 Not identified An observational follow up study of a randomised parallel group phase IV
of 1 July 2017 study to evaluate the duration of the immune response to vaccine and
non-vaccine HPV types in UK adolescent females who received either
Cervarix or Gardasil Human Papillomavirus (HPV) vaccines.
32 Not completed/ongoing as 2016-002455-20 Not identified Efficacy study of the quadrivalent Human Papilloma Virus (HPV) vaccine to
of 1 July 2017 prevent recurrence of External Genital Warts (EGW) in patients who were
cured in the first place
33 Not completed/ongoing as 999909106 09-C-N106 NCT00867464 Extended Follow-up of Young Women in Costa Rica Who Received
of 1 July 2017 Vaccination Against Human Papillomavirus Types 16 and 18 and
Unvaccinated Controls
34 Not completed/ongoing as ACTRN12613001207707 Not identified Pilot Study for a trial in Human Papillomavirus (HPV) vaccinated and
of 1 July 2017 unvaccinated women presenting for cervical screening of 6 yearly HPV
screening versus 3-yearly cytology screening to assess the feasibility via
recruitment rates, cross sectional positivity rate and laboratory
implementation.
35 Not completed/ongoing as CIHR-MOP-125949 NCT01824537 Transmission Reduction and Prevention With HPV Vaccination (TRAP-HPV)
of 1 July 2017 Study: A Randomized Controlled Trial of the Efficacy of HPV Vaccination in
Preventing Transmission of HPV Infection in Heterosexual Couples
36 Not completed/ongoing as GINI Study NCT02363660 A Randomized, Parallel Design Study to Compare the Safety and
of 1 July 2017 Effectiveness of Gardasil When Delivered Per Standard Practice (Full Dose,
Intramuscular (IM) Delivery Using a Needle and Syringe) vs. Full Dose, IM
Delivery Via Needle-free Jet Injection or Reduced Dose, Intradermal
Delivery Via Needle-free Jet Injection
37 Not completed/ongoing as Merck-MISP-53183 NCT02750202 Prophylactic Vaccines as Therapy: Prevention of Recurrence of Extensive
of 1 July 2017 Genital Warts
38 Not completed/ongoing as NL45200.018.13 NCT02087384 Quadrivalent HPV Vaccination After Effective Treatment of Anal
of 1 July 2017 Intraepithelial Neoplasia in HIV+ Men
39 Not randomised 883 NCT01717118 Evaluation of Immunogenicity Levels in Women With HPV Vaccine in
Mexico
40 Not randomised N01AI80006C NCT01030562 Immunogenicity of the HPV-6, 11, 16, 18 Vaccine Among Adolescent Girls
Who Receive Vaccine Doses at Non-recommended Intervals and Factors
Related to Non-adherence
41 Not randomised 09-C-0024 NCT00798265 A Phase I Study of Quadrivalent Human Papilloma Virus (HPV) (Types 6, 11,
16, 18) Recombinant Vaccine in HIV-Infected and HIV-Negative Pre-
Adolescents, Adolescents and Young Adults
42 Not randomised 2014.5.FNRERC.5.SU NCT02276521 Evaluation of Long-term Immunological Responses Following Reduced
Dose Quadrivalent Human Papillomavirus (HPV) Vaccine Schedules: A
Phase II/III Clinical Trial
43 Not randomised A5240 NCT00604175 A Phase II Study to Evaluate the Immunogenicity and Safety of a
Quadrivalent Human Papillomavirus Vaccine in HIV-1-Infected Females
44 Not randomised ATN 064 NCT00710593 Immunogenicity, Safety, Tolerability, and Behavioural Consequences of an
HPV-6, -11, -16, -18 Vaccine in HIV-Infected Young Women
45 Not randomised DDEAMC NCT00501189 Gardasil Vaccination as Therapy in Low Grade Cervical Abnormalities
46 Not randomised DRKS00005278 Not identified Immunity against Human Papillomavirus (HPV) in vaccinated and non-
vaccinated probands
47 Not randomised HP-00041372 NCT00257738 A Phase 1 Open Label, Dose Escalation Study to Evaluate the Effect of Four
Doses of MAGE-A3/HPV 16 Trojan Peptides 0001 and 0002 Administered
Subcutaneously in Combination With Montanide and GM-CSF on
Immunological Response, Safety, Tolerability, and Preliminary Efficacy in
Patients With Squamous Cell Carcinoma of the Head and Neck
48 Not randomised HP-41372 NCT00704041 A Phase 1 Open Label, Dose Escalation Study to Evaluate the Effect of Four
Doses of MAGE-A3/HPV 16 Trojan Peptides 0001 and 0002 Administered
Subcutaneously
49 Not randomised Not identified NCT01896986 HPV Immunisation Protecting Special Risk Group Patients From Cervical
Cancer: 5 Year Follow-up Post-vaccination
Page 339 of 637

50 Not randomised NL26.113.000.08 NCT00815282 Immune Response After Human Papillomavirus Vaccination in Patients
With Autoimmune Disease
51 Not randomised NTWC/CREC/15035 NCT02477254 Long-term Immunogenicity of a Quadrivalent Human Papillomavirus (HPV)
Vaccine in Patients With Systemic Lupus Erythematosus: a Case-control
Study
52 Not randomised Pro00014388 NCT00862810 Alternate Dosing Schedules Study for HPV Vaccine
53 Not randomised Pro00014388_1 NCT02280642 Alternate Dosing Schedules Study for HPV Vaccine
54 Not randomised SG09-EN01 NCT00949572 Characterisation of Human Disseminated Cellular and Humoral Immune
Responses Following Sublingual or Intramuscular Deposition of Antigens
55 Phase 1 study cycdc2016-4 NCT02888418 Random, Double Blind, Placebo Controlled Phase I Clinical Trials to
Estimate the Safety and Preliminary Immunogenicity of Tetravalent
Recombinant Human Papilloma Virus Vaccine (6,11,16,18 Type)
(Hansenula Polymorpha) in Women of 9 to 30 Years Old and Men of 9to 17
Years Old
56 Phase 1 study https://doi.org/10.1093/jnci/93.4.2 Not identified Safety and immunogenicity trial in adult volunteers of a human
84 papillomavirus 16 L1 virus-like particle vaccine.
57 Phase 1 study PE1201 NCT02564237 A Phase I, Randomized, Observer-blind, Comparator-controlled, Study of
the Safety and Immunogenicity of StreptAnova™ Vaccine in Healthy Adults
Age ≥ 18 - 50 Years
58 Prematurely terminated EUCTR2012-000445-12-DK Not identified Immune Response to Bivalent or Quadrivalent Human Papilloma Virus
vaccination in Patients with Chronic Renal Failure - IPAR
59 Prematurely terminated LIS144 NCT01082861 A Phase 4, Randomized, Open Label Clinical Trial to Determine Efficacy and
Immunomodulation of Simultaneous HPV/HBV Vaccination Tango-trial)
60 No report obtained ISRCTN32729817 Not identified Human papillomavirus infection: a randomised controlled trial of
Imiquimod cream (5%) versus Podophyllotoxin cream (0.15%), in
combination with quadrivalent human papillomavirus or control
vaccination in the treatment and prevention of recurrence of anogenital
warts (HIPvac Trial)
61 No report obtained NCI-2012-02623 NCT00091130 An Exploratory Study to Evaluate the Effect of HPV 16 Vaccine on the
Reduction of Viral Load in HPV 16 Positive Women With Persistent Viral
Infection, But Low Grade Disease (ASCUS/LSIL)

Page 340 of 637

Additional file 4
Benefits and harms of the HPV vaccines: meta-analyses
Table of contents
1. Combined results of HPV-related outcomes 2
2. Anal outcomes 10
3. Cervical outcomes 11
4. Oropharyngeal outcomes 18
5. Penile outcomes 21
6. Vaginal outcomes 23
7. Vulvar outcomes 27
8. Referral procedures 31
9. Fatal harms 35
10. Serious harms 37
11. New onset diseases 82
12. General harms 126
13. Harms of special interest 167
14. Post hoc exploratory harm analyses 172
15. Definitions of harm categories 202
16. References 204
1
Page 341 of 637
1. Combined results of HPV-related outcomes
1.1. All-cause mortality/deaths*: intention to treat analysis
*1.1. Risk ratio for GlaxoSmithKline studies (i.e., HPV-0xx): 1.43 [0.65, 3.15]; risk ratio for Merck Sharp & Dohme studies (i.e., V50x-xxx): 1.08 [0.40, 2.96].
2
Page 342 of 637
1.2. Mortality/deaths from HPV-related cancers (anal, cervical, oropharyngeal, penile, vaginal and vulvar)
irrespective of HPV type*: intention to treat analysis
3
Page 343 of 637
1.3. Incidence of HPV-related cancers (anal, cervical, oropharyngeal, penile, vaginal and vulvar) irrespective of
HPV type*: intention to treat analysis
There were no reported cases of anal or penile cancer.
4
Page 344 of 637
1.4. Incidence of HPV-related carcinoma in situ (anal intraepithelial neoplasia grade 3 [AIN3], cervical
adenocarcinoma in situ [AIS], cervical intraepithelial neoplasia grade 3 [CIN3], penile intraepithelial
neoplasia grade 3 [PIN3], vaginal intraepithelial neoplasia grade 3 [VIN3] and vulvar intraepithelial
neoplasia grade 3 [VaIN3]) irrespective of HPV type*: intention to treat analysis
There were no reports of AIN3, PIN3, VIN3 or VaIN3 irrespective of HPV type.
5
Page 345 of 637
1.5. Incidence of HPV-related moderate intraepithelial neoplasia (anal intraepithelial neoplasia grade 2 [AIN2],
cervical intraepithelial neoplasia grade 2 [CIN2], penile intraepithelial neoplasia grade 2 [PIN2], vaginal
intraepithelial neoplasia grade 2 [VIN2] and vulvar intraepithelial neoplasia grade 2 [VaIN2]) irrespective
of HPV type*: intention to treat analysis
There were no reports of AIN2, PIN2, VIN2 or VaIN2 irrespective of HPV type.
6
Page 346 of 637
1.6. Incidence of HPV-related carcinoma in situ or worse (AIN3+, CIN3+, PIN3+, VIN3+, VaIN3+) irrespective of
There were no reports of AIN3+, PIN3+, VIN3+ or VaIN3+ irrespective of HPV type.
7
Page 347 of 637
1.7. Incidence of HPV-related moderate intraepithelial neoplasia or worse (AIN2+, CIN2+, PIN2+, VIN2+, VaIN2+)
There were no reports of AIN2+ irrespective of HPV type.
8
Page 348 of 637
1.8. Incidence of HPV-related external genital lesions (EGL) irrespective of HPV type*: intention to treat
analysis
*1.8. Risk ratio for GlaxoSmithKline studies (i.e., HPV-0xx): not applicable; risk ratio for Merck Sharp & Dohme studies (i.e., V50x-xxx): 0.56 [0.39, 0.82].
9
Page 349 of 637
2. Anal outcomes
2.1. Mortality from anal cancer irrespective of HPV type: intention to treat analysis
No cases/data/reports.
2.2. Incidence of anal cancer irrespective of HPV type: intention to treat analysis
2.3. Incidence of high grade anal intraepithelial neoplasia (AIN3) irrespective of HPV type: intention to treat
analysis
2.4. Incidence of moderate grade anal intraepithelial neoplasia (AIN2) irrespective of HPV type: intention to
treat analysis
2.5. Incidence of high grade anal intraepithelial neoplasia or worse (AIN3+) irrespective of HPV type: intention
to treat analysis
2.6. Incidence of moderate grade anal intraepithelial neoplasia or worse (AIN2+) irrespective of HPV type:
intention to treat analysis
10
Page 350 of 637
3. Cervical outcomes
3.1. Mortality from cervical cancer irrespective of HPV type*: intention to treat analysis
*3.1. Risk ratio for GlaxoSmithKline studies (i.e., HPV-0xx): 2.99 [0.12, 73.33]; risk ratio for Merck Sharp & Dohme studies (i.e., V50x-xxx): not applicable.
11
Page 351 of 637
3.2. Incidence of cervical cancer irrespective of HPV type*: intention to treat analysis
12
Page 352 of 637
3.3. Incidence of cervical adenocarcinoma in situ (AIS) irrespective of HPV type*: intention to treat analysis
13
Page 353 of 637
3.4. Incidence of cervical intraepithelial neoplasia grade 3 (CIN3) irrespective of HPV type*: intention to treat
analysis
14
Page 354 of 637
3.5. Incidence of cervical intraepithelial neoplasia grade 2 (CIN2) irrespective of HPV type*: intention to treat
analysis
15
Page 355 of 637
3.6. Incidence of cervical intraepithelial neoplasia grade 3 or worse (CIN3+) irrespective of HPV type*:
16
Page 356 of 637
3.7. Incidence of cervical intraepithelial neoplasia grade 2 or worse (CIN2+) irrespective of HPV type*:
17
Page 357 of 637
4. Oropharyngeal outcomes
4.1. Mortality from oropharyngeal cancer irrespective of HPV type*: intention to treat analysis
18
Page 358 of 637
4.2. Incidence of oropharyngeal cancer irrespective of HPV type*: intention to treat analysis
19
Page 359 of 637
4.3. Incidence of oropharyngeal carcinoma in situ irrespective of HPV type*: intention to treat analysis
Not applicable.
20
Page 360 of 637
5. Penile outcomes
5.1. Mortality from penile cancer irrespective of HPV type: intention to treat analysis
5.2. Incidence of penile cancer irrespective of HPV type: intention to treat analysis
5.3. Incidence of penile intraepithelial neoplasia grade 3 (PIN3) irrespective of HPV type: intention to treat
analysis
5.4. Incidence of penile intraepithelial neoplasia grade 2 (PIN2) irrespective of HPV type: intention to treat
analysis
5.5. Incidence of penile intraepithelial neoplasia grade 3 or worse (PIN3+) irrespective of HPV type: intention
to treat analysis
21
Page 361 of 637
5.6. Incidence of penile intraepithelial neoplasia grade 2 or worse (PIN2+) irrespective of HPV type*: intention
to treat analysis
22
Page 362 of 637
6. Vaginal outcomes
6.1. Mortality from vaginal cancer irrespective of HPV type: intention to treat analysis
23
Page 363 of 637
6.2. Incidence of vaginal cancer irrespective of HPV type*: intention to treat analysis
24
Page 364 of 637
6.3. Incidence of vaginal intraepithelial neoplasia grade 3 (VaIN3) irrespective of HPV type: intention to treat
analysis
6.4. Incidence of vaginal intraepithelial neoplasia grade 2 (VaIN2) irrespective of HPV type: intention to treat
analysis
6.5. Incidence of vaginal intraepithelial neoplasia grade 3 or worse (VaIN3+) irrespective of HPV type: intention
to treat analysis
25
Page 365 of 637
6.6. Incidence of vaginal intraepithelial neoplasia grade 2 or worse (VaIN2+) irrespective of HPV type*:
26
Page 366 of 637
7. Vulvar outcomes
7.1. Mortality from vulvar cancer irrespective of HPV type: intention to treat analysis
27
Page 367 of 637
7.2. Incidence of vulvar cancer irrespective of HPV type*: intention to treat analysis
*7.2. Risk ratio for GlaxoSmithKline studies (i.e., HPV-0xx): 3.00 [0.12, 73.68]; risk ratio for Merck Sharp & Dohme studies (i.e., V50x-xxx): 3.00 [0.12,
73.61].
28
Page 368 of 637
7.3. Incidence of VIN3 irrespective of HPV type: intention to treat analysis
7.4. Incidence of VIN2 irrespective of HPV type: intention to treat analysis

7.5. Incidence of VIN3+ irrespective of HPV type: intention to treat analysis

29
Page 369 of 637
7.6. Incidence of VIN2+ irrespective of HPV type*: intention to treat analysis
30
Page 370 of 637
8. Referral procedures
8.1. Number of participants with ‘any’ referral procedures performed (endoscopy, biopsy, surgical treatment
and non-surgical treatment) due to HPV-related diseases irrespective of HPV type*: intention to treat
analysis
31
Page 371 of 637
8.2. Number of participants with endoscopies performed (anoscopy, colposcopy and rectoscopy) due to HPV-
related diseases irrespective of HPV type*: intention to treat analysis
There were no reports of anoscopy or rectoscopy irrespective of HPV type.
32
Page 372 of 637
8.3. Number of participants with biopsies performed due to HPV-related diseases irrespective of HPV type*:
Only cervical biopsy: 2,006 in the HPV vaccine group vs. 2,267 in the control group, risk ratio 0.89 [0.83, 0.95]; only EGL biopsy: 443 vs. 754, risk ratio 0.62
[0.50, 0.76].
33
Page 373 of 637
8.4. Number of treatment procedures (both surgical and non-surgical treatment) due to HPV-related diseases
Only cervical procedure: 844 in the HPV vaccine group vs. 1,139 in the control group, risk ratio 0.74 [0.65, 0.84]; only EGL procedure: 174 vs. 277, risk ratio
0.63 [0.50, 0.80].
34
Page 374 of 637
9. Fatal harms
9.1. Number of participants with fatal harms*: intention to treat analysis
The most common fatal serious harms were: ‘road traffic accident’ (five in the HPV vaccine group and seven in the control group, risk ratio 0.77 [0.24,
2.46]); ‘completed suicide’ (four and eight, risk ratio 0.58 [0.15, 2.19]); ‘cardiorespiratory arrest’ (three and two, risk ratio 0.99 [0.13, 7.65]); ‘gunshot
wound’ (two and three, risk ratio 0.74 [0.09, 5.85]); and ‘homicide’ (two and two, risk ratio 0.95 [0.14, 6.50]). The fatal serious harms most increased by
the HPV vaccines were: ‘cardiac arrest’ (two in the HPV vaccine group and none in the control group, risk ratio 3.00 [0.31, 28.82]); ‘traumatic intracranial
haemorrhage’ (two and none, risk ratio 3.00 [0.31, 28.82]); ‘systemic lupus erythematosus’ (two and none, risk ratio 3.00 [0.31, 28.82]); ‘metastases to
lung’ (two and none, risk ratio 3.00 [0.31, 28.82]); and ‘renal failure acute’ (two and none, risk ratio 3.00 [0.31, 28.82]). The fatal serious harms most
decreased by the HPV vaccines were: ‘completed suicide’ (four in the HPV vaccine group and eight in the control group, risk ratio 0.58 [0.15, 2.19]); and
‘road traffic accident’ (five and seven, risk ratio 0.77 [0.24, 2.46]).
35
Page 375 of 637
9.2. Fatal harms judged related to the HPV vaccine by study investigator: intention to treat analysis
No fatal harm was judged HPV vaccine-related by the study investigators.
36
Page 376 of 637
10. Serious harms
10.1. Serious harms*: intention to treat analysis
37
Page 377 of 637
10.2. Serious harms judged related to the HPV vaccine by study investigator*: intention to treat analysis
38
Page 378 of 637
10.3. Number of participants that withdrew due to a serious harm*: intention to treat analysis
39
Page 379 of 637
10.4. Serious harms reported within the MedDRA system organ class 'blood and lymphatic system disorders
(10005329)'*: intention to treat analysis
40
Page 380 of 637
10.5. Serious harms reported within the MedDRA system organ class 'cardiac disorders (10007541)'*: intention
to treat analysis
41
Page 381 of 637
10.6. Serious harms reported within the MedDRA system organ class 'congenital familial and genetic disorders
42
Page 382 of 637
10.7. Serious harms reported within the MedDRA system organ class 'ear and labyrinth disorders (10013993)'*:
73.48].
43
Page 383 of 637
10.8. Serious harms reported within the MedDRA system organ class 'endocrine disorders (10014698)'*:
73.48].
44
Page 384 of 637
10.9. Serious harms reported within the MedDRA system organ class 'eye disorders (10015919)'*: intention to
treat analysis
45
Page 385 of 637
10.10. Serious harms reported within the MedDRA system organ class 'gastrointestinal disorders (10017947)'*:
1.46].
46
Page 386 of 637
10.11. Serious harms reported within the MedDRA system organ class 'general disorders and administration site
conditions (10018065)'*: intention to treat analysis
12.14].
47
Page 387 of 637
10.12. Serious harms reported within the MedDRA system organ class 'hepatobiliary disorders (10019805)'*:
16.68].
48
Page 388 of 637
10.13. Serious harms reported within the MedDRA system organ class 'immune system disorders (10021428)'*:
10.47].
49
Page 389 of 637
10.14. Serious harms reported within the MedDRA system organ class 'infections and infestations (10021881)'*:
3.33].
50
Page 390 of 637
10.15. Serious harms reported within the MedDRA system organ class 'injury poisoning and procedural
complications (10022117)'*: intention to treat analysis
1.12].
51
Page 391 of 637
10.16. Serious harms reported within the MedDRA system organ class 'investigations (10022891)'*: intention to
treat analysis
52
Page 392 of 637
10.17. Serious harms reported within the MedDRA system organ class 'metabolism and nutrition disorders
12.40].
53
Page 393 of 637
10.18. Serious harms reported within the MedDRA system organ class 'musculoskeletal and connective tissue
disorders (10028395)'*: intention to treat analysis
3.49].
54
Page 394 of 637
10.19. Serious harms reported within the MedDRA system organ class 'neoplasms benign, malignant and
unspecified (incl. cysts and polyps) (10029104)'*: intention to treat analysis
11.17].
55
Page 395 of 637
10.20. Serious harms reported within the MedDRA system organ class 'nervous system disorders (10029205)'*:
3.97].
56
Page 396 of 637
- Reported MedDRA preferred terms and number of harms per MedDRA term for serious nervous system disorders:
MedDRA system organ class MedDRA preferred term HPV vaccine Comparator
Nervous system disorders (10029205) Altered state of consciousness 0 1
Nervous system disorders (10029205) Anoxic encephalopathy (10050750) 1 0
Nervous system disorders (10029205) Aphasia (10002948) 1 0
Nervous system disorders (10029205) Basal ganglia haemorrhage 0 1
Nervous system disorders (10029205) Benign intracranial hypertension (10004277) 0 1
Nervous system disorders (10029205) Brain injury (10067967) 0 1
Nervous system disorders (10029205) Cerebral cyst 1 0
Nervous system disorders (10029205) Cerebral haemorrhage (10008111) 2 2
Nervous system disorders (10029205) Cerebral ischemia 0 1
Nervous system disorders (10029205) Cerebrovascular accident (10008190) 2 2
Nervous system disorders (10029205) Convulsion (10010904) 5 5
Nervous system disorders (10029205) Cubital tunnel syndrome (10056473) 1 0
Nervous system disorders (10029205) Diabetic coma 1 0
Nervous system disorders (10029205) Dizziness (10013573) 5 2
Nervous system disorders (10029205) Dystonia (10013983) 1 0
Nervous system disorders (10029205) Epilepsy (10015037) 3 3
Nervous system disorders (10029205) Facial palsy (10016060) 2 2
Nervous system disorders (10029205) Facial paresis (10051267) 1 0
Nervous system disorders (10029205) Grand mal convulsion (10018659) 2 0
Nervous system disorders (10029205) Haemorrhage intracranial 1 0
Nervous system disorders (10029205) Headache (10019211) 11 4
Nervous system disorders (10029205) Hydrocephalus (10020508) 0 1
Nervous system disorders (10029205) Intracranial hematoma 0 1
Nervous system disorders (10029205) Ischemic stroke 0 1
Nervous system disorders (10029205) Intracranial aneurysm (10022758) 1 1
Nervous system disorders (10029205) Intracranial venous sinus thrombosis (10061251) 1 0
Nervous system disorders (10029205) Migraine (10027599) 9 3
Nervous system disorders (10029205) Migraine with aura (10027607) 0 1
Nervous system disorders (10029205) Monoparesis 0 1
Nervous system disorders (10029205) Moyamoya disease 0 1
Nervous system disorders (10029205) Multiple sclerosis (10028245) 4 1
Nervous system disorders (10029205) Neuropathy peripheral (10029331) 0 1
Nervous system disorders (10029205) Not specified 1 0
Nervous system disorders (10029205) Optic neuritis (10030942) 3 1
Nervous system disorders (10029205) Paraesthesia (10033775) 1 1
Nervous system disorders (10029205) Peroneal nerve palsy (10034701) 0 1
Nervous system disorders (10029205) Pleocytosis (10035551) 0 1
Nervous system disorders (10029205) Polyneuropathy (10036105) 1 0
Nervous system disorders (10029205) Sciatica (10039674) 0 1
Nervous system disorders (10029205) Sedation 1 0
Nervous system disorders (10029205) Subarachnoid haemorrhage (10042316) 1 1
Nervous system disorders (10029205) Syncope (10042772) 4 3
Nervous system disorders (10029205) Tension headache (10043269) 3 0
Nervous system disorders (10029205) Trigeminal neuralgia 1 0
Nervous system disorders (10029205) Vertebral artery dissection 1 0
Total 72 46
57
Page 397 of 637
10.21. Serious harms reported within the MedDRA system organ class 'pregnancy, puerperium and perinatal
conditions (10036585)'*: intention to treat analysis
1.49].
58
Page 398 of 637
10.22. Serious harms reported within the MedDRA system organ class 'psychiatric disorders (10037175)'*:
1.28].
59
Page 399 of 637
10.23. Serious harms reported within the MedDRA system organ class 'renal and urinary disorders (10038359)'*:
4.13].
60
Page 400 of 637
10.24. Serious harms reported within the MedDRA system organ class 'reproductive system and breast disorders
2.92].
61
Page 401 of 637
10.25. Serious harms reported within the MedDRA system organ class 'respiratory, thoracic and mediastinal
disorders (10038738)'*: intention to treat analysis
3.46].
62
Page 402 of 637
10.26. Serious harms reported within the MedDRA system organ class 'skin and subcutaneous tissue disorders
3.63].
63
Page 403 of 637
10.27. Serious harms reported within the MedDRA system organ class 'social circumstances (10041244)'*:
64
Page 404 of 637
10.28. Serious harms reported within the MedDRA system organ class 'surgical and medical procedures
(10042613)': intention to treat analysis
65
Page 405 of 637
10.29. Serious harms reported within the MedDRA system organ class 'vascular disorders (10047065)': intention
to treat analysis
6.89].
66
Page 406 of 637
10.30. Most common serious harms - ‘abortion spontaneous’*: intention to treat analysis
The total risk ratio for all included serious abortion categories (‘abortion complete,’ ‘abortion incomplete,’ ‘abortion missed,’ ‘abortion spontaneous,’
‘abortion spontaneous complete,’ ‘abortion spontaneous incomplete’ and ‘abortion threatened’) is 1.08 [0.91, 1.28] (271/13,271 vs. 254/13,272).
67
Page 407 of 637
10.31. Most common serious harms - ‘appendicitis’*: intention to treat analysis
6.68]. The risk ratio for all serious appendicitis categories (‘appendicitis’ and ‘appendicitis perforated’) is 0.84 [0.61, 1.15] (74/30,071 vs. 85/28,003).
68
Page 408 of 637
10.32. Most common serious harms - ‘abortion spontaneous incomplete’*: intention to treat analysis
69
Page 409 of 637
10.33. Most common serious harms - ‘abortion spontaneous complete’*: intention to treat analysis
70
Page 410 of 637
10.34. Most common serious harms - ‘abortion missed’*: intention to treat analysis
71
Page 411 of 637
10.35. Serious harms most increased by the HPV vaccines - ‘abortion spontaneous incomplete’*: intention to
treat analysis
See analysis 10.32.
72
Page 412 of 637
10.36. Serious harms most increased by the HPV vaccines - ‘pyelonephritis’*: intention to treat analysis
4.16]. The risk ratio of ‘pyelonephritis’ together with ‘pyelonephritis acute’ is 1.56 [0.72, 3.35] (53/21,410 vs. 32/24,498)
73
Page 413 of 637
10.37. Serious harms most increased by the HPV vaccines - ‘abortion spontaneous’*: intention to treat analysis
See analysis 10.30.
74
Page 414 of 637
10.38. Serious harms most increased by the HPV vaccines - ‘pneumonia’*: intention to treat analysis
4.02]. The risk ratio of all serious pneumonia categories (‘pneumonia,’ ‘pneumonia bacterial,’ ‘pneumonia influenza,’ ‘pneumonia mycoplasmal’ and
‘pneumonia streptococcal’) is 1.26 [0.72, 2.23] (29/26,946 vs. 22/25,765).
75
Page 415 of 637
10.39. Serious harms most increased by the HPV vaccines - ‘tonsillitis’*: intention to treat analysis
36.42]. The risk ratio of all serious tonsillitis categories (‘tonsillitis’ and ‘tonsillitis streptococcal’) is 1.54 [0.71, 3.33] (19/17,987 vs. 10/16,501).
76
Page 416 of 637
10.40. Serious harms most decreased by the HPV vaccines - ‘appendicitis’*: intention to treat analysis
See analysis 10.31.
77
Page 417 of 637
10.41. Serious harms most decreased by the HPV vaccines - ‘overdose’*: intention to treat analysis
1.26].
78
Page 418 of 637
10.42. Serious harms most decreased by the HPV vaccines - ‘abortion missed’*: intention to treat analysis
79
Page 419 of 637
10.43. Serious harms most decreased by the HPV vaccines - ‘ligament rupture’*: intention to treat analysis
Risk ratio for all serious ligament injury categories (‘ligament injury,’ ‘ligament rupture’ and ‘ligament sprain’): 0.51 [0.21, 1.29] (7/12,576 vs. 14/12,573).
80
Page 420 of 637
10.44. Serious harms most decreased by the HPV vaccines - ‘ovarian cyst rupture’*: intention to treat analysis
15.98].
81
Page 421 of 637
11. New onset diseases
11.1. New onset diseases ('medically significant conditions' and 'new medical history'*): intention to treat
analysis
*11.1. Risk ratio for 'medically significant conditions’ (GlaxoSmithKline): 0.98 [0.90, 1.06]; risk ratio for ‘new medical history’ (Merck Sharp & Dohme): 1.00
[0.97, 1.03]; risk ratio for the follow-up periods for the trials V501-005, V501-019 and V501-020 (Merck Sharp & Dohme): 0.98 [0.94, 1.01] (2,296
participants with new medical history in the HPV vaccine group vs. 2,365 participants with new medical history in the control group. The trials V501-005,
V501-019 and V501-020 split the reporting of new onset diseases into the vaccination period and the follow-up period. To avoid double counting of
participants in the total risk ratio estimate, we only included the new onset diseases reported in the vaccination period for the trials V501-005, V501-019
and V501-020).
82
Page 422 of 637
11.2. New onset diseases ('medically significant conditions' and 'new medical history'*) reported within the
MedDRA system organ class ‘blood and lymphatic system disorders (10005329)': intention to treat
analysis
[0.86, 1.13]; risk ratio for the follow-up periods for the trials V501-019 and V501-020: 1.14 [0.73, 1.77]. The trials V501-005, V501-019 and V501-020 split
the reporting of new onset diseases into the vaccination period and the follow-up period. To avoid double counting of participants in the total risk ratio
estimate, we only included the new onset diseases reported in the vaccination period for the trials V501-019 and V501-020.
83
Page 423 of 637
MedDRA system organ class ‘cardiac disorders (10007541)': intention to treat analysis
[0.33, 2.03]; risk ratio for the follow-up periods for the trials V501-019 and V501-020: 0.70 [0.30, 1.67]. The trials V501-019 and V501-020 split the
reporting of new onset diseases into the vaccination period and the follow-up period. To avoid double counting of participants in the total risk ratio
84
Page 424 of 637
MedDRA system organ class ‘congenital familial and genetic disorders (10010331)': intention to treat
analysis
85
Page 425 of 637
MedDRA system organ class ‘ear and labyrinth disorders (10013993)': intention to treat analysis
86
Page 426 of 637
MedDRA system organ class ‘endocrine disorders (10014698)': intention to treat analysis
87
Page 427 of 637
MedDRA system organ class ‘eye disorders (10015919)': intention to treat analysis
[0.74, 1.08]; risk ratio for the follow-up periods for the trials V501-005, V501-019 and V501-020: 1.03 [0.75, 1.41]. The trials V501-005, V501-019 and
V501-020 split the reporting of new onset diseases into the vaccination period and the follow-up period. To avoid double counting of participants in the
total risk ratio estimate, we only included the new onset diseases reported in the vaccination period for the trials V501-005, V501-019 and V501-020.
88
Page 428 of 637
MedDRA system organ class ‘gastrointestinal disorders (10017947)': intention to treat analysis
89
Page 429 of 637
MedDRA system organ class ‘general disorders and administration site conditions (10018065)': intention
to treat analysis
90
Page 430 of 637
MedDRA system organ class ‘hepatobiliary disorders (10019805)': intention to treat analysis
*11.10. Risk ratio for 'medically significant conditions’ (GlaxoSmithKline): 0.93 [0.69, 1.26]; risk ratio for ‘new medical history’ (Merck Sharp & Dohme):
0.81 [0.31, 2.15]; risk ratio for the follow-up period for the trial V501-019: 1.20 [0.52, 2.77]. The trial V501-019 split the reporting of new onset diseases
into the vaccination period and the follow-up period. To avoid double counting of participants in the total risk ratio estimate, we only included the new
onset diseases reported in the vaccination period for the trial V501-019.
91
Page 431 of 637
MedDRA system organ class ‘immune system disorders (10021428)': intention to treat analysis
1.04 [0.88, 1.22]; risk ratio for the follow-up periods for the trials V501-005, V501-019 and V501-020: 0.95 [0.68, 1.34]. The trials V501-005, V501-019 and
92
Page 432 of 637
MedDRA system organ class ‘infections and infestations (10021881)': intention to treat analysis
total risk ratio estimate, we only included the new onset diseases reported in the vaccination fperiod for the trials V501-005, V501-019 and V501-020.
93
Page 433 of 637
MedDRA system organ class ‘injury poisoning and procedural complications (10022117)': intention to
treat analysis
94
Page 434 of 637
MedDRA system organ class ‘investigations (10022891)': intention to treat analysis
95
Page 435 of 637
MedDRA system organ class ‘metabolism and nutrition disorders (10027433)': intention to treat analysis
1.12 [0.91, 1.37]; risk ratio for the follow-up periods for the trials V501-019 and V501-020: 1.10 [0.83, 1.47]. The trials V501-019 and V501-020 split the
96
Page 436 of 637
MedDRA system organ class ‘musculoskeletal and connective tissue disorders (10028395)': intention to
treat analysis
97
Page 437 of 637
MedDRA system organ class ‘neoplasms benign, malignant and unspecified (incl. cysts and polyps)
(10029104)': intention to treat analysis
98
Page 438 of 637
MedDRA system organ class ‘nervous system disorders (10029205)': intention to treat analysis
99
Page 439 of 637
MedDRA system organ class ‘pregnancy, puerperium and perinatal conditions (10036585)': intention to
treat analysis
100
Page 440 of 637
MedDRA system organ class ‘psychiatric disorders (10037175)': intention to treat analysis
101
Page 441 of 637
MedDRA system organ class ‘renal and urinary disorders (10038359)': intention to treat analysis
1.02 [0.87, 1.19] risk ratio for the follow-up periods for the trials V501-005, V501-019 and V501-020: 0.77 [0.53, 1.10]. The trials V501-005, V501-019 and
102
Page 442 of 637
MedDRA system organ class ‘reproductive system and breast disorders (10038604)': intention to treat
analysis
103
Page 443 of 637
MedDRA system organ class ‘respiratory, thoracic and mediastinal disorders (10038738)': intention to
treat analysis
104
Page 444 of 637
MedDRA system organ class ‘skin and subcutaneous tissue disorders (10040785)': intention to treat
analysis
105
Page 445 of 637
MedDRA system organ class ‘social circumstances (10041244)': intention to treat analysis
106
Page 446 of 637
MedDRA system organ class ‘surgical and medical procedures (10042613)': intention to treat analysis
107
Page 447 of 637
MedDRA system organ class ‘vascular disorders (10047065)': intention to treat analysis
108
Page 448 of 637
11.28. Most common new onset diseases (‘medically significant conditions’*) - ‘gynaecological chlamydia
infection’: intention to treat analysis
109
Page 449 of 637
11.29. Most common new onset diseases (‘medically significant conditions’*) - ‘depression’: intention to treat
analysis
110
Page 450 of 637
11.30. Most common new onset diseases (‘medically significant conditions’*) - ‘genitourinary tract gonococcal
111
Page 451 of 637
11.31. Most common new onset diseases (‘new medical history’*) - ‘vaginal candidiasis’: intention to treat
analysis
*11.31. Risk ratio for 'medically significant conditions’ (GlaxoSmithKline): not applicable; risk ratio for ‘new medical history’ (Merck Sharp & Dohme): 0.95
[0.89, 1.02]; risk ratio for the follow-up period for the trial V501-005: 1.12 [0.81, 1.55]. The trial V501-005 split the reporting of new onset diseases into
the vaccination period and the follow-up period. To avoid double counting of participants in the total risk ratio estimate, we only included the new onset
diseases reported in the vaccination period for the trial V501-005.
112
Page 452 of 637
11.32. Most common new onset diseases (‘new medical history’*) - ‘vaginitis bacterial’: intention to treat
analysis
113
Page 453 of 637
11.33. Most common new onset diseases (‘new medical history’*) - ‘urinary tract infection’: intention to treat
analysis
114
Page 454 of 637
11.34. New onset diseases most increased by the HPV vaccines (‘medically significant conditions’*) - ‘back pain’:
115
Page 455 of 637
11.35. New onset diseases most increased by the HPV vaccines (‘medically significant conditions’*) - ‘abdominal
pain’: intention to treat analysis
116
Page 456 of 637
11.36. New onset diseases most increased by the HPV vaccines (‘medically significant conditions’*) - ‘headache’:
117
Page 457 of 637
11.37. New onset diseases most increased by the HPV vaccines (‘new medical history’*) - ‘headache’: intention
to treat analysis
See analysis 11.36.
118
Page 458 of 637
11.38. New onset diseases most increased by the HPV vaccines (‘new medical history’*) - ‘joint sprain’: intention
to treat analysis
119
Page 459 of 637
11.39. New onset diseases most increased by the HPV vaccines (‘new medical history’*) - ‘amenorrhoea’:
120
Page 460 of 637
11.40. New onset diseases most decreased by the HPV vaccines (‘medically significant conditions’*) -
‘gynaecological chlamydia infection’: intention to treat analysis
See analysis 11.28.
121
Page 461 of 637
11.41. New onset diseases most decreased by the HPV vaccines (‘medically significant conditions’*) - ‘cystitis’:
122
Page 462 of 637
11.42. New onset diseases most decreased by the HPV vaccines (‘medically significant conditions’*) - ‘type 2
diabetes mellitus’: intention to treat analysis
123
Page 463 of 637
11.43. New onset diseases most decreased by the HPV vaccines (‘new medical history’*) - ‘vaginal infection’:
[0.76, 1.00]; risk ratio for the follow-up period for the trial V501-019: 0.61 [0.26, 1.48]. The trial V501-019 split the reporting of new onset diseases into
the vaccination period and the follow-up period. To avoid double counting of participants in the total risk ratio estimate, we only included the new onset
diseases reported in the vaccination period for the trial V501-019.
124
Page 464 of 637
11.44. New onset diseases most decreased by the HPV vaccines (‘new medical history’*) - ‘vaginal candidiasis’:
See analysis 11.31.
11.45. New onset diseases most decreased by the HPV vaccines (‘new medical history’*) - ‘urinary tract
See analysis 11.33.
125
Page 465 of 637
12. General harms
12.1. General harms (‘solicited and unsolicited’ and ‘systemic adverse events’*): intention to treat analysis
*12.1. Risk ratio for ‘solicited and solicited’ (GlaxoSmithKline): 1.11 [1.06, 1.16]; risk ratio for ‘systemic adverse events’ (Merck Sharp & Dohme): 1.01
[0.98, 1.03]. The total numbers of participants with general harms in GlaxoSmithKline studies were reported as ‘solicited [SGAE] and unsolicited [UGAE]’
combined.
126
Page 466 of 637
12.2. General harms (‘solicited,’ ‘unsolicited’ and ‘systemic adverse events’*) reported within the MedDRA
system organ class ‘blood and lymphatic system disorders (10005329)': intention to treat analysis
*12.2. Risk ratio for ‘solicited’ (GlaxoSmithKline): not applicable; risk ratio for ‘unsolicited’ (GlaxoSmithKline): 0.65 [0.15, 2.89]; risk ratio for ‘systemic
adverse events’ (Merck Sharp & Dohme): 1.85 [0.55, 6.25].
127
Page 467 of 637
system organ class ‘cardiac disorders (10007541)': intention to treat analysis
128
Page 468 of 637
system organ class ‘congenital familial and genetic disorders (10010331)': intention to treat analysis
adverse events’ (Merck Sharp & Dohme): not applicable.
129
Page 469 of 637
system organ class ‘ear and labyrinth disorders (10013993)': intention to treat analysis
130
Page 470 of 637
system organ class ‘endocrine disorders (10014698)': intention to treat analysis
131
Page 471 of 637
system organ class ‘eye disorders (10015919)': intention to treat analysis
132
Page 472 of 637
system organ class ‘gastrointestinal disorders (10017947)': intention to treat analysis
*12.8. Risk ratio for 'solicited’ (i.e., ‘gastrointestinal symptoms’) only: 1.12 [1.02, 1.22]; risk ratio for ‘unsolicited’ (GlaxoSmithKline): 1.05 [0.58, 1.90]; risk
ratio for ‘systemic adverse events’ (Merck Sharp & Dohme): 0.98 [0.85, 1.13]. To avoid double counting of participants in the total risk ratio estimate, we
excluded the ‘unsolicited’ adverse events from total risk ratio estimate for studies that reported ‘solicited’ adverse events.
133
Page 473 of 637
system organ class ‘general disorders and administration site conditions (10018065)': intention to treat
analysis
*12.9. Risk ratio for ‘solicited’ (GlaxoSmithKline): 1.11 [1.05, 1.19]; risk ratio for ‘unsolicited’ (GlaxoSmithKline): 1.31 [0.99, 1.74]; risk ratio for ‘systemic
adverse events’ (Merck Sharp & Dohme): 1.01 [0.84, 1.22]. To avoid double counting of participants in the total risk ratio estimate, we excluded the
‘unsolicited’ adverse events from total risk ratio estimate for studies that reported ‘solicited’ adverse events.
134
Page 474 of 637
system organ class ‘hepatobiliary disorders (10019805)': intention to treat analysis
135
Page 475 of 637
system organ class ‘immune system disorders (10021428)': intention to treat analysis
136
Page 476 of 637
system organ class ‘infections and infestations (10021881)': intention to treat analysis
137
Page 477 of 637
system organ class ‘injury poisoning and procedural complications (10022117)': intention to treat analysis
138
Page 478 of 637
system organ class ‘investigations (10022891)': intention to treat analysis
139
Page 479 of 637
system organ class ‘metabolism and nutrition disorders (10027433)': intention to treat analysis
140
Page 480 of 637
system organ class ‘musculoskeletal and connective tissue disorders (10028395)': intention to treat
analysis
141
Page 481 of 637
system organ class ‘neoplasms benign, malignant and unspecified (incl. cysts and polyps) (10029104)':
142
Page 482 of 637
system organ class ‘nervous system disorders (10029205)': intention to treat analysis
*12.18. Risk ratio for 'solicited’ only: 1.08 [1.03, 1.14]; risk ratio for ‘unsolicited’ (GlaxoSmithKline): 1.62 [1.17, 2.24]; risk ratio for ‘systemic adverse
events’ (Merck Sharp & Dohme): 1.10 [0.96, 1.26]. To avoid double counting of participants in the total risk ratio estimate, we excluded the ‘unsolicited’
adverse events from total risk ratio estimate for studies that reported ‘solicited’ adverse events.
143
Page 483 of 637
system organ class ‘pregnancy, puerperium and perinatal conditions (10036585)': intention to treat
analysis
144
Page 484 of 637
system organ class ‘psychiatric disorders (10037175)': intention to treat analysis
145
Page 485 of 637
system organ class 'renal and urinary disorders (10038359)': intention to treat analysis
146
Page 486 of 637
system organ class ‘reproductive system and breast disorders (10038604)': intention to treat analysis
147
Page 487 of 637
system organ class ‘respiratory, thoracic and mediastinal disorders (10038738)': intention to treat analysis
148
Page 488 of 637
system organ class ‘skin and subcutaneous tissue disorders (10040785)': intention to treat analysis
149
Page 489 of 637
system organ class ‘social circumstances (10041244)': intention to treat analysis
150
Page 490 of 637
12.26. General harms (‘solicited,’ ‘unsolicited’ and ‘systemic adverse events’) reported within the MedDRA
system organ class ‘surgical and medical procedures (10042613)': intention to treat analysis
151
Page 491 of 637
system organ class ‘vascular disorders (10047065)': intention to treat analysis
152
Page 492 of 637
12.28. Most common general harms (‘solicited’ and ‘unsolicited’*) - ‘fatigue’: intention to treat analysis
153
Page 493 of 637
12.29. Most common general harms (‘solicited’ and ‘unsolicited’*) - ‘headache’: intention to treat analysis
154
Page 494 of 637
12.30. Most common general harms (‘solicited’ and ‘unsolicited’*) - ‘myalgia’: intention to treat analysis
155
Page 495 of 637
12.31. Most common general harms (‘systemic adverse events’) - ‘headache’: intention to treat analysis
See analysis 12.29.
156
Page 496 of 637
12.32. Most common general harms (‘systemic adverse events’*) - ‘pyrexia’: intention to treat analysis
157
Page 497 of 637
12.33. Most common general harms (‘systemic adverse events’*) - ‘nasopharyngitis’: intention to treat analysis
158
Page 498 of 637
12.34. General harms most increased by the HPV vaccines (‘solicited’ and ‘unsolicited’) - ‘myalgia’: intention to
treat analysis
See analysis 12.30.
12.35. General harms most increased by the HPV vaccines (‘solicited’ and ‘unsolicited’) - ‘fatigue’: intention to
treat analysis
See analysis 12.28.
12.36. General harms most increased by the HPV vaccines (‘solicited’ and ‘unsolicited’) - ‘headache’: intention to
treat analysis
See analysis 12.29.
12.37. General harms most increased by the HPV vaccines (‘systemic adverse events’) - ‘myalgia’: intention to
treat analysis
See analysis 12.30.
12.38. General harms most increased by the HPV vaccines (‘systemic adverse events’) - ‘pyrexia’: intention to
treat analysis
See analysis 12.32.
159
Page 499 of 637
12.39. General harms most increased by the HPV vaccines (‘systemic adverse events’*) - ‘nausea’: intention to
treat analysis
160
Page 500 of 637
12.40. General harms most decreased by the HPV vaccines (‘solicited’ and ‘unsolicited’*) - ‘influenza’: intention
to treat analysis
161
Page 501 of 637
12.41. General harms most decreased by the HPV vaccines (‘solicited’ and ‘unsolicited’*) - ‘cough’: intention to
treat analysis
162
Page 502 of 637
12.42. General harms most decreased by the HPV vaccines (‘solicited’ and ‘unsolicited’*) - ‘oropharyngeal pain’:
163
Page 503 of 637
12.43. General harms most decreased by the HPV vaccines (‘systemic adverse events’*) - ‘fungal infection’:
164
Page 504 of 637
12.44. General harms most decreased by the HPV vaccines (‘systemic adverse events’*) - ‘sinus headache’:
*12.44. Risk ratio for ‘solicited’ (GlaxoSmithKline): not applicable; risk ratio for ‘unsolicited’ (GlaxoSmithKline): not applicable; risk ratio for ‘systemic
165
Page 505 of 637
12.45. General harms most decreased by the HPV vaccines (‘systemic adverse events’*) - ‘joint injury’: intention
to treat analysis
166
Page 506 of 637
13. Harms of special interest
13.1. Anaphylaxis*: intention to treat analysis
*13.1. Risk ratio for serious anaphylactic shock or reaction: 0.59 [0.13, 2.82]. Risk ratio for new onset anaphylactic shock or reaction: 1.18 [0.48, 2.91].
167
Page 507 of 637
13.2. Chronic fatigue syndrome (CFS): intention to treat analysis
13.3. Chronic regional pain syndrome (CRPS): intention to treat analysis

13.4. Guillain–Barré syndrome (GBS): intention to treat analysis

168
Page 508 of 637
13.5. Premature ovarian failure (POF): intention to treat analysis
[0.12, 73.48]
169
Page 509 of 637
13.6. Postural orthostatic tachycardia syndrome (POTS): intention to treat analysis
170
Page 510 of 637
13.7. Syncope: intention to treat analysis
*13.7. Risk ratio for serious syncope: 0.94 [0.23, 3.81]. Risk ratio for new onset syncope: 1.03 [0.58, 1.84]. Risk ratio for general syncope: 0.77 [0.25, 2.34].
171
Page 511 of 637
14. Post hoc exploratory harm analyses
14.1. Serious harms judged as ‘definitely associated’* with chronic regional pain syndrome (CRPS): intention to
treat analysis
We asked a physician with clinical expertise in CRPS to assess the reported MedDRA preferred terms as 'definitely,' 'probably,' 'probably not' or ‘definitely
not’ associated with the syndromes. We sent an Excel sheet to the physician with all the reported MedDRA terms. The physician was blinded, as the Excel
sheet contained no outcome data. When the physician had assessed all the MedDRA terms, we synthesized the data for those MedDRA terms that the
physician judged ‘definitely’ associated with CRPS and compared it to the reported serious harms.
172
Page 512 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the serious harms that were judged ‘definitely associated’ with chronic regional
pain syndrome (CRPS):
HPV Compara
Physician judgment MedDRA system organ class MedDRA preferred term
vaccine tor
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Arrhythmia (10003119)) 2 0
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Sinus tachycardia (10040752) 2 1
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Supraventricular tachycardia (10042604) 3 1
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Tachycardia (10043071) 1 0
‘Definitely associated with CRPS’ Ear and labyrinth disorders (10013993) Vertigo (10047340) 3 2
‘Definitely associated with CRPS’ Eye disorders (10015919) Vision blurred (10047513) 1 0
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Abdominal pain (10000081) 25 18
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Abdominal pain lower (10000084) 5 5
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Abdominal pain upper (10000087) 4 5
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Constipation (10010774) 2 2
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Diarrhoea (10012735) 4 1
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Dyspepsia (10013946) 1 2
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Irritable bowel syndrome (10023003) 4 1
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Nausea (10028813) 2 1
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Vomiting (10047700) 3 2
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Fatigue (10016256) 1 1
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Non-cardiac chest pain 1 0
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Back pain (10003988) 9 5
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Musculoskeletal pain (10028391) 0 1
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Pain in extremity (10033425) 2 1
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Dizziness (10013573) 5 2
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Headache (10019211) 11 4
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Neuropathy peripheral (10029331) 0 1
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Paraesthesia (10033775) 1 1
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Tension headache (10043269) 3 0
Total 95 57
173
Page 513 of 637
14.2. Serious harms judged as ‘definitely associated’* with postural orthostatic tachycardia syndrome (POTS):
We asked a physician with clinical expertise in POTS to assess the reported MedDRA preferred terms as 'definitely,' 'probably,' 'probably not' or ‘definitely
not’ associated with the syndromes. We sent an Excel sheet to the physician with all the reported MedDRA terms. The physician was blinded, as the Excel
sheet contained no outcome data. When the physician had assessed all the MedDRA terms, we synthesized the data for those MedDRA terms that the
physician judged ‘definitely’ associated with POTS and compared it to the reported serious harms.
174
Page 514 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the serious harms that were judged ‘definitely associated’ with postural
orthostatic tachycardia syndrome (POTS):
Physician judgment MedDRA system organ class MedDRA preferred term HPV vaccine Comparator
‘Definitely associated with POTS’ Cardiac disorders (10007541) Sinus tachycardia (10040752) 2 1
‘Definitely associated with POTS’ Cardiac disorders (10007541) Supraventricular tachycardia 3 1
‘Definitely associated with POTS’ Cardiac disorders (10007541) Tachycardia (10043071) 1 0
‘Definitely associated with POTS’ Ear and labyrinth disorders (10013993) Vertigo (10047340) 3 2
‘Definitely associated with POTS’ Ear and labyrinth disorders (10013993) Vertigo positional 0 1
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Vomiting (10047700) 1 0
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Constipation (10010774) 2 2
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Diarrhoea (10012735) 4 1
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Dyspepsia (10013946) 1 2
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Irritable bowel syndrome 4 1
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Nausea (10028813) 2 1
‘Definitely associated with POTS’ General disorders and administration site conditions (10018065) Fatigue (10016256) 1 1
‘Definitely associated with POTS’ General disorders and administration site conditions (10018065) Non-cardiac chest pain 1 0
‘Definitely associated with POTS’ Nervous system disorders (10029205) Dizziness (10013573) 5 2
‘Definitely associated with POTS’ Nervous system disorders (10029205) Headache (10019211) 11 4
‘Definitely associated with POTS’ Nervous system disorders (10029205) Syncope (10042772) 4 3
‘Definitely associated with POTS’ Nervous system disorders (10029205) Tension headache (10043269) 3 0
‘Definitely associated with POTS’ Respiratory, thoracic and mediastinal disorders (10038738) Dyspnoea (10013968) 2 1
‘Definitely associated with POTS’ Respiratory, thoracic and mediastinal disorders (10038738) Hyperventilation (10020910) 2 0
‘Definitely associated with POTS’ Vascular disorders (10047065) Hypotension (10021097) 1 1
Total 56 26
175
Page 515 of 637
14.3. New onset diseases ('medically significant conditions' and 'new medical history'*) judged as ‘definitely
associated’** with chronic regional pain syndrome (CRPS): intention to treat analysis
[0.96, 1.03]. **We asked a physician with clinical expertise in CRPS to assess the reported MedDRA preferred terms as 'definitely,' 'probably,' 'probably
not' or ‘definitely not’ associated with the syndromes. We sent an Excel sheet to the physician with all the reported MedDRA terms. The physician was
blinded, as the Excel sheet contained no outcome data. When the physician had assessed all the MedDRA terms, we synthesized the data for those
MedDRA terms that the physician judged ‘definitely’ associated with CRPS and compared it to the reported new onset diseases.
176
Page 516 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the new onset diseases judged as ‘definitely associated’ with chronic regional pain
syndrome (CRPS)
HPV
Physician judgement MedDRA system organ class MedDRA preferred term Comparator
vaccine
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Arrhythmia (10003119) 19 12
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Atrial tachycardia 1 1
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Cardiovascular disorder 1 1
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Cardiovascular insufficiency 1 0
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Extra systoles 2 0
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Palpitations (10033557) 23 16
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Sinus arrhythmia 0 1
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Sinus tachycardia (10040752) 5 5
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Supraventricular tachycardia (10042604) 7 2
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Tachycardia (10043071) 16 17
‘Definitely associated with CRPS’ Cardiac disorders (10007541) Tachycardia paroxysmal 1 1
‘Definitely associated with CRPS’ Ear and labyrinth disorders (10013993) Tinnitus (10043882) 13 11
‘Definitely associated with CRPS’ Ear and labyrinth disorders (10013993) Vertigo (10047340) 75 66
‘Definitely associated with CRPS’ Ear and labyrinth disorders (10013993) Vertigo positional 6 7
‘Definitely associated with CRPS’ Eye disorders (10015919) Dry eye (10013774) 8 10
‘Definitely associated with CRPS’ Eye disorders (10015919) Eye disorder 1 1
‘Definitely associated with CRPS’ Eye disorders (10015919) Photophobia 1 0
‘Definitely associated with CRPS’ Eye disorders (10015919) Vision blurred (10047513) 4 7
‘Definitely associated with CRPS’ Eye disorders (10015919) Visual acuity reduced (10047531) 4 4
‘Definitely associated with CRPS’ Eye disorders (10015919) Visual disturbance 5 1
‘Definitely associated with CRPS’ Eye disorders (10015919) Visual impairment (10047571) 1 1
‘Definitely associated with CRPS’ Eye disorders (10015919) Vitreous detachment 0 1
‘Definitely associated with CRPS’ Eye disorders (10015919) Xerophthalmia (10048221) 2 0
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Abdominal discomfort (10000059) 20 23
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Abdominal pain (10000081) 457 400
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Abdominal pain lower (10000084) 183 187
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Abdominal pain upper (10000087) 143 133
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Abdominal symptom 2 0
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Abdominal tenderness 4 6
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Bowel movement irregularity 1 1
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Change of bowel habit (10008399) 5 0
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Constipation (10010774) 160 130
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Diarrhoea (10012735) 320 273
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Dry mouth 1 2
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Dyspepsia (10013946) 154 141
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Epigastric discomfort (10053155) 3 4
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Food poisoning (10016952) 60 48
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Frequent bowel movements 1 0
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Gastric Disorder 3 1
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Gastrointestinal disorder (10017944) 6 12
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Gastrointestinal hypomotility 0 1
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Gastrointestinal pain 4 5
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Intestinal functional disorder (10061247) 1 2
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Irritable bowel syndrome (10023003) 117 114
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Nausea (10028813) 168 188
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Stomach discomfort 2 11
‘Definitely associated with CRPS’ Gastrointestinal disorders (10017947) Vomiting (10047700) 112 117
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Asthenia (10003549) 21 16
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Axillary pain (10048750) 4 2
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Discomfort 3 0
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Facial pain (10016059) 2 2
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Fatigue (10016256) 73 72
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Feeling cold 3 2
177
Page 517 of 637
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Feeling of body temperature change 3 6
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) General Symptom 2 1
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Local swelling (10024770) 5 5
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Malaise (10025482) 20 14
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Mucosal dryness 1 0
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Non-cardiac chest pain (10062501) 19 9
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Pain (10033371) 17 28
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Suprapubic pain 3 2
‘Definitely associated with CRPS’ General disorders and administration site conditions (10018065) Tenderness 3 1
‘Definitely associated with CRPS’ Investigations (10022891) Abdomen Scan 2 0
‘Definitely associated with CRPS’ Investigations (10022891) Abdomen scan normal 0 1
‘Definitely associated with CRPS’ Investigations (10022891) Abdominal X-Ray 3 4
‘Definitely associated with CRPS’ Investigations (10022891) Blood glucose 1 2
‘Definitely associated with CRPS’ Investigations (10022891) Blood glucose normal 1 1
‘Definitely associated with CRPS’ Investigations (10022891) Blood Test 13 13
‘Definitely associated with CRPS’ Investigations (10022891) Diagnostic procedure 1 1
‘Definitely associated with CRPS’ Investigations (10022891) Heart rate irregular (10019304) 4 0
‘Definitely associated with CRPS’ Investigations (10022891) Investigation 2 0
‘Definitely associated with CRPS’ Investigations (10022891) Laboratory Test 0 3
‘Definitely associated with CRPS’ Investigations (10022891) Lumbar puncture 3 2
‘Definitely associated with CRPS’ Investigations (10022891) Medical observation 1 0
‘Definitely associated with CRPS’ Investigations (10022891) Physical examination 1 1
‘Definitely associated with CRPS’ Investigations (10022891) Tilt table test 1 0
‘Definitely associated with CRPS’ Investigations (10022891) Ultrasound abdomen 41 44
‘Definitely associated with CRPS’ Investigations (10022891) Ultrasound abdomen normal 7 8
‘Definitely associated with CRPS’ Investigations (10022891) Ultrasound scan 475 519
‘Definitely associated with CRPS’ Investigations (10022891) Ultrasound scan normal 1 4
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Arthralgia (10003239) 190 193
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Back pain (10003988) 462 410
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Fibromyalgia (10048439) 13 12
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Limb discomfort (10061224) 2 1
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Muscle atrophy 0 1
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Muscle contracture (10062575) 21 21
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Muscle cramp 1 0
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Muscle disorder 1 5
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Muscle spasms (10028334) 51 38
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Muscle twitching (10028347) 3 3
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Muscle weakness 2 0
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Muscular weakness 2 2
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Musculoskeletal discomfort (10053156) 3 3
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Musculoskeletal pain (10028391) 37 40
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Musculoskeletal stiffness (10052904) 16 17
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Myalgia (10028411) 79 78
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Neck pain (10028836) 52 52
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Osteopenia (10049088) 2 11
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Osteoporosis (10031282) 3 6
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Pain in extremity (10033425) 95 90
‘Definitely associated with CRPS’ Musculoskeletal and connective tissue disorders (10028395) Pain in jaw (10033433) 8 9
‘Definitely associated with CRPS’ Neoplasm’s benign, malignant and unspecified (10029104) Nervous system neoplasm benign 1 0
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Autonomous nervous system imbalance 3 1
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Balance disorder (10049848) 3 1
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Burning sensation (10006784) 3 1
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Disturbance in attention (10013496) 1 1
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Dizziness (10013573) 139 131
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Dizziness postural 0 2
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Dysesthesia 0 2
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Headache (10019211) 855 756
178
Page 518 of 637
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Hyperaesthesia 3 1
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Lethargy (10024264) 7 8
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Nervous system disorder 1 1
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Neuralgia (10029223) 6 8
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Neurological symptom 1 0
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Neuropathy peripheral (10029331) 2 7
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Paraesthesia (10033775) 15 11
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Restless legs syndrome (10058920) 3 11
‘Definitely associated with CRPS’ Nervous system disorders (10029205) Tension headache (10043269) 80 68
‘Definitely associated with CRPS’ Renal and urinary disorders (10038359) Cystitis interstitial (10011796) 5 3
‘Definitely associated with CRPS’ Renal and urinary disorders (10038359) Cystitis non-infective 1 4
‘Definitely associated with CRPS’ Renal and urinary disorders (10038359) Cystitis-like symptom 39 44
‘Definitely associated with CRPS’ Renal and urinary disorders (10038359) Micturition disorder (10027561) 2 0
‘Definitely associated with CRPS’ Renal and urinary disorders (10038359) Urogenital disorder 4 1
‘Definitely associated with CRPS’ Skin and subcutaneous tissue disorders (10040785) Nail growth abnormal 1 0
‘Definitely associated with CRPS’ Skin and subcutaneous tissue disorders (10040785) Pain of skin (10033474) 0 1
‘Definitely associated with CRPS’ Skin and subcutaneous tissue disorders (10040785) Skin burning sensation 0 1
‘Definitely associated with CRPS’ Vascular disorders (10047065) Peripheral vascular disorder 0 1
‘Definitely associated with CRPS’ Vascular disorders (10047065) Raynaud's phenomenon (10037912) 2 6
Total 5,079 4,790
179
Page 519 of 637
14.4. New onset diseases ('medically significant conditions' and 'new medical history'*) judged as ‘definitely
associated’** with postural orthostatic tachycardia syndrome (POTS): intention to treat analysis
[0.98, 1.07]. **We asked a physician with clinical expertise in POTS to assess the reported MedDRA preferred terms as 'definitely,' 'probably,' 'probably
not' or ‘definitely not’ associated with the syndromes. We sent an Excel sheet to the physician with all the reported MedDRA terms. The physician was
blinded, as the Excel sheet contained no outcome data. When the physician had assessed all the MedDRA terms, we synthesized the data for those
MedDRA terms that the physician judged ‘definitely’ associated with POTS and compared it to the reported new onset diseases.
180
Page 520 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the new onset diseases that were judged ‘definitely associated’ with postural
orthostatic tachycardia syndrome (POTS):
HPV Compara
Physician judgment MedDRA system organ class MedDRA preferred term
vaccine tor
‘Definitely associated with POTS’ Cardiac disorders (10007541) Arrhythmia (10003119) 9 12
‘Definitely associated with POTS’ Cardiac disorders (10007541) Atrial tachycardia 1 1
‘Definitely associated with POTS’ Cardiac disorders (10007541) Cardiovascular disorder 1 1
‘Definitely associated with POTS’ Cardiac disorders (10007541) Cardiovascular insufficiency 1 0
‘Definitely associated with POTS’ Cardiac disorders (10007541) Extra systoles 2 0
‘Definitely associated with POTS’ Cardiac disorders (10007541) Mitral valve prolapse (10027730) 9 10
‘Definitely associated with POTS’ Cardiac disorders (10007541) Palpitations (10033557) 23 16
‘Definitely associated with POTS’ Cardiac disorders (10007541) Sinus arrhythmia 0 1
‘Definitely associated with POTS’ Cardiac disorders (10007541) Sinus tachycardia (10040752) 5 5
‘Definitely associated with POTS’ Cardiac disorders (10007541) Supraventricular tachycardia (10042604) 7 2
‘Definitely associated with POTS’ Cardiac disorders (10007541) Tachycardia (10043071) 16 17
‘Definitely associated with POTS’ Cardiac disorders (10007541) Tachycardia paroxysmal 1 1
‘Definitely associated with POTS’ Ear and labyrinth disorders (10013993) Tinnitus (10043882) 13 11
‘Definitely associated with POTS’ Ear and labyrinth disorders (10013993) Vertigo (10047340) 75 66
‘Definitely associated with POTS’ Ear and labyrinth disorders (10013993) Vertigo positional 6 7
‘Definitely associated with POTS’ Eye disorders (10015919) Dry eye (10013774) 8 10
‘Definitely associated with POTS’ Eye disorders (10015919) Photophobia 1 0
‘Definitely associated with POTS’ Eye disorders (10015919) Vision blurred (10047513) 4 7
‘Definitely associated with POTS’ Eye disorders (10015919) Visual acuity reduced (10047531) 4 4
‘Definitely associated with POTS’ Eye disorders (10015919) Visual disturbance 5 1
‘Definitely associated with POTS’ Eye disorders (10015919) Visual impairment (10047571) 1 1
‘Definitely associated with POTS’ Eye disorders (10015919) Vitreous detachment 0 1
‘Definitely associated with POTS’ Eye disorders (10015919) Xerophthalmia (10048221) 2 0
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Abdominal discomfort (10000059) 20 23
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Abdominal pain (10000081) 457 400
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Abdominal pain lower (10000084) 183 187
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Abdominal pain upper (10000087) 143 133
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Abdominal symptom 2 0
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Abdominal tenderness 4 6
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Bowel movement irregularity 1 1
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Change of bowel habit (10008399) 5 0
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Constipation (10010774) 160 130
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Diarrhoea (10012735) 320 273
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Dry Mouth 1 2
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Dyspepsia (10013946) 154 141
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Epigastric discomfort (10053155) 3 4
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Food poisoning (10016952) 60 48
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Frequent bowel movements 1 0
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Gastric Disorder 3 1
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Gastrointestinal disorder (10017944) 6 12
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Gastrointestinal hypomotility 4 1
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Gastrointestinal pain 4 5
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Intestinal functional disorder (10061247) 1 2
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Irritable bowel syndrome (10023003) 117 114
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Nausea (10028813) 168 188
‘Definitely associated with POTS’ Gastrointestinal disorders (10017947) Stomach discomfort 2 11
‘Definitely associated with POTS’ General disorders and administration site conditions (10018065) Asthenia (10003549) 21 16
‘Definitely associated with POTS’ General disorders and administration site conditions (10018065) Chest discomfort (10008469) 6 3
‘Definitely associated with POTS’ General disorders and administration site conditions (10018065) Chest pain (10008479) 35 23
‘Definitely associated with POTS’ General disorders and administration site conditions (10018065) Discomfort 3 0
‘Definitely associated with POTS’ General disorders and administration site conditions (10018065) Fatigue (10016256) 73 72
‘Definitely associated with POTS’ General disorders and administration site conditions (10018065) General Symptom 2 1
181
Page 521 of 637
‘Definitely associated with POTS’ General disorders and administration site conditions (10018065) Malaise (10025482) 20 14
‘Definitely associated with POTS’ General disorders and administration site conditions (10018065) Mucosal dryness 1 0
‘Definitely associated with POTS’ General disorders and administration site conditions (10018065) Non-cardiac chest pain (10062501) 19 9
‘Definitely associated with POTS’ Investigations (10022891) Cardiac imaging Procedure 1 0
‘Definitely associated with POTS’ Investigations (10022891) Cardiac stress test 0 1
‘Definitely associated with POTS’ Investigations (10022891) Chest X-ray 19 16
‘Definitely associated with POTS’ Investigations (10022891) Chest X-ray normal 2 5
‘Definitely associated with POTS’ Investigations (10022891) Diagnostic procedure 1 1
‘Definitely associated with POTS’ Investigations (10022891) Heart rate increased 1 1
‘Definitely associated with POTS’ Investigations (10022891) Heart rate irregular (10019304) 4 0
‘Definitely associated with POTS’ Investigations (10022891) Investigation 2 0
‘Definitely associated with POTS’ Investigations (10022891) Laboratory test 0 3
‘Definitely associated with POTS’ Investigations (10022891) Medical observation 1 0
‘Definitely associated with POTS’ Investigations (10022891) Physical examination 1 1
‘Definitely associated with POTS’ Investigations (10022891) Tilt table test 1 0
‘Definitely associated with POTS’ Musculoskeletal and connective tissue disorders (10028395) Musculoskeletal chest pain (10050819) 25 26
‘Definitely associated with POTS’ Neoplasms benign, malignant and unspecified (10029104) Nervous system neoplasm benign 1 0
‘Definitely associated with POTS’ Nervous system disorders (10029205) Autonomic nervous system imbalance 3 1
‘Definitely associated with POTS’ Nervous system disorders (10029205) Balance disorder (10049848) 3 1
‘Definitely associated with POTS’ Nervous system disorders (10029205) Circadian rhythm disorder 1 0
‘Definitely associated with POTS’ Nervous system disorders (10029205) Disturbance in attention (10013496) 1 1
‘Definitely associated with POTS’ Nervous system disorders (10029205) Dizziness (10013573) 139 131
‘Definitely associated with POTS’ Nervous system disorders (10029205) Dizziness postural 0 2
‘Definitely associated with POTS’ Nervous system disorders (10029205) Headache (10019211) 855 756
‘Definitely associated with POTS’ Nervous system disorders (10029205) Lethargy (10024264) 7 8
‘Definitely associated with POTS’ Nervous system disorders (10029205) Loss of consciousness 8 5
‘Definitely associated with POTS’ Nervous system disorders (10029205) Nervous system disorder 1 1
‘Definitely associated with POTS’ Nervous system disorders (10029205) Neurological symptom 1 0
‘Definitely associated with POTS’ Nervous system disorders (10029205) Presyncope (10036653) 4 6
‘Definitely associated with POTS’ Nervous system disorders (10029205) Restless legs syndrome (10058920) 9 11
‘Definitely associated with POTS’ Nervous system disorders (10029205) Syncope (10042772) 62 60
‘Definitely associated with POTS’ Nervous system disorders (10029205) Syncope vasovagal 12 7
‘Definitely associated with POTS’ Nervous system disorders (10029205) Tension headache (10043269) 80 68
‘Definitely associated with POTS’ Renal and urinary disorders (10038359) Cystitis interstitial (10011796) 5 3
‘Definitely associated with POTS’ Renal and urinary disorders (10038359) Cystitis non-infective 1 4
‘Definitely associated with POTS’ Renal and urinary disorders (10038359) Cystitis-like symptom 39 44
‘Definitely associated with POTS’ Renal and urinary disorders (10038359) Micturition disorder (10027561) 2 0
‘Definitely associated with POTS’ Renal and urinary disorders (10038359) Urogenital disorder 4 1
‘Definitely associated with POTS’ Respiratory, thoracic and mediastinal disorders (10038738) Dyspnoea (10013968) 43 24
‘Definitely associated with POTS’ Respiratory, thoracic and mediastinal disorders (10038738) Hyperventilation (10020910) 5 11
‘Definitely associated with POTS’ Vascular disorders (10047065) Hypotension (10021097) 20 29
‘Definitely associated with POTS’ Vascular disorders (10047065) Orthostatic hypotension 4 6
‘Definitely associated with POTS’ Vascular disorders (10047065) Peripheral vascular disorder 0 1
‘Definitely associated with POTS’ Vascular disorders (10047065) Raynaud's phenomenon (10037912) 2 6
Total 3,675 3,352
182
Page 522 of 637
14.5. General harms (‘solicited,’ ‘unsolicited’ and ‘systemic adverse events’) judged as ‘definitely associated’
with chronic regional pain syndrome (CRPS): intention to treat analysis
This meta-analysis was inappropriate, as some of the numerators exceeded the denominators making the
result nonsensical.
183
Page 523 of 637
14.6. General harms (‘solicited,’ ‘unsolicited’ and ‘systemic adverse events’) judged as ‘definitely associated’
with postural orthostatic tachycardia syndrome (POTS): intention to treat analysis
result nonsensical.
184
Page 524 of 637
14.7. Serious harms part of VigiBase’s largest HPV vaccine-associated harm clusters* – ‘expected systemic
reactions’: intention to treat analysis
VigiBase’s largest HPV vaccine-associated harm cluster consists of the harms ‘headache, nausea, pyrexia, dizziness and vomiting’ (1).
185
Page 525 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the serious harms that were part of VigiBase’s largest HPV vaccine-associated
harm clusters ‘expected systemic reactions’:
VigiBase term MedDRA system organ class MedDRA preferred term HPV vaccine Comparator
Dizziness Nervous system disorders (10029205) Dizziness (10013573) 5 2
Headache Nervous system disorders (10029205) Headache (10019211) 11 4
Nausea Gastrointestinal disorders (10017947) Nausea (10028813) 2 1
Pyrexia General disorders and administration site conditions (10018065) Pyrexia (10037660) 4 2
Vomiting Gastrointestinal disorders (10017947) Vomiting (10047700) 3 2
Total 25 11
186
Page 526 of 637
14.8. Serious harms part of VigiBase’s 2nd largest HPV vaccine-associated harm cluster* -
‘allergic/hypersensitivity reactions’: intention to treat analysis
VigiBase’s 2nd largest HPV vaccine-associated harm cluster consists of ‘pruritis, urticaria, rash and erythema’ (1).
187
Page 527 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the serious harms that were part of VigiBase’s 2nd largest HPV vaccine-associated
harm cluster ‘allergic/hypersensitivity reactions’:
VigiBase term MedDRA system organ class MedDRA preferred term HPV vaccine Comparator
Erythema Skin and subcutaneous tissue disorders (10040785) Erythema 0 0
Pruritus Skin and subcutaneous tissue disorders (10040785) Pruritus 0 0
Rash Skin and subcutaneous tissue disorders (10040785) Rash 0 0
Urticaria Skin and subcutaneous tissue disorders (10040785) Urticaria (10046735) 2 2
Total 2 2
188
Page 528 of 637
14.9. Serious harms part of VigiBase’s 3rd largest HPV vaccine-associated harm cluster* - ‘vasovagal reactions’:
VigiBase’s 3rd largest HPV vaccine-associated harm cluster consists of ‘syncope, dizziness, loss of consciousness, pallor and seizure’ (1).
189
Page 529 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the serious harms that were part of VigiBase’s 3rd largest HPV vaccine-associated
harm cluster ‘vasovagal reactions’:
VigiBase term MedDRA system organ class MedDRA term HPV vaccine Comparator
Dizziness Nervous system disorders (10029205) Dizziness (10013573) 5 2
Loss of consciousness Nervous system disorders (10029205) Loss of consciousness 0 0
Pallor Vascular disorders (10047065) Pallor 0 0
Seizure Nervous system disorders (10029205) Seizure 0 0
Syncope Nervous system disorders (10029205) Syncope (10042772) 4 3
Total 9 5
190
Page 530 of 637
14.10. New onset diseases ('medically significant conditions' and 'new medical history'*) part of VigiBase’s
largest HPV vaccine-associated harm clusters** - ‘expected systemic reactions’: intention to treat analysis
1.00 [0.90, 1.11]. **VigiBase’s largest HPV vaccine-associated harm cluster consists of ‘headache, nausea, pyrexia, dizziness and vomiting’ (1).
191
Page 531 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the new onset diseases that were part of VigiBase’s largest HPV vaccine-associated
harm clusters ‘expected systemic reactions’:
VigiBase MedDRA preferred

New onset disease category MedDRA system organ class HPV vaccine Comparator
term term
‘Medically significant conditions’ Nervous system disorders (10029205) Headache (10019211) 89 66
Headache
‘New medical history’ Nervous system disorders (10029205) Headache (10019211) 766 690
‘Medically significant conditions’ Gastrointestinal disorders (10017947) Nausea (10028813) 21 16
Nausea
‘New medical history’ Gastrointestinal disorders (10017947) Nausea (10028813) 147 172
‘Medically significant conditions’ General disorders and administration site conditions (10018065) Pyrexia (10037660) 29 25
Pyrexia
‘New medical history’ General disorders and administration site conditions (10018065) Pyrexia (10037660) 162 141
‘Medically significant conditions’ Nervous system disorders (10029205) Dizziness (10013573) 30 29
Dizziness
‘New medical history’ Nervous system disorders (10029205) Dizziness (10013573) 109 102
‘Medically significant conditions’ Gastrointestinal disorders (10017947) Vomiting (10047700) 15 21
Vomiting
‘New medical history’ Gastrointestinal disorders (10017947) Vomiting (10047700) 97 96
Total 1,465 1,358
192
Page 532 of 637
14.11. New onset diseases ('medically significant conditions' and 'new medical history'*) part of VigiBase’s 2nd
largest HPV vaccine-associated harm cluster** - ‘allergic/hypersensitivity reactions’: intention to treat
analysis
1.19 [0.87, 1.64]. **VigiBase’s 2nd largest HPV vaccine-associated harm cluster consists of ‘pruritis, urticaria, rash and erythema’ (1).
193
Page 533 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the new onset diseases that were part of VigiBase’s 2nd largest HPV vaccine-
associated harm cluster ‘allergic/hypersensitivity reactions’:
VigiBase term New onset disease category MedDRA system organ class MedDRA preferred term HPV vaccine Comparator
‘Medically significant conditions’ Skin and subcutaneous tissue disorders (10040785) Pruritus (10037087) 8 5
Pruritus
‘New medical history’ Skin and subcutaneous tissue disorders (10040785) Pruritus (10037087) 46 37
‘Medically significant conditions’ Skin and subcutaneous tissue disorders (10040785) Urticaria (10046735) 37 28
Urticaria
‘New medical history’ Skin and subcutaneous tissue disorders (10040785) Urticaria (10046735) 54 57
‘Medically significant conditions’ Skin and subcutaneous tissue disorders (10040785) Rash (10037844) 38 33
Rash
‘New medical history’ Skin and subcutaneous tissue disorders (10040785) Rash (10037844) 86 108
‘Medically significant conditions’ Skin and subcutaneous tissue disorders (10040785) Erythema (10015150) 0 3
Erythema
‘New medical history’ Skin and subcutaneous tissue disorders (10040785) Erythema (10015150) 15 8
Total 284 279
194
Page 534 of 637
14.12. New onset diseases ('medically significant conditions' and 'new medical history'*) part of VigiBase’s 3rd
largest HPV vaccine-associated harm cluster** - ‘vasovagal reactions’: intention to treat analysis
1.00 [0.74, 1.34]. **VigiBase’s 3rd largest HPV vaccine-associated harm cluster consists of ‘syncope, dizziness, loss of consciousness, pallor and seizure’ (1).
195
Page 535 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the new onset diseases that were part of VigiBase’s 3rd largest HPV vaccine-
associated harm cluster ‘vasovagal reactions’:
VigiBase term New onset disease category MedDRA system organ class MedDRA preferred term HPV vaccine Comparator
‘Medically significant conditions’ Nervous system disorders (10029205) Syncope (10042772) 19 14
Syncope
‘New medical history’ Nervous system disorders (10029205) Syncope (10042772) 43 46
‘Medically significant conditions’ Nervous system disorders (10029205) Dizziness (10013573) 30 29
Dizziness
‘New medical history’ Nervous system disorders (10029205) Dizziness (10013573) 109 102
Loss of ‘Medically significant conditions’ Nervous system disorders (10029205) Loss of consciousness 1 2
consciousness ‘New medical history’ Nervous system disorders (10029205) Loss of consciousness 7 3
‘Medically significant conditions’ Vascular disorders (10047065) Pallor 0 0
Pallor
‘New medical history’ Vascular disorders (10047065) Pallor 2 3
‘Medically significant conditions’ Nervous system disorders (10029205) Convulsion (10010904) 9 6
‘New medical history’ Nervous system disorders (10029205) Convulsion (10010904) 6 6
‘Medically significant conditions’ Nervous system disorders (10029205) Generalized seizure 1 0
Seizure
‘New medical history’ Nervous system disorders (10029205) Generalized seizure 0 0
‘Medically significant conditions’ Nervous system disorders (10029205) Grand mal convulsion (10018659) 4 0
‘New medical history’ Nervous system disorders (10029205) Grand mal convulsion (10018659) 1 1
Total 232 212
196
Page 536 of 637
14.13. General harms (‘solicited,’ ‘unsolicited’ and ‘systemic adverse events’) part of VigiBase’s largest HPV
vaccine-associated harm clusters - ‘expected systemic reactions’: intention to treat analysis
result nonsensical, but there were more ‘expected systemic reactions’ in the HPV vaccine group:
VigiBase term General harms category MedDRA system organ class MedDRA preferred term HPV vaccine Comparator
‘Solicited’ Nervous system disorders (10029205) Headache (10019211) 7,621 7,125
Headache ‘Unsolicited’ Nervous system disorders (10029205) Headache (10019211) 86 53
‘Systemic adverse events’ Nervous system disorders (10029205) Headache (10019211) 899 812
‘Solicited’ Gastrointestinal disorders (10017947) Nausea (10028813) 0 0
Nausea ‘Unsolicited’ Gastrointestinal disorders (10017947) Nausea (10028813) 5 3
‘Systemic adverse events’ Gastrointestinal disorders (10017947) Nausea (10028813) 208 145
‘Solicited’ General disorders (10018065) Pyrexia/fever (10037660) 1,531 1,378
Pyrexia ‘Unsolicited’ General disorders (10018065) Pyrexia/fever (10037660) 40 35
‘Systemic adverse events’ General disorders (10018065) Pyrexia/fever (10037660) 304 234
‘Solicited’ Nervous system disorders (10029205) Dizziness (10013573) 0 0
Dizziness ‘Unsolicited’ Nervous system disorders (10029205) Dizziness (10013573) 37 19
‘Systemic adverse events’ Nervous system disorders (10029205) Dizziness (10013573) 127 95
‘Solicited’ Gastrointestinal disorders (10017947) Vomiting (10047700) 0 0
Vomiting ‘Unsolicited’ Gastrointestinal disorders (10017947) Vomiting (10047700) 2 9
‘Systemic adverse events’ Gastrointestinal disorders (10017947) Vomiting (10047700) 66 40
Total 10,926 9,948
197
Page 537 of 637
14.14. General harms (‘solicited,’ ‘unsolicited’ and ‘systemic adverse events’*) part of VigiBase’s 2nd largest HPV
vaccine-associated harm cluster** - ‘allergic/hypersensitivity reactions’: intention to treat analysis
adverse events’ (Merck Sharp & Dohme): 0.75 [0.43, 1.30]. **VigiBase’s 2nd largest HPV vaccine-associated harm cluster consists of ‘pruritis, urticaria, rash
and erythema’ (1).
198
Page 538 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the general harms that were part of VigiBase’s 2nd largest HPV vaccine-associated
harm cluster ‘allergic/hypersensitivity reactions’:
‘Solicited’ Skin and subcutaneous tissue disorders (10040785) Pruritis 0 0
Pruritis ‘Unsolicited’ Skin and subcutaneous tissue disorders (10040785) Pruritis 0 0
‘Systemic adverse events’ Skin and subcutaneous tissue disorders (10040785) Pruritis 0 0
‘Solicited’ Skin and subcutaneous tissue disorders (10040785) Urticaria 888 712
Urticaria ‘Unsolicited’ Skin and subcutaneous tissue disorders (10040785) Urticaria 9 5
‘Systemic adverse events’ Skin and subcutaneous tissue disorders (10040785) Urticaria 8 7
‘Solicited’ Skin and subcutaneous tissue disorders (10040785) Rash (10037844) 980 713
Rash ‘Unsolicited’ Skin and subcutaneous tissue disorders (10040785) Rash (10037844) 9 7
‘Systemic adverse events’ Skin and subcutaneous tissue disorders (10040785) Rash (10037844) 16 22
‘Solicited’ Skin and subcutaneous tissue disorders (10040785) Erythema (10015150) 0 0
Erythema ‘Unsolicited’ Skin and subcutaneous tissue disorders (10040785) Erythema (10015150) 0 2
‘Systemic adverse events’ Skin and subcutaneous tissue disorders (10040785) Erythema (10015150) 2 1
Total 1,912 1,469
199
Page 539 of 637
14.15. General harms (‘solicited,’ ‘unsolicited’ and ‘systemic adverse events’*) part of VigiBase’s 3rd largest HPV
vaccine-associated harm cluster** - ‘vasovagal reactions’: intention to treat analysis
adverse events’ (Merck Sharp & Dohme): 1.15 [0.75, 1.74]. **VigiBase’s 3rd largest HPV vaccine-associated harm cluster consists of ‘syncope, dizziness,
loss of consciousness, pallor and seizure’ (1).
200
Page 540 of 637
- Reported MedDRA terms and number of harms per MedDRA term for the general harms that were part of VigiBase’s 3rd largest HPV vaccine-associated
harm cluster ‘vasovagal reactions’:
‘Solicited’ Nervous system disorders (10029205) Syncope (10042772) 0 0
Syncope ‘Unsolicited’ Nervous system disorders (10029205) Syncope (10042772) 3 4
‘Systemic adverse events’ Nervous system disorders (10029205) Syncope (10042772) 4 3
‘Solicited’ Nervous system disorders (10029205) Dizziness (10013573) 0 0
Dizziness ‘Unsolicited’ Nervous system disorders (10029205) Dizziness (10013573) 37 19
‘Systemic adverse events’ Nervous system disorders (10029205) Dizziness (10013573) 127 95
‘Solicited’ Nervous system disorders (10029205) Pyrexia/fever (10037660) 0 0
Loss of consciousness ‘Unsolicited’ Nervous system disorders (10029205) Pyrexia/fever (10037660) 0 0
‘Systemic adverse events’ Nervous system disorders (10029205) Pyrexia/fever (10037660) 0 0
‘Solicited’ Nervous system disorders (10029205) Seizure 0 0
Seizure ‘Unsolicited’ Nervous system disorders (10029205) Seizure 0 0
‘Systemic adverse events’ Nervous system disorders (10029205) Convulsion (10010904) 1 0
‘Solicited’ Vascular disorders (10047065) Pallor 0 0
Pallor ‘Unsolicited’ Vascular disorders (10047065) Pallor 0 0
‘Systemic adverse events’ Vascular disorders (10047065) Pallor 1 2
Total 173 123
201
Page 541 of 637
15. Definitions of harms categories
• Serious harms
o GlaxoSmithKline: “any untoward medical occurrence that: a. resulted in death, b. was life-
threatening, NOTE: The term 'life-threatening' in the definition of 'serious' refers to an event in
which the subject was at risk of death at the time of the event. It did not refer to an event,
which hypothetically might have caused death, if it were more severe. c. required
hospitalization or prolongation of existing hospitalization, NOTE: In general, hospitalization
signified that the subject had been detained (usually involving at least an overnight stay) at the
hospital or emergency ward for observation and/or treatment that would not have been
appropriate in the physician's office or out-patient setting. Complications that occurred during
hospitalization were AEs [adverse events]. If a complication prolonged hospitalization or fulfilled
any other serious criteria, the event was serious. When in doubt as to whether "hospitalization"
occurred or was necessary, the AE was to be considered serious. Hospitalization for elective
treatment of a pre-existing condition that did not worsen from baseline was not considered an
AE. d. resulted in disability/incapacity, NOTE: The term disability means a substantial disruption
of a person's ability to conduct normal life functions. This definition was not intended to include
experiences of relatively minor medical significance such as uncomplicated headache, nausea,
vomiting, diarrhoea, influenza, and accidental trauma (e.g. sprained ankle) which may interfere
or prevent everyday life functions but did not constitute a substantial disruption. e. was a
congenital anomaly/birth defect in the offspring of a study subject.”
o Merck Sharp & Dohme: “A serious adverse experience is any adverse experience occurring at
any dose that: Results in death; or that is life threatening (places the subject/patient, in the
view of the investigator, at immediate risk of death from the experience as it occurred. [Note:
This does not include an adverse experience that, had it occurred in a more severe form, might
have caused death.]); or that results in a persistent or significant disability/incapacity
(substantial disruption of one's ability to conduct normal life functions); or that results in or
prolongs an existing inpatient hospitalization (hospitalized is defined as an inpatient admission,
regardless of length of stay, even if the hospitalization is a precautionary measure for continued
observation.) (Note: Hospitalization [including hospitalization for an elective procedure] for a
pre-existing condition which has not worsened does not constitute a serious adverse
experience); or that is a congenital anomaly/birth defect (in offspring of subject taking the
product regardless of time to diagnosis); or ALSO: Other important medical events that may not
result in death, not be life threatening, or not require hospitalization may be considered a
serious adverse experience when, based upon appropriate medical judgment, the event may
jeopardize the subject/patient and may require medical or surgical intervention to prevent one
of the (t) outcomes listed above. In addition, Merck Sharp & Dohme & Co., Inc. requires the
collection of the following: cancer, or overdose (whether accidental or intentional).”
• New onset diseases
o ‘Medically significant conditions’ (GlaxoSmithKline): “Adverse events prompting emergency
room or physician visits that are not (1) related to common diseases or (2) routine visits for
physical examination or vaccination, or SAEs that are not related to common diseases. Serious
adverse events related to common diseases were reported but are not classified as medically
significant conditions for analysis purposes. Common diseases include: upper respiratory
infections, sinusitis, pharyngitis, gastroenteritis, urinary tract infections, cervicovaginal yeast
infections, menstrual cycle abnormalities and injury.”
o ‘New medical history’ (Merck Sharp & Dohme): Merck Sharp & Dohme did not provide a formal
definition for 'new medical history' but described 'new medical history' as “all new reported
diagnoses” in the clinical study report of trial V501-019.
• General harms
o ‘Solicited’ general adverse events (GlaxoSmithKline): “Adverse events to be recorded as
endpoints in the clinical study. The presence/occurrence/intensity of these events is actively
solicited from the subject or an observer during a specified post-vaccination follow-up period.”
o ‘Unsolicited’ general adverse event (GlaxoSmithKline): “Any AE [adverse event] reported in
addition to those solicited during the clinical study. Also, any "solicited" symptom with onset
202
Page 542 of 637
outside the specified period of follow-up for solicited symptoms was reported as an unsolicited
AE.”
o ‘Systemic adverse event’ (Merck Sharp & Dohme): “…any systemic clinical adverse event that
developed on the day of vaccination or during the 14 days after vaccination was recorded on
the VRC [vaccination report card] along with the date it started and the last date it was
present.”
203
Page 543 of 637
16. References
1. Chandler RE, Juhlin K, Fransson J, Caster O, Edwards IR, Norén GN. Current Safety Concerns with Human
Papillomavirus Vaccine: A Cluster Analysis of Reports in VigiBase(®). Drug Saf. 2017 Jan;40(1):81–90.
204
Page 544 of 637
Paper 5
Comparison of HPV vaccine study documents
Page 545 of 637

Jørgensen L, Gøtzsche PC and Jefferson T. Benefits and harms of the human papillomavirus (HPV) vaccines:
comparison of clinical study reports with trial registry entries and journal publications. Submitted for publication.
2018. Protocol: https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20180320.pdf.

(A,B,C)
development


November professor
2018
November MRCGP, FFPHM
2018
Page 546 of 637
2
Page 547 of 637
1
6 Benefits and harms of the human papillomavirus (HPV) vaccines: comparison of trial data from clinical
7 study reports with corresponding trial register entries and journal publications
10 Word count: 3,737
11
12
13 Dr Lars Jørgensen MD (lj@cochrane.dk), 1,2
14 Prof Peter C. Gøtzsche DrMedSci (pcg@cochrane.dk), 1
15 Dr Tom Jefferson MD (tj@cochrane.dk), 1
16
17 The protocol for our comparison is registered on PROSPERO as an addendum to our systematic review of
18 the benefits and harms of the HPV vaccines:
20
21 Our index of the HPV vaccine studies was published on January 2018:
22 https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-018-0675-z
23
24 A description of our difficulties obtaining the clinical study reports is published here:
25 https://www.bmj.com/content/362/bmj.k3694.full?ijkey=0ibTwph3m0aErxL&keytype=ref
26
27
28 1Nordic Cochrane Centre, +45 35457112, Rigshospitalet 7811, Tagensvej 21, 2100 Copenhagen, Denmark.
29 2Corresponding author: lj@cochrane.dk; ORCID 0000-0002-9737-0555; ResearcherID T-6254-2017.
Page 548 of 637

30 Abstract
31 Background: No study has looked at differences of pooled estimates—such as meta-analyses—of
32 corresponding study documents of the same intervention. In this study, we compared meta-analyses of
33 human papillomavirus (HPV) vaccine trial data from clinical study reports with trial data from
34 corresponding trial register entries and journal publications.
35 Methods: We obtained clinical study reports from the European Medicines Agency and GlaxoSmithKline;
36 corresponding trial register entries from ClinicalTrials.gov; and corresponding journal publications via the
37 Cochrane Collaboration's Central Register of Controlled Trials, Google Scholar and PubMed. Two
38 researchers extracted data. We compared reporting of trial design aspects and 20 prespecified benefits and
39 harms outcomes extracted from each study document type. Risk ratios were calculated with the random
40 effects inverse variance method.
41 Results: We included study documents from 22 randomized clinical trials and two follow-up studies with
42 95,670 healthy participants and non-HPV vaccine comparators (placebo, HPV vaccine adjuvants and
43 hepatitis vaccines). We obtained 24 clinical study reports, 24 corresponding trial register entries and 23
44 corresponding journal publications; the median number of pages was 1,351 (range 357 to 11,456), 32
45 (range 11 to 167) and 11 (range 7 to 83), respectively. All 24 (100%) clinical study reports, no (0%) trial
46 register entries and nine (39%) journal publications reported on all six major design-related biases defined
47 by the Cochrane Handbook version 2011. The clinical study reports reported more inclusion criteria (mean:
48 7.0 vs. 5.8 [trial register entries] and 4.0 [journal publications]) and exclusion criteria (mean: 17.8 vs. 11.7
49 and 5.0) but fewer primary outcomes (mean: 1.6 vs. 3.5 and 1.2) and secondary outcomes (mean: 8.8 vs.
50 13.0 and 3.2) than the trial register entries. Results were posted for 19 trial register entries (79%).
51 Compared to the clinical study reports, the trial register entries and journal publications contained 1% and
52 44% of the seven assessed benefit data points (6,879 vs. 230 and 3,015) and 38% and 31% of the 13
53 assessed harm data points (167,550 vs. 64,143 and 51,899). No meta-analysis estimate differed significantly
54 when we compared pooled risk ratio estimates of corresponding study document data as ratios of relative
55 risk.
2
Page 549 of 637
56 Conclusion: There were no significant differences in the meta-analysis estimates of the assessed outcomes
57 from corresponding study documents. The clinical study reports were the quantitively and qualitatively
58 superior study documents and should be used as primary data sources in systematic reviews.
59 Systematic review registration: CRD42017056093.
60
61 Key words
62 Human papillomavirus vaccine, randomized clinical trial, clinical study report, trial register entry, journal
63 publication, meta-analysis, systematic review, risk of bias.
64
65 Background
66 Since 1995, the pharmaceutical industry has written structured clinical study reports of randomized clinical
67 trials following international guidelines to document their products’ benefits and harms when applying for
68 marketing approval (1). Clinical study reports are usually confidential documents, but can be requested or
69 downloaded from the European Medicines Agency (EMA) (2), ClinicalStudyDataRequest.com (CSDR),
70 GlaxoSmithKline’s trial register website and, in the future, possibly also from the US Food and Drug
71 Administration (FDA) (3). Publicly available trial data mainly come from biomedical journal publications and
72 trial register entries such as those on ClinicalTrials.gov. There can be important differences in results from
73 published (4) and unpublished (5) versions of corresponding study documents. Clinical study reports
74 include highly detailed information on all aspects of a trial (6) and are on average about 2,000 pages long
75 (7), but it can be difficult to obtain complete and unredacted clinical study reports (8).
76 We carried out a systematic review of the human papillomavirus (HPV) vaccines’ clinical study
77 reports (9) based on an index we constructed of 206 HPV vaccine studies (10). As of July 2017, 62%
78 (92/149) of the completed studies were not published in journal publications and 48% (71/147) of the
79 completed studies on ClinicalTrials.gov had no study results posted (10). Systematic reviewers often only
80 use journal publications and trial registers for their reviews, which may increase the risk of using a data set
81 influenced by selective outcome reporting.
82 To our knowledge, no study has looked at differences of pooled estimates—such as meta-
83 analyses—of corresponding study documents of the same intervention. Our primary aim in this study was
3
Page 550 of 637
84 to compare meta-analyses of HPV vaccine data from clinical study reports with data from corresponding
85 trial register entries and journal publications. Our secondary aim was to compare the reporting of study
86 design aspects of the corresponding study documents.
87
88 Methods
89 We compared corresponding HPV vaccine study documents of clinical study reports, trial register entries
90 and journal publications to investigate the degree of reporting bias for prespecified outcomes and the
91 reporting of trial design aspects; see our protocol on PROSPERO (11) (registered as “Protocol amendment
92 no. 3” for our systematic review of the HPV vaccines (9)).
93 Clinical study reports were obtained from EMA and GlaxoSmithKline (9). We identified the clinical
94 study reports’ corresponding trial register entries on ClinicalTrials.gov and corresponding primary journal
95 publications from our published index of the HPV vaccine studies that include the search strings we used
96 (10). We assessed all identified journal publications for a study (including supplementary documents and
97 erratas) for eligible information and chose the primary publication that corresponded to the clinical study
98 report for our comparison. We did not check for eligible information in additional trial registers (such as the
99 EU Clinical Trials Register) or letters to the editors.
100 Data extraction and comparison of the study documents were carried out by two researchers (LJ
101 extracted the data; TJ checked the extractions; and PCG arbitrated). For each study document, the
102 following data were compared: study ID; number of pages; date of document; time from study completion
103 to publication in a journal; results availability; protocol availability (including pre-specification of outcomes
104 and inclusion of a statistical analysis plan); reporting of PICO criteria (participants, interventions,
105 comparisons and outcomes); and reporting of six major design-related biases defined by the Cochrane
106 Handbook (version 2011) for the Cochrane risk of bias tool (12) (random sequence generation, allocation
107 concealment, blinding of outcome assessors, blinding of personnel, blinding of participants and loss to
108 follow-up). We collected these data, as they are important to evaluate a study’s internal and external
109 validity. We did not include the Cochrane risk of bias tool domain ‘selective outcome reporting,’ since we
110 compared this domain quantitatively between corresponding documents (see below).
4
Page 551 of 637
111 For each study document, we extracted and compared data on the outcomes we assessed in our
112 systematic review (9). As our review contained 166 meta-analyses, we only compared the 20 most clinically
113 relevant outcomes (or statistically significant outcomes with a p-value ≤ 0.05; noted in parentheses).
114 Benefit outcomes: all-cause mortality; HPV-related cancer mortality; HPV-related cancer incidence; HPV-
115 related carcinoma in situ; HPV-related moderate intraepithelial neoplasia; HPV-related moderate
116 intraepithelial neoplasia or worse; and HPV-related treatment procedures. Harm outcomes: fatal harms;
117 serious harms (including those judged as 'definitely associated' with postural orthostatic tachycardia
118 syndrome [POTS] and complex regional pain syndrome [CRPS; see our systematic review protocol
119 amendment (13) for these two post hoc exploratory analyses] and the nervous system disorders that were
120 MedDRA [Medical Dictionary for Regulatory Activities] classified in this system organ class); new onset
121 diseases (including back pain and vaginal infection and the vascular disorders that were MedDRA classified
122 in this system organ class); and general harms (including fatigue, headache and myalgia). Histological
123 outcomes were assessed irrespective of involved HPV types. The most aggregated data account
124 (participants with events over the total number of participants) was used for the meta-analyses and the
125 most detailed harm account of MedDRA preferred terms was used for event comparisons. For example, if
126 harms were registered separately per harm, we would count the separate harms and summarize them as a
127 total number of harms. For all GlaxoSmithKline clinical study reports and for serious harms for Merck
128 clinical study reports, we pooled MedDRA preferred terms in their respective system organ classes. A
129 participant could potentially be included more than once in a separate analysis (e.g., if a participant
130 experienced both serious ‘headache’ and serious ‘dizziness,’ the participant would be counted twice in the
131 MedDRA system organ class analysis of serious nervous system disorders); we therefore consider the
132 MedDRA system organ class analyses exploratory.
133 Merck Sharp & Dohme did not provide a formal definition for its new onset disease category—'new
134 medical history'—but described the category as “all new reported diagnoses” in the clinical study report of
135 trial V501-019. Although 'new medical history' was not explicitly mentioned in the trial register entries and
136 journal publications, we included eligible new reported diagnoses not reported as serious or general harms
137 in this category.
5
Page 552 of 637
138 For our meta-analyses, we used the intention to treat principle. Risk ratios (RRs) were calculated
139 with the random effects inverse variance method. Random effects estimates were compared to fixed effect
140 estimates, as the former method may weigh small trials unduly if there is considerable heterogeneity
141 between trials (12).
142
143 Results
144 We included study documents from 22 randomized clinical trials and two follow-up studies and obtained 24
145 clinical study reports, 24 corresponding trial register entries and 23 corresponding primary journal
146 publications (for the remaining journal publication—HPV-003, of 61 participants—the manufacturer
147 confirmed that no journal publication had been published (10)). See Additional file 1 for our study’s PRISMA
148 statement.
149
150 Characteristics of included studies
151 The 24 included studies investigated four different HPV vaccines: Cervarix™, Gardasil™, Gardasil 9™ and an
152 HPV type 16 vaccine, and included 95,670 healthy participants (79,102 females and 16,568 males) aged 8 to
153 72. One (4%) study used a saline placebo comparator, but its participants had been HPV vaccinated before
154 randomization. Fourteen (58%) studies used vaccine adjuvants as comparators: amorphous aluminium
155 hydroxyphosphate sulphate (AAHS), aluminium hydroxide (Al[OH]3) or carrier solution. Nine (38%) studies
156 used hepatitis vaccine comparators: Aimmugen™, Engerix™, Havrix™ and Twinrix Paediatric™.
157
158 Characteristics of included study documents
159 Nearly all study documents (70/72) reported data from study start to completion; except for the clinical
160 study report and journal publication of study HPV-040 that described interim analyses. The median number
161 of pages in the clinical study reports was 1,351 (range 357 to 11,456) (see Table 1). For four studies (HPV-
162 008, HPV-013, HPV-015 and HPV-040), we obtained clinical study reports from both EMA and
163 GlaxoSmithKline (we did not account for duplicate pages). EMA’s clinical study reports were only 22% of
164 the length of the corresponding GlaxoSmithKline reports (5,316 vs. 23,645 pages). After transformation to
165 PDFs, the median number of pages in the trial register entries was 32 (range 11 to 167). Results were
6
Page 553 of 637
166 posted on ClinicalTrials.gov for 19 studies (79%) but were not posted for five studies: HPV-001, HPV-003,
167 HPV-013, HPV-033 and HPV-035. The median number of pages in the journal publications—including
168 supplementary appendices—was 11 (range 7 to 83). Twelve (52%) journal publications contained
169 supplementary appendices. The mean time from study completion to journal publication was 2.3 years (see
170 Table 1).
171
172 Inclusion of protocols
173 Ten clinical study reports (42%), no trial register entries (0%) and two journal publications (9%) included
174 protocols. All 12 protocols listed prespecified outcomes and contained statistical analysis plans (see Table
175 2). The GlaxoSmithKline trial register entries contained protocol hyperlinks to
176 ClinicalStudyDataRequest.com, but the protocols were not freely available and had to be requested. We did
177 not request the protocols, as this required us to sign a data sharing agreement, which would restrict our
178 ability to publish our results.
179
180 Reporting of major design-related biases
181 All 24 (100%) clinical study reports, no (0%) trial register entries and nine (39%) journal publications
182 reported explicitly on all six domains to be assessed for bias according to the Cochrane Handbook version
183 2011 (12) (see Table 2).
184
185 Reporting of PICO criteria
186 Compared to the trial register entries and journal publications, the clinical study reports reported on
187 average more inclusion criteria (mean: 7.0 vs. 5.8 and 4.0, respectively) and exclusion criteria (mean: 17.8
188 vs. 11.7 and 5.0) (see Table 2). As an example, while 20 (83%) clinical study reports reported that
189 participants with immunological disorders were excluded, only 12 (50%) trial register entries and nine
190 (39%) journal publications reported this criterion. All clinical study reports and journal publications
191 specified the intervention and comparator contents (including antigens, adjuvants and doses), whereas
192 only 18 (75%) and eight (33%) trial register entries specified these. Active comparators (AAHS, Al[OH]3 and
193 carrier solution) were referred to as “placebos” in 14 (58%) clinical study reports, 13 (54%) trial register
7
Page 554 of 637
194 entries and 17 (74%) journal publications. The mean number of reported primary outcomes was higher in
195 the trial register entries (3.5) than in the clinical study reports (1.6) and the journal publications (1.2). This
196 was also the case for secondary outcomes (13.0 vs. 8.8 and 3.2) (see Table 2).
197
198 Meta-analyses of benefits
199 Of our seven prespecified benefit outcomes from the clinical study reports, the trial register entries
200 included data for two (29%) and the journal publications for six (86%) (see Table 3a and Additional file 2).
202 44% of the assessed benefit data points (6,879 vs. 230 and 3,015). Due to the lack of data in the trial
203 register entries and journal publications, it was only possible to calculate the ratios of relative risk for half
204 (10/21) of the prespecified benefit comparisons (see Table 3b). The meta-analyses risk ratio estimates from
205 corresponding study documents did not differ much (see Table 3a) and the ratio of relative risk differences
206 that could be calculated were not statistically significant (see Table 3b).
207
208 Meta-analyses of harms
209 Of our 13 prespecified harm outcomes from the clinical study reports, the trial register entries included
210 data for 11 (85%) and the journal publications for 10 (77%) (see Tables 3a and 3b and Additional file 2).
212 31% of the assessed harm data points (167,550 vs. 64,143 and 51,899). It was only possible to calculate the
213 ratios of relative risk for 80% (31/39) of the prespecified harm comparisons (see Table 3b). The meta-
214 analyses risk ratio estimates did not differ much (see Table 3a) and the ratio of relative risk differences that
215 could be calculated were not statistically significant (see Table 3b).
216
217 Random effects vs. fixed effect analyses
218 We found similar results with the fixed effect model but with narrower confidence intervals, as the
219 between-trial variance is not included in this model.
220
221 Subgroup analyses

8
Page 555 of 637
222 When we excluded the studies that had no results posted on their corresponding trial register entries (HPV-
223 001, HPV-003, HPV-013, HPV-033 and HPV-035) from the clinical study report meta-analyses, the results
224 did not differ significantly.
225
226 Study document differences
227 There were substantial differences between the amount of data in the three study document types (see
228 Figures 1 to 5). For example, the journal publication for V501-013 included more cases of HPV-related
229 moderate intraepithelial neoplasia or worse compared to its clinical study report (417 vs. 370; see Figure 1).
230 The trial register entry for HPV-015 reported fewer HPV-related treatment procedures than the clinical
231 study report (160 vs. 198; see Figure 2). The trial registry entry of HPV-040 reported 10 deaths (five in each
232 group), whereas the clinical study report reported “no deaths considered as possibly related to vaccination
233 according to the investigator (up to 30 April 2011),” and the journal publication reported “No deaths had
234 been reported at the time of this interim analysis (up to April 2011).” Compared to the corresponding
235 clinical study report, the journal publication of HPV-008 only contained an aggregate total number of
236 serious harms (1,400), whereas the clinical study report contained all individual serious harms classified
237 with MedDRA preferred terms (2,043). Only the trial register entries and journal publications for HPV-023
238 and HPV-032 included serious harms classified with MedDRA preferred terms (see Figure 3). No journal
239 publication of Merck Sharp & Dohme studies included their new onset disease category: “new medical
240 history” (V501-005 to V503-006). Merck Sharp & Dohme did not provide a formal definition for 'new
241 medical history' but described the category as “all new reported diagnoses” in the clinical study report of
242 trial V501-019. Although not mentioned as an explicit category, the trial register entries reported fewer
243 events of new reported diagnoses than the clinical study reports (e.g., for V501-015: 329 vs. 35,546; see
244 Figure 4). Only the trial registry report of HPV-032 and the journal publication of V501-013 included general
245 harms (see Figure 5).
246
247 Discussion
248 There were on average 50 and 121 times more pages in the clinical study reports than in their
249 corresponding trial register entries and journal publications. This was likely a main reason why the clinical
9
Page 556 of 637
250 study reports were superior at reporting trial design aspects. If our systematic review of clinical study
251 reports (9) had relied on trial register entries or journal publications, it would have had no data for a
252 quarter of our prespecified outcomes (11/40). Although the inclusion of clinical study reports led to
253 significantly more eligible and available data, no changes in the direction of available results occurred when
254 comparing the risk ratios of corresponding meta-analyses as ratios of relative risks. This may have several
255 explanations. First, GlaxoSmithKline might be more transparent than other pharmaceutical companies (14),
256 so corresponding study documents from GlaxoSmithKline could be more consistent compared to
257 corresponding study documents from other companies (15–18). Second, we used the random effects
258 model, but more risk ratios had a narrower confidence intervals with a fixed effect model. Third, there
259 were low event numbers for several outcomes; differences in low event numbers may be overestimated
260 when using risk ratios (12). Finally, the studies were designed with lack of placebo controls and incomplete
261 reporting of harms (8) and the trial register entries and journal publications only included very few of the
262 assessed data points (from 1% to 44%) compared to the clinical study reports. This may have skewed some
263 of our comparison results towards being false-negative and led to an underestimation of harms caused by
264 the HPV vaccines. Major study design features such as the use of active comparators and the reporting
265 format of harms are not affected by the number of pages in a study document, but the vast increase in the
266 amount of detail in clinical study reports allows for a more complete understanding that might impact
267 conclusions. We have expanded on the issues of lack of placebo controls and incomplete harms reporting
268 elsewhere (8).
269
270 Strengths
271 Our comparison included 71 of 72 primary study documents (except for the journal publication of trial HPV-
272 003 with 61 participants, which does not exist). Nearly all corresponding study documents (70/72) reported
273 data from initiation to completion. To our knowledge, our study is the first with the aim of comparing
274 meta-analyses from different study document data. The majority of study document comparison studies
275 have mainly looked at harms (15–18); we looked at both benefits and harms.
276
277 Study document limitations

10
Page 557 of 637
278 The clinical study reports were incomplete and often redacted, and some eligible data may have been left
279 out, which we have described elsewhere (8). Cervarix™ clinical study reports obtained from EMA were a
280 fifth of the length of the reports that we downloaded from GlaxoSmithKline’s trial register. Merck Sharp &
281 Dohme clinical study reports (of Gardasil™, Gardasil 9™ and the HPV type 16 vaccine) were only obtained
282 from EMA. These consisted of 9,588 pages for seven trials. Thus, potentially 40,000 pages remain
283 undisclosed for our comparison of Merck Sharp & Dohme clinical study reports (8).
284 Only 12 of 71 study documents contained the study protocol. We believe that all study publications
285 should include the study protocol, as readers otherwise are unable to evaluate whether any selective
286 outcome reporting, protocol amendments or post hoc analyses were present in the study publication.
287 It was not possible to compare meta-analyses of per-protocol and intention to treat populations, as
288 we had prespecified (11). In the trial register entries and journal publications, per-protocol benefit
289 outcomes were not reported irrespective of HPV type and harm results were not reported for per-protocol
290 populations. Differences might have been more marked for these comparisons. For example, in the journal
291 publication for HPV-015, it was stated that “Few cases of CIN2+ [moderate cervical intraepithelial neoplasia
292 or worse] were recorded” for the per protocol population for CIN2+ related to HPV types 16 and 18 (25 vs.
293 34), but the corresponding clinical study report reported four times as many CIN2+ cases for the intention to
294 treat population irrespective of HPV type (103 vs. 108).
295 The lower amount of data points in journal publications might be due to space restrictions, but in
296 many biomedical journals it is possible to include large electronic appendices. As there is no space
297 restriction on ClinicalTrials.gov (19), the lower amount of data points was likely due to incomplete
298 reporting.
299 Journal publications for five studies (HPV-031, HPV-035, HPV-040, HPV-058 and HPV-069) only
300 included figures with graphs of general harms without exact numbers. We could calculate the absolute
301 numbers from the percentages of general harms that were provided for four of the five journal publications
302 (HPV-035, HPV-040, HPV-058 and HPV-069).
303 No journal publication of Merck Sharp & Dohme mentioned ‘new medical history’—a category used
304 in all seven Merck clinical study reports. Merck Sharp & Dohme described 'new medical history' as “all new
305 reported diagnoses.”

11
Page 558 of 637
306 Some data in the trial register entries and journal publications were not comparable for our
307 prespecified outcomes, for example, whereas the clinical study reports had reported an aggregate number
308 of participants experiencing ‘solicited and unsolicited’ harms, the trial register entries and journal
309 publications only reported general harms as ‘solicited’ and ‘unsolicited’ harms and that on a MedDRA
310 preferred term and total level, respectively. We decided to compare such data as number of events but
311 excluded non-aggregated data from the meta-analyses, as the data would constitute a considerable risk of
312 counting participants more than once in an analysis (e.g., for trial register entries for GlaxoSmithKline
313 studies, we only used “unsolicited” events for general harms, as these were reported aggregately). For trial
314 register entries for Merck studies, general harms were reported aggregately with local harms. We had not
315 prespecified local harms as an outcome, so we did not use these data.
316 Since a journal publication page usually has a higher word and character count than a clinical study
317 report page (that usually has a higher word count than a trial register PDF page), it may have been more
318 appropriate to compare the word count of the study documents instead of the number of pages. As we
319 received clinical study reports both from EMA and GlaxoSmithKline for some clinical study reports, some of
320 the pages were duplicates and the median number of pages was therefore overestimated to some extent.
321
322 Similar studies
323 Our study supplements earlier studies that found reporting bias from clinical study reports to trial register
324 entries and journal publications (16–18,20). Golder et al. performed a systematic review of 11 comparison
325 studies that compared the number of harms in corresponding published and unpublished study documents
326 (15). Golder et al. found that 62% (mean) of the harms and 2-100% of the serious harms would have been
327 missed if the comparison studies had relied on journal publications. Similarly, our systematic review of the
328 HPV vaccines of clinical study reports would have missed 62% of the assessed harm data points if it had
329 relied on trial register entries and 69% of the harms if it had relied on journal publications. Our systematic
330 review would have included 1% more serious harms classified with MedDRA preferred terms if it had relied
331 on trial registers but missed 26% serious harms classified with MedDRA preferred terms if it was based on
332 journal publications. It would also have missed 99% of the benefit data points if it had relied on trial
333 register entries and 56% if it had relied on journal publications.

12
Page 559 of 637
334 We found a mean time from trial completion to journal publication of 2.3 years. This is similar to a
335 study by Sreekrishnan et al.—from 2018, of 2,000 neurology studies—that found a mean time to
336 publication of 2.2 years (21), but less similar to a study by Ross et al.—from 2013, of 1,336 clinical trials—
337 that found a mean time to publication of 1.8 years (22).
338
339 Conclusion
340 There were no significant differences in the meta-analysis estimates of the assessed outcomes from
341 corresponding study documents. The clinical study reports were the quantitively and qualitatively superior
342 study documents and should be used as primary data sources in systematic reviews; trial register entries
343 and journal publications should be used concomitantly, as some data may only be available in trial register
344 entries or journal publications. A systematic review of the HPV vaccines would have had considerably less
345 information and data included if it relied on trial register entries and journal publications instead of clinical
346 study reports. A full data set would be expected to be available from case report forms and individual
347 participant data, but there are regulatory barriers that need to be lifted before independent researchers
348 can access such data (8). Corresponding study documents ought to use consistent terminology and provide
349 all aggregate and individual benefits and harms data. To test our results’ generalisability, we recommend
350 that other researchers replicate our method of comparison for other interventions.
351
352 List of abbreviations
353 AAHS Amorphous aluminium hydroxyphosphate sulphate
354 Al(OH)3 Aluminium hydroxide
355 CRPS Chronic regional pain syndrome
356 EMA European Medicines Agency
357 FDA Food and Drug Administration
358 GSK GlaxoSmithKline
359 HPV Human papillomavirus
360 ICH International Council for Harmonisation of Technical Requirements for Pharmaceuticals for
361 Human Use

13
Page 560 of 637
362 MedDRA Medical Dictionary for Regulatory Activities
363 Merck Merck & Co., Inc. or Merck Sharp & Dohme outside the United States and Canada
364 PICO Patient, intervention, comparator and outcome
365 POTS Postural orthostatic tachycardia syndrome
366 PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses
367 PROSPERO International Prospective Register of Systematic Reviews
368 Declarations
369 Ethics approval and consent to participate
370 Not applicable.
371
372 Consent for publication
373 Not applicable.
374
375 Availability of data and material
376 The datasets generated and analysed during our study are available from the corresponding author (LJ)
377 upon request.
378
379 Competing interests
380 All authors have completed the ICMJE uniform disclosure form. LJ declares no support from any
381 organisation for the submitted work, no financial relationships with any organisations that might have an
382 interest in the submitted work, no other relationships or activities that could appear to have influenced the
383 submitted work. PCG spoke by video link about the HPV vaccines at the IFICA conference in 2018 but
384 received no fee or reimbursement for this. PCG and TJ were co-signatories of a complaint to the European
385 Ombudsman on maladministration in relation to the EMA investigation of possible harms from HPV
386 vaccines. PCG does not regard this as a competing interest. TJ was a co-recipient of a UK National Institute
387 for Health Research grant (HTA – 10/80/01 Update and amalgamation of two Cochrane Reviews:
388 neuraminidase inhibitors for preventing and treating influenza in healthy adults and children—
14
Page 561 of 637
389 https://www.journalslibrary.nihr.ac.uk/programmes/hta/108001#/). TJ is also in receipt of a Cochrane
390 Methods Innovations Fund grant to develop guidance on the use of regulatory data in Cochrane reviews. TJ
391 is occasionally interviewed by market research companies about phase I or II pharmaceutical products. In
392 2011-14, TJ acted as an expert witness in a litigation case related to the antiviral oseltamivir, in two
393 litigation cases on potential vaccine-related damage and in a labour case on influenza vaccines in
394 healthcare workers in Canada. He has acted as a consultant for Roche (1997-99), GSK (2001-2), Sanofi-
395 Synthelabo (2003) and IMS Health (2013). In 2014-16, TJ was a member of three advisory boards for
396 Boehringer Ingelheim. TJ was a member of an independent data monitoring committee for a Sanofi Pasteur
397 clinical trial on an influenza vaccine.
398
399 Funding
400 This study was funded by the Nordic Cochrane Centre, which is funded by the Danish government.
401
402 Authors' contributions
403 LJ wrote the first draft. LJ and TJ contributed to the conception of the review, the design of the review, the
404 collection and assembly of data, the analysis and interpretation of the data, the drafting of the article, the
405 critical revision of the article for important intellectual content and the final approval of the article. PCG
406 contributed to the conception of the review, the critical revision of the article for important intellectual
407 content and the final approval of the article. All authors had full access to all the data in the study and takes
408 responsibility for the integrity of the data and the accuracy of the data analysis.
409
410 Acknowledgements
411 We would like to thank EMA for its assistance.
412
413 References
414 1. Guidelines: ICH. Available from: http://www.ich.org/products/guidelines.html
15
Page 562 of 637
415 2. Doshi P, Jefferson T. Open data 5 years on: a case series of 12 freedom of information requests for
416 regulatory data to the European Medicines Agency. Trials. 2016 Feb 11;17:78. doi: 10.1186/s13063-
417 016-1194-7.
418 3. Doshi P. FDA to begin releasing clinical study reports in pilot programme. BMJ. 2018 Jan 23;360:k294.
419 4. Jefferson T, Jones M, Doshi P, Mar CD. Neuraminidase inhibitors for preventing and treating influenza
420 in healthy adults: systematic review and meta-analysis. BMJ. 2009 Dec 8;339:b5106. doi:
421 10.1136/bmj.b5106
422 5. Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJ, et al. Neuraminidase inhibitors
423 for preventing and treating influenza in adults and children. Cochrane Database Syst Rev. 2014 Apr
424 10. doi: 10.1002/14651858.CD008965.pub4.
425 6. Jefferson T, Jørgensen L. Redefining the ‘E’ in EBM. BMJ Evid Based Med. 2018;23:46-47. doi:
426 10.1136/bmjebm-2018-110918.
427 7. Doshi P, Jefferson T. Clinical study reports of randomised controlled trials: an exploratory review of
428 previously confidential industry reports. BMJ Open. 2013 Jan 1;3(2):e002496. doi: 10.1136/bmjopen-
429 2012-002496.
430 8. Jørgensen L, Doshi P, Gøtzsche PC, Jefferson T. Challenges of independent assessment of potential
431 harms of HPV vaccines. BMJ. 2018 Sep 24;362:k3694. doi: 10.1136/bmj.k3694.
432 9. Jørgensen L, Gøtzsche. PC, Jefferson T. Benefits and harms of the human papillomavirus vaccines:
433 systematic review of industry and non-industry study reports. PROSPERO. 2017 Jan. Available from:
434 https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20170030.pdf.
435 10. Jørgensen L, Gøtzsche PC, Jefferson T. Index of the human papillomavirus (HPV) vaccine industry
436 clinical study programmes and non-industry funded studies: a necessary basis to address reporting
437 bias in a systematic review. Syst Rev. 2018 Jan 18;7(1):8. doi: 10.1186/s13643-018-0675-z.
438 11. Jørgensen L, Gøtzsche PC, Jefferson T. Protocol: Comparison of clinical study reports with trial registry
439 reports and journal publications. PROSPERO. 2018 Mar 12. Available from:
441 12. Cochrane Handbook for Systematic Reviews of Interventions. Available from:
442 http://training.cochrane.org/handbook.
16
Page 563 of 637
443 13. Jørgensen L, Gøtzsche PC, Jefferson T. Protocol amendment no. 1 and 2: Benefits and harms of the
444 human papillomavirus vaccines: systematic review of industry and non-industry study reports.
445 PROSPERO. 2017 Nov 16. Available from:
447 14. Nisen P, Rockhold F. Access to Patient-Level Data from GlaxoSmithKline Clinical Trials. N Engl J Med.
448 2013 Aug 1;369(5):475-8. doi: 10.1056/NEJMsr1302541.
449 15. Golder S, Loke YK, Wright K, Norman G. Reporting of Adverse Events in Published and Unpublished
450 Studies of Health Care Interventions: A Systematic Review. PLoS Med. 2016 Sep 20;13(9):e1002127.
451 doi: 10.1371/journal.pmed.1002127.
452 16. Maund E, Guski LS, Gøtzsche PC. Considering benefits and harms of duloxetine for treatment of stress
453 urinary incontinence: a meta-analysis of clinical study reports. CMAJ. 2017 Feb 6;189(5):E194-E203.
454 doi: 10.1503/cmaj.151104.
455 17. Schroll JB, Penninga EI, Gøtzsche PC. Assessment of Adverse Events in Protocols, Clinical Study
456 Reports, and Published Papers of Trials of Orlistat: A Document Analysis. PLoS Med. 2016 Aug
457 16;13(8):e1002101. doi: 10.1371/journal.pmed.1002101
458 18. Sharma T, Guski LS, Freund N, Gøtzsche PC. Suicidality and aggression during antidepressant
459 treatment: systematic review and meta-analyses based on clinical study reports. BMJ. 2016 Jan
460 27;352:i65. doi: 10.1136/bmj.i65.
461 19. Frequently Asked Questions - ClinicalTrials.gov. Available from: https://clinicaltrials.gov/ct2/manage-
462 recs/faq.
463 20. Hughes S, Cohen D, Jaggi R. Differences in reporting serious adverse events in industry sponsored
464 clinical trial registries and journal articles on antidepressant and antipsychotic drugs: a cross-sectional
465 study. BMJ Open. 2014 Jul 9;4(7):e005535. doi: 10.1136/bmjopen-2014-005535.
466 21. Sreekrishnan A, Mampre D, Ormseth C, Miyares L, Leasure A, Ross JS, et al. Publication and
467 Dissemination of Results in Clinical Trials of Neurology. JAMA Neurol. 2018;75(7):890-891. doi:
468 10.1001/jamaneurol.2018.0674.
469 22. Ross JS, Mocanu M, Lampropulos JF, Tse T, Krumholz HM. Time to Publication Among Completed
470 Clinical Trials. JAMA Intern Med. 2013 May 13;173(9):825–8. doi: 10.1001/jamainternmed.2013.136.
17
Page 564 of 637
471
472 Additional Material
473 Figure 1. Word document: Comparison of HPV vaccine study documents: number of reported cases of HPV-
474 related moderate intraepithelial neoplasia or worse
475 Figure 2. Word document: Comparison of HPV vaccine study documents: number of reported cases of HPV-
476 related referral procedures
477 Figure 3. Word document: Comparison of HPV vaccine study documents: number of reported serious
478 harms
479 Figure 4. Word document: Comparison of HPV vaccine study documents: number of reported new onset
480 diseases
481 Figure 5. Word document: Comparison of HPV vaccine study documents: number of reported general
482 harms
483 Additional file 1. Word document: Comparison of HPV vaccine study documents: PRISMA 2009 checklist
484 Additional file 2. PDF document: Comparison of HPV vaccine study documents: meta-analyses
18
Page 565 of 637
485 Table 1: Comparison of HPV vaccine clinical study reports with trial register entries and journal
486 publications: date and availability of clinical study reports, trial registry report results and journal
487 publications
Manufacturer Clinical study report Trial register entries from ClinicalTrials.gov Journal publication
Study ID N= Date of NCT ID N= Results Date results Reference N= pages Date

pagesa report pages posted posted published
GlaxoSmithKli HPV-001 5,813 November NCT00689741 19 No Not Harper DM et al. Efficacy of a Bivalent L1 Virus-like Particle Vaccine in Prevention of Infection with Human 10 November
ne 13, 2004 applicable Papillomavirus Types 16 and 18 in Young Women: A Randomized Controlled Trial. Lancet. 2004 Nov 13- 13, 2004
19;364(9447):1757-65. doi: 10.1016/S0140-6736(04)17398-4.
HPV-003 799 April 13, NCT00263744 12 No Not Not published Not Not
2003 applicable applicable applicable
HPV-008 11,456 July 1, NCT00122681 132 Yes January 20, Paavonen J et al. Efficacy of human papillomavirus (HPV)-16/18 AS04-adjuvanted vaccine against cervical infection 25 July 25,
2009 2010 and precancer caused by oncogenic HPV types (PATRICIA): final analysis of a double-blind, randomized study in young 2009
women. Lancet. 2009 Jul 25;374(9686):301-14. doi: 10.1016/S0140-6736(09)61248-4.
HPV-013 8,323 December NCT00196924 12 No September Medina DM et al. Safety and immunogenicity of the HPV-16/18 AS04-adjuvanted vaccine: a randomized, controlled 8 May 1,
1, 2005 20, 2005 trial in adolescent girls. J Adolesc Health. 2010 May;46(5):414-21. doi: 10.1016/j.jadohealth.2010.02.006. 2010
HPV-015 6,290 March 31, NCT00294047 136 Yes March 27, Skinner S et al. Efficacy, safety, and immunogenicity of the human papillomavirus 16/18 AS04-adjuvanted vaccine in 20 December
2015 2012 women older than 25 years: 4-year interim follow-up of the phase 3, double-blind, randomized controlled VIVIANE 20, 2014
study. Lancet. 2014 Dec 20;384(9961):2213-27. doi: 10.1016/S0140-6736(14)60920-X.
HPV-023 936 November NCT00518336 167 Yes October 25, Naud PS et al. Sustained efficacy, immunogenicity, and safety of the HPV-16/18 AS04-adjuvanted vaccine, Human 19 June 19,
12, 2009 2011 Vaccines & Immunotherapeutics. 2014 Jun 19;10:8. doi: 10.4161/hv.29532. 2014
HPV-029 1,543 June 9, NCT00578227 50 Yes January 6, Pedersen C et al. Randomized trial: immunogenicity and safety of coadministered human papillomavirus-16/18 AS04- 9 January 1,
2009 2010 adjuvanted vaccine and combined hepatitis A and B vaccine in girls. J Adolesc Health. 2012 Jan;50(1):38-46. doi: 2012
10.1016/j.jadohealth.2011.10.009.
HPV-030 1,351 June 17, NCT00652938 52 Yes August 31, Schmeink CE et al. Co-administration of human papillomavirus-16/18 AS04-adjuvanted vaccine with hepatitis B 8 November
2010 2010 vaccine: randomized study in healthy girls. Vaccine. 2011 Nov 15;29(49):9276-83. doi: 10.1016/j.vaccine.2011.08.037. 15, 2011
HPV-031 476 December NCT00344032 25 Yes December Bhatla N et al. Immunogenicity and safety of human papillomavirus-16/18 AS04-adjuvanted cervical cancer vaccine in 10 February 4,
4, 2013 15, 2009 healthy Indian women. J Obstet Gynaecol Res. 2010 Feb;36(1):123-32. doi: 10.1111/j.1447-0756.2009.01167.x. 2010
HPV-032 2,912 November NCT00316693 30 Yes December Konno R et al. Efficacy of human papillomavirus type 16/18 AS04-adjuvanted vaccine in Japanese women aged 20 to 9 July 4,
1, 2008 16, 2009 25 years: final analysis of a phase 2 double-blind, randomized controlled trial. Int J Gynecol Cancer. 2010 Jul;20(5):847- 2010
55. doi: 10.1111/IGC.0b013e3181da2128.
HPV-033 587 March 27, NCT00290277 11 No Not Kim YJ et al. Vaccination with a human papillomavirus (HPV)-16/18 AS04-adjuvanted cervical cancer vaccine in Korean 8 August 1,
2007 applicable girls aged 10-14 years. J Korean Med Sci. 2010 Aug;25(8):1197-204. doi: 10.3346/jkms.2010.25.8.1197. 2010
HPV-035 451 June 9, NCT00306241 14 No March 23, Ngan HY et al. Human papillomavirus-16/18 AS04-adjuvanted cervical cancer vaccine: immunogenicity and safety in 9 June 15,
2008 2006 healthy Chinese women from Hong Kong. Hong Kong Med J. 2010 Jun;16(3):171-9. 2010
HPV-038 957 August 5, NCT00485732 28 Yes December Kim SC et al. Human papillomavirus 16/18 AS04-adjuvanted cervical cancer vaccine: immunogenicity and safety in 15- 9 June 30,
2009 17, 2009 25 years old healthy Korean women. J Gynecol Oncol. 2011 Jun 30; 22(2): 67–75. doi: 10.3802/jgo.2011.22.2.67. 2011
HPV-040 2,892 April 13, NCT00534638 45 Yes January 26, Lehtinen M et al. Safety of the human papillomavirus (HPV)-16/18 AS04-adjuvanted vaccine in adolescents aged 12– 14 March 3,
2016 2016 15 years: Interim analysis of a large community-randomized controlled trial. Hum Vaccin Immunother. 2016 Dec; 2015
12(12): 3177–3185. doi: 10.1080/21645515.2016.1183847.
HPV-058 1,745 May 28, NCT00996125 22 Yes June 27, 2012 Zhu F et al. Immunogenicity and safety of the HPV-16/18 AS04-adjuvanted vaccine in healthy Chinese girls and women 17 July 1,
2012 aged 9 to 45 years. Hum Vaccin Immunother. 2014;10(7):1795-806. doi: 10.4161/hv.28702. 2014
HPV-063 1,474 July 19, NCT00929526 41 Yes October 15, Konno R et al. Efficacy of the human papillomavirus (HPV)-16/18 AS04-adjuvanted vaccine against cervical 19 July 1,
2013 2012 intraepithelial neoplasia and cervical infection in young Japanese women. Hum Vaccin Immunother. 2014;10(7):1781- 2014
94. doi: 10.4161/hv.28712.
HPV-069 819 June 6, NCT01277042 32 Yes December 3, Zhu F et al. Immunogenicity and safety of the HPV-16/18 AS04-adjuvanted vaccine in healthy Chinese girls and women 17 July 1,
2013 2013 aged 9 to 45 years. Hum Vaccin Immunother. 2014;10(7):1795-806. doi: 10.4161/hv.28702. 2014
Merck Sharp V501-005 357 March 8, NCT00365378 28 Yes April 9, 2010 Koutsky LA et al. A controlled trial of a human papillomavirus type 16 vaccine. N Engl J Med. 2002 Nov 7 November
& Dohme 2005 21;347(21):1645-51. doi: 10.1056/NEJMoa020586. 21, 2002
V501-013 1,797 November NCT00092521 48 Yes November Garland SM. Quadrivalent vaccine against human papillomavirus to prevent anogenital diseases. N Engl J Med. 2007 30 May 10,
12, 2007 20, 2009 May 10;356(19):1928-43. doi: 10.1056/NEJMoa061760. 2007
V501-015 713 November NCT00092534 45 Yes November The FUTURE II Study Group. Quadrivalent vaccine against human papillomavirus to prevent anogenital diseases. N 36 May 10,
13, 2007 26, 2009 Engl J Med. 2007 May 10;356(19):1928-43. doi: 10.1056/NEJMoa061760 2007
V501-018 1,014 August 8, NCT00092547 60 Yes May 4, 2010 Reisinger KS et al. Safety and persistent immunogenicity of a quadrivalent human papillomavirus types 6, 11, 16, 18 L1 11 August 18,
2005 virus-like particle vaccine in preadolescents and adolescents: a randomized controlled trial. Pediatr Infect Dis J. 2007 2014
Mar;26(3):201-9.
V501-019 2,645 November NCT00090220 83 Yes February 1, Muñoz N et al. Safety, immunogenicity, and efficacy of quadrivalent human papillomavirus (types 6, 11, 16, 18) 9 June 6,
17, 2009 2010 recombinant vaccine in women aged 24-45 years: a randomized, double-blind trial. Lancet. 2009 Jun 2009
6;373(9679):1949-57. doi: 10.1016/S0140-6736(09)60691-7.
V501-020 2,595 January 27, NCT00090285 32 Yes November Giuliano AR et al. Efficacy of quadrivalent HPV vaccine against HPV Infection and disease in males. N Engl J Med. 2011 76 February 3,
2010 19, 2009 Feb 3;364(5):401-11. doi: 10.1056/NEJMoa0909537. 2011
V503-006 467 June 10, NCT01047345 33 Yes December Garland SM et al. Safety and immunogenicity of a 9-valent HPV vaccine in females 12-26 years of age who previously 83 November
2011 22, 2014 received the quadrivalent HPV vaccine. Vaccine. 2015 Nov 27;33(48):6855-64. doi: 10.1016/j.vaccine.2015.08.059. 27, 2015
Total pages 58,412 1,157 463
488 aA page was defined as one A4 PDF page regardless of the number of words or characters per page.
489
19
Page 566 of 637
490 Table 2: Comparison of HPV vaccine clinical study reports with trial register entries and journal
491 publications: inclusion of protocol and reporting of trial design aspects including PICO criteria
492
Inclusion of protocol and reporting of trial design aspects including PICO Clinical study Trial register Publications:
criteria reports: N=24 entries: N=24 N=23
Protocol
Included in study document 10 (42%) 0 (0%) 2 (9%)
- Prespecified outcomes 10 (100%) Not applicable 2 (100%)
- Included statistical analysis plan 10 (100%) Not applicable 2 (100%)
Reporting of six major design-related biases defined by the Cochrane
Handbooka
Randomization method was explicitly specified 24 (100%) 0 (0%) 22 (96%)
Allocation concealment was explicitly specified 24 (100%) 0 (0%) 17 (74%)
Blinding of outcome assessors was explicitly specified 24 (100%) 23 (96%) 17 (74%)
Blinding of personnel was explicitly specified 24 (100%) 11 (46%) 12 (52%)
Blinding of participants was explicitly specified 24 (100%) 23 (96%) 12 (52%)
Loss to follow-up (attrition) was explicitly accounted for 24 (100%) 20 (83%) 23 (100%)
Population
Specified inclusion criteria 24 (100%) 24 (100%) 22 (96%)
- Mean number of inclusion criteria 7.0 5.8 4.0
Specified exclusion criteria 24 (100%) 24 (100%) 20 (87%)
- Mean number of exclusion criteria 17.8 11.7 5.0
Intervention
Specified HPV vaccine antigens 24 (100%) 18 (75%) 23 (100%)
Specified HPV vaccine adjuvants 24 (100%) 8 (33%) 23 (100%)
Specified dose 24 (100%) 6 (25%) 21 (91%)
Comparator
Specified content 24 (100%) 8 (33%) 23 (100%)
Specified dose 24 (100%) 6 (25%) 21 (91%)
Reported active comparator as a “placebo”b 14 (58%) 13 (54%) 17 (74%)
Outcomes
Primary outcomes explicitly specified 24 (100%) 24 (100%) 18 (78%)
- Mean number of primary outcomes 1.6 3.5 1.2
Secondary outcomes explicitly specified 24 (100%) 24 (100%) 14 (61%)
- Mean number of secondary outcomes 8.8 13.0 3.2
493 aCochrane Handbook: http://training.cochrane.org/handbook.
494 bActive comparators included amorphous aluminium hydroxyphosphate sulphate (AAHS), aluminium hydroxide (Al[OH] ), carrier solution and
3
495 hepatitis vaccines (Aimmugen™, Engerix-B™, Havrix™ and Twinrix Paediatric™).
496
20
Page 567 of 637
497 Table 3a: Comparison of HPV vaccine clinical study reports with trial register entries and journal
498 publications: results of benefits and harms meta-analyses of intention to treat analyses irrespective of
499 HPV type
500
Results of benefits and harms Clinical study reports Trial register entries Journal publications
meta-analyses of intention to HPV vaccine Comparator Risk ratioe HPV Compar Risk ratioe HPV vaccine Comparator Risk ratioe
treat analyses irrespective of (n=47,075) (n=48,595) [95% CI] vaccine ator [95% CI] (n=47,044f) (n=48,565f) [95% CI]
HPV typea (n=47,0 (n=48,5
75) 95)
Benefits
All-cause mortality 45 38 1.19 [0.65, 2.19] 39 31 1.30 [0.73, 2.30] 35 28 1.20 [0.51, 2.80]
HPV-related cancer mortality 2 1 1.44 [0.23, 9.12] 0 0 Not applicable 0 0 Not applicable
HPV-related cancer incidence 7 3 1.68 [0.51, 5.49] 0 0 Not applicable 1 0 3.01 [0.12, 73.85]
HPV-related carcinoma in situ 367 490 0.73 [0.53, 1.00] 0 0 Not applicable 212 247 0.85 [0.61, 1.19]
HPV-related moderate 538 763 0.81 [0.59, 1.11] 0 0 Not applicable 251 308 0.82 [0.69, 0.96]
intraepithelial neoplasia
HPV-related moderate 952 1,239 0.78 [0.66, 0.91] 0 0 Not applicable 665 848 0.77 [0.65, 0.92]
intraepithelial neoplasia or worse
HPV-related treatment 76 84 0.90 [0.66, 1.22] 180 240 0.75 [0.62, 0.91]
1,018 1,416 0.71 [0.63, 0.80]
procedures
Total reported benefit data 2,929 3,950 Not applicable 115 115 Not applicable 1,344 1,671 Not applicable
points
Harms
Participants with fatal harms 45 38 1.19 [0.65, 2.19] 39 31 1.30 [0.73, 2.30] 35 28 1.20 [0.51, 2.80]
Total number of fatal harms or 79 51 Not applicable 39 31 Not applicable 35 28 Not applicable
MedDRA classified fatal harms
Participants with serious harms 1,404 1,357 1.01 [0.94, 1.08] 1,398 1,349 1.01 [0.94, 1.09] 1,241 1,234 1.01 [0.93, 1.09]
Total number of serious harms or 1,741 1,628 Not applicable 1,763 1,636 Not applicable 1,255 1,249 Not applicable
MedDRA classified serious harms
- Judged ‘definitely associated’ 95 57 1.54 [1.11, 2.14] 88 55 1.52 [1.08, 2.12] 9 2 1.94 [0.57, 6.57]
with CRPSb
- Judged ‘definitely associated’ 56 26 1.92 [1.21, 3.07] 52 23 2.00 [1.23, 3.25] 6 2 1.79 [0.45, 7.22]
with POTSb
- Nervous system disorders 72 46 1.49 [1.02, 2.16] 69 45 1.47 [1.01, 2.15] 12 7 1.45 [0.53, 3.94]
Participants with new onset 14,258 14,014 0.99 [0.97, 1.02] 4,874 4,779 1.02 [0.95, 1.10] 4,740 4,801 1.00 [0.92, 1.09]
diseasesc
Total number of new onset 47,474 46,662 Not applicable 9,972 8,673 Not applicable 4,740 4,801 Not applicable
diseases or MedDRA classified
new onset diseases
- Back pain 397 336 1.15 [1.00, 1.33] 68 63 1.08 [0.77, 1.52] 0 0 Not applicable
- Vaginal infection 369 420 0.87 [0.76, 1.00] 0 0 Not applicable 0 0 Not applicable
- Vascular disorders 234 294 0.80 [0.67, 0.94] 0 0 Not applicable 0 0 Not applicable
Participants with general harmsd 13,248 12,394 1.07 [1.03, 1.11] 3,522 3,468 1.07 [1.00, 1.15] 8,457 7,697 1.05 [1.01, 1.10]
Total number of general harms 37,999 31,916 Not applicable 22,236 19,793 Not applicable 21,001 18,790 Not applicable
or MedDRA classified general
harms
- Fatigue 4,933 4,489 1.13 [1.08, 1.18] 4,255 3,901 1.13 [1.07, 1.19] 2,343 2,210 1.15 [1.04, 1.26]
- Headache 5,561 5,246 1.06 [1.02, 1.11] 4,934 4,587 1.07 [1.03, 1.12] 2,443 2,372 1.08 [1.01, 1.16]
- Myalgia 3,989 3,047 1.41 [1.24, 1.60] 3,508 2,688 1.44 [1.21, 1.71] 1,868 1,193 1.57 [1.23, 2.01]
Total reported MedDRA classified 87,293 80,257 Not applicable 34,010 30,133 Not applicable 27,031 24,868 Not applicable
data points
501 aSee Additional file 2 for the meta-analyses. It was not feasible to present this summary table for the 16 subgroups that the 24 included studies
502 comprised (based on age-group, gender, type of HPV vaccine and comparator).
503 bWe asked a physician with clinical expertise in complex regional pain syndrome (CRPS) and postural orthostatic tachycardia syndrome (POTS) to
504 assess the reported MedDRA preferred terms as 'definitely,' 'probably,' 'probably not' or 'definitely not' associated with the syndromes. We sent an
505 Excel sheet to the physician with all the reported MedDRA terms. The physician was blinded, as the Excel sheet contained no outcome data. When
506 the physician had assessed all the MedDRA terms, we synthesized the data for those MedDRA terms that the physician judged 'definitely'
507 associated with POTS or CRPS.
508 cNew onset diseases were compiled of the harm categories ‘medically significant conditions’ (for Cervarix) and ‘new medical history’ (for Gardasil,
509 Gardasil 9 and the HPV 16 vaccine). GlaxoSmithKline defined 'medically significant conditions' as "Adverse events prompting emergency room or
510 physician visits that are not (1) related to common diseases or (2) routine visits for physical examination or vaccination, or SAEs [serious adverse
511 events] that are not related to common diseases. Serious adverse events related to common diseases were reported but are not classified as
512 medically significant conditions for analysis purposes. Common diseases include: upper respiratory infections, sinusitis, pharyngitis, gastroenteritis,
513 urinary tract infections, cervicovaginal yeast infections, menstrual cycle abnormalities and injury." Merck Sharp & Dohme did not provide a formal
514 definition for 'new medical history' but described the category as “all new reported diagnoses” in the clinical study report of study V501-019.
515 dGeneral harms was compiled of the harm categories ‘solicited general symptoms,’ ‘unsolicited general symptoms’ (for Cervarix) and ‘systemic
516 adverse experiences’ (for Gardasil, Gardasil 9 and the HPV 16 vaccine). GlaxoSmithKline defined 'solicited' general adverse events as, "Adverse
517 events to be recorded as endpoints in the clinical study. The presence/occurrence/intensity of these events is actively solicited from the subject or
518 an observer during a specified post-vaccination follow-up period." GlaxoSmithKline defined 'unsolicited' general adverse event as, "Any AE [adverse
519 event] reported in addition to those solicited during the clinical study. Also, any "solicited" symptom with onset outside the specified period of
520 follow-up for solicited symptoms was reported as an unsolicited AE." Merck Sharp & Dohme defined 'systemic adverse event' as, "…any systemic
521 clinical adverse event that developed on the day of vaccination or during the 14 days after vaccination was recorded on the VRC [vaccination report
522 card] along with the date it started and the last date it was present."
523 eRisk ratios were calculated with the random effects inverse variance method.
524 fThe numbers of participants for ‘HPV vaccine’ and ‘comparator’ in the journal publication column were subtracted by 31 and 30 participants,
525 respectively, as no journal publication existed for trial HPV-003 that included 31 and 30 participants.
21
Page 568 of 637
526 Table 3b: Comparison of HPV vaccine clinical study reports with trial register entries and journal
527 publications: ratio of relative risk differences of results of benefits and harms
528
Ratios of relative risk (RRR) of results of RRR of clinical study reports RRR of clinical study reports RRR of trial register entries
benefits and harmsa vs. trial register entries vs. journal publications vs. journal publications
Benefits
All-cause mortality 0.95 [0.41, 2.18] 1.03 [0.36, 2.92] 1.08 [0.39, 3.02]
- HPV-related cancer mortality Not applicabled Not applicable Not applicable
HPV-related cancer incidence Not applicable 0.55 [0.02, 17.13] Not applicable
HPV-related carcinoma in situ Not applicable 0.85 [0.54, 1.36] Not applicable
HPV-related moderate intraepithelial Not applicable 0.98 [0.69, 1.41] Not applicable
neoplasia
HPV-related moderate intraepithelial Not applicable 1.02 [0.80, 1.28] Not applicable
neoplasia or worse
HPV-related treatment procedures 0.79 [0.57, 1.09] 0.95 [0.76, 1.19] 1.20 [0.84, 1.72]
Harms
Fatal harms 0.95 [0.41, 2.18] 1.03 [0.36, 2.92] 1.08 [0.39, 3.02]
Serious harms 1.00 [0.90, 1.11] 1.00 [0.90, 1.11] 1.00 [0.93, 1.09]
- Judged ‘definitely associated’ with CRPSb 1.01 [0.63, 1.62] 0.79 [0.22, 2.81] 0.78 [0.22, 2.78]
- Judged ‘definitely associated’ with POTSc 0.96 [0.49, 1.88] 1.07 [0.25, 4.64] 1.12 [0.26, 4.86]
- Nervous system disorders 1.01 [0.60, 1.73] 1.03 [0.35, 3.00] 1.01 [0.35, 2.96]
New onset diseases 0.97 [0.90, 1.05] 0.99 [0.91, 1.08] 1.02 [0.92, 1.09]
- Back pain 1.06 [0.73, 1.54] Not applicable Not applicable
- Vaginal infection Not applicable Not applicable Not applicable
- Vascular disorders Not applicable Not applicable Not applicable
General harms 1.00 [0.92, 1.08] 1.02 [0.96, 1.08] 1.02 [0.94, 1.11]
- Fatigue 1.00 [0.93, 1.07] 0.98 [0.88, 1.09] 0.98 [0.88, 1.09]
- Headache 0.99 [0.93, 1.06] 0.98 [0.91, 1.06] 0.99 [0.91, 1.07]
- Myalgia 0.98 [0.79, 1.21] 0.90 [0.68, 1.18] 0.92 [0.68, 1.24]
529 aRelativerisk ratio differences were calculated as a risk ratio calculated with the random effects inverse variance method vs. a risk ratio calculated
530 with the random effects inverse variance method (see Table 3a).
531 bCRPS: complex regional pain syndrome (see Table 3a).
532 cPOTS: postural orthostatic tachycardia syndrome (see Table 3a).
533 dNot applicable: when no data were available for the outcome in one (or both) of the compared study document groups (see Table 3a).
22
Page 569 of 637
Figure 1: Comparison of HPV vaccine trial documents: number of reported cases of HPV-related
moderate intraepithelial neoplasia or worse
900
800
700
600
500
400
300
200
100
0
V5 005
V5 013
V5 015
V5 018
V5 019
V5 020
06
HP 01
HP 03
HP 08
HP 13
HP 15
HP 23
HP 29
HP 30
HP 31
HP 32
HP 33
HP 35
HP 38
HP 40
HP 58
HP 63
V5 069
-0
0
0
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
-
01
01
01
01
01
01
03
HP
Clinical study report Trial registry report Journal publication
Page 570 of 637

Figure 2: Comparison of HPV vaccine trial documents: number of reported cases of HPV-related
treatment procedures
900
800
700
600
500
400
300
200
100
0
V5 005
V5 013
V5 015
V5 018
V5 019
V5 020
06
HP 01
HP 03
HP 08
HP 13
HP 15
HP 23
HP 29
HP 30
HP 31
HP 32
HP 33
HP 35
HP 38
HP 40
HP 58
HP 63
V5 069
-0
0
0
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
-
01
01
01
01
01
01
03
HP
Page 571 of 637

0
500
1000
1500
2000
2500
HP
V-
HP 001
V-
HP 003
V-
HP 008
V-
HP 013
V-
HP 015
V-
HP 023
V-
HP 029
V-
HP 030
V-
Clinical study report

HP 031
V-
HP 032
V-
HP 033
V-
HP 035
V-
HP 038
Page 572 of 637

V-
HP 040
Trial registry report

V-
HP 058
V-
HP 063
V
V5 -06
01 9
V5 -00
01 5
V5 -01
01 3
Journal publication
V5 -01
01 5
V5 -01
01 8
V5 -01
Figure 3: Comparison of HPV vaccine trial documents: number of reported serious harms
01 9
V5 -02
03 0
-0
06
Figure 4: Comparison of HPV vaccine trial documents: number of reported new onset diseases
40000
35000
30000
25000
20000
15000
10000
5000
0
01 5
01 3
01 5
01 8
01 9
03 0
06
HP 001
HP 003
HP 008
HP 013
HP 015
HP 023
HP 029
HP 030
HP 031
HP 032
HP 033
HP 035
HP 038
HP 040
HP 058
HP 063
01 9
V5 -00
V5 -01
V5 -01
V5 -01
V5 -01
V5 -02
V5 -06
-0
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V-
V
HP
Page 573 of 637

0
5000
10000
15000
20000
HP 25000
V-
HP 001
V-
HP 003
V-
HP 008
V-
HP 013
V-
HP 015
V-
HP 023
V-
HP 029
V-
HP 030
V-
Clinical study report

HP 031
V-
HP 032
V-
HP 033
V-
HP 035
V-
HP 038
V-
Page 574 of 637

HP 040
Trial registry report

V-
HP 058
V-
HP 063
V
V5 -06
01 9
V5 -00
01 5
V5 -01
01 3
Journal publication
V5 -01
01 5
V5 -01
01 8
V5 -01
01 9
V5 -02
Figure 5: Comparison of HPV vaccine trial documents: number of reported general harms
03 0
-0
06
Additional file 1: Comparison of HPV vaccine trial documents: PRISMA 2009 checklist
Reported on
page #
TITLE
Title 1 Identify the report as a systematic review, meta-analysis or both. Not
applicable
ABSTRACT
Structured summary 2 Provide a structured summary including, as applicable: background; objectives; 2-3
INTRODUCTION
Rationale 3 Describe the rationale for the review in the context of what is already known. 3-4
Objectives 4 Provide an explicit statement of questions being addressed with reference to 3-4
METHODS
Protocol and 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web 3
registration address), and, if available, provide registration information including registration
number.
Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report 3-4
Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact 3-4
with study authors to identify additional studies) in the search and date last
searched.
Search 8 Present full electronic search strategy for at least one database, including any See our
limits used, such that it could be repeated. indexa
Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in See our
systematic review, and, if applicable, included in the meta-analysis). systematic
reviewb
Data collection 10 Describe method of data extraction from reports (e.g., piloted forms, 4
process independently, in duplicate) and any processes for obtaining and confirming data
from investigators.
Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding 3-4
Risk of bias in 12 Describe methods used for assessing risk of bias of individual studies (including 4
Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, 5
including measures of consistency (e.g., I2) for each meta-analysis.
Reported on
page #
Page 575 of 637

Additional file 1: Comparison of HPV vaccine trial documents: PRISMA 2009 checklist
Risk of bias across 15 Specify any assessment of risk of bias that may affect the cumulative 6
studies evidence (e.g., publication bias, selective reporting within studies).
Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup 8

RESULTS
Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in See our
the review, with reasons for exclusions at each stage, ideally with a flow systematic
diagram. reviewb
Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., 5-6
Risk of bias within 19 Present data on risk of bias of each study and, if available, any outcome See our
studies level assessment (see item 12). systematic
reviewb
Results of individual 20 For all outcomes considered (benefits or harms), present, for each study: See Additional
studies (a) simple summary data for each intervention group (b) effect estimates file 2
and confidence intervals, ideally with a forest plot.
Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals See Table 3a
and measures of consistency.
Risk of bias across 22 Present results of any assessment of risk of bias across studies (see Item See our
studies 15). systematic
reviewb
Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup 8
DISCUSSION
Summary of evidence 24 Summarize the main findings including the strength of evidence for each 9-10
main outcome; consider their relevance to key groups (e.g., healthcare
providers, users, and policy makers).
Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at 10
review-level (e.g., incomplete retrieval of identified research, reporting
bias).
FUNDING
Funding 27 Describe sources of funding for the systematic review and other support 13
(e.g., supply of data); role of funders for the systematic review.
a
Jørgensen L, Gøtzsche PC, Jefferson T. Index of the human papillomavirus (HPV) vaccine industry clinical study programmes
and non-industry funded studies: a necessary basis to address reporting bias in a systematic review. Systematic Reviews. 2018
Jan 18;7:8. DOI: 10.1186/s13643-018-0675-z
b
Jørgensen L, Gøtzsche. PC, Jefferson T. Benefits and harms of the human papillomavirus vaccines: systematic review of
industry and non-industry study reports. PROSPERO. 2017 Jan. Available from:
https://www.crd.york.ac.uk/PROSPEROFILES/56093_PROTOCOL_20170030.pdf
Page 576 of 637

Additional file 2
Comparison of HPV vaccine study documents: meta-analyses
Table of contents
1. Clinical study reports 2
2. Trial register entries 22
3. Journal publications 42
1
Page 577 of 637
1. Clinical study reports
2
Page 578 of 637
3
Page 579 of 637
1.3. Incidence of HPV-related cancers (anal, cervical, oropharyngeal, penile, vaginal and vulvar) irrespective of HPV
type*: intention to treat analysis
*1.3. Risk ratio for GlaxoSmithKline studies (i.e., HPV-0xx): 2.24 [0.47, 10.74]; risk ratio for Merck Sharp & Dohme studies (i.e., V50x-xxx): 1.14 [0.19, 6.99]. There
were no reported cases of anal or penile cancer.
4
Page 580 of 637
adenocarcinoma in situ [AIS], cervical intraepithelial neoplasia grade 3 [CIN3], penile intraepithelial neoplasia
grade 3 [PIN3], vaginal intraepithelial neoplasia grade 3 [VIN3] and vulvar intraepithelial neoplasia grade 3
[VaIN3]) irrespective of HPV type*: intention to treat analysis
were no reports of AIN3, PIN3, VIN3 or VaIN3 irrespective of HPV type.
5
Page 581 of 637
intraepithelial neoplasia grade 2 [VIN2] and vulvar intraepithelial neoplasia grade 2 [VaIN2]) irrespective of
6
Page 582 of 637
were no reports of AIN2+ irrespective of HPV type.
7
Page 583 of 637
*1.7. Risk ratio for GlaxoSmithKline studies (i.e., HPV-0xx): 0.70 [0.59, 0.84]; risk ratio for Merck Sharp & Dohme studies (i.e., V50x-xxx): 0.71 [0.60, 0.83]. Only
cervical procedure: 844 in the HPV vaccine group vs. 1,139 in the control group, risk ratio 0.74 [0.65, 0.84]; only EGL procedure: 174 vs. 277, risk ratio 0.63 [0.50,
0.80].
8
Page 584 of 637
1.8. Fatal harms*: intention to treat analysis
*1.8. Risk ratio for GlaxoSmithKline studies (i.e., HPV-0xx): 1.43 [0.65, 3.15]; risk ratio for Merck Sharp & Dohme studies (i.e., V50x-xxx): 1.08 [0.40, 2.96]. The
most common fatal serious harms were: ‘road traffic accident’ (five in the HPV vaccine group and seven in the control group, risk ratio 0.77 [0.24, 2.46]);
‘completed suicide’ (four and eight, risk ratio 0.58 [0.15, 2.19]); ‘cardiorespiratory arrest’ (three and two, risk ratio 0.99 [0.13, 7.65]); ‘gunshot wound’ (two and
three, risk ratio 0.74 [0.09, 5.85]); and ‘homicide’ (two and two, risk ratio 0.95 [0.14, 6.50]). The fatal serious harms most increased by the HPV vaccines were:
‘cardiac arrest’ (two in the HPV vaccine group and none in the control group, risk ratio 3.00 [0.31, 28.82]); ‘traumatic intracranial haemorrhage’ (two and none,
risk ratio 3.00 [0.31, 28.82]); ‘systemic lupus erythematosus’ (two and none, risk ratio 3.00 [0.31, 28.82]); ‘metastases to lung’ (two and none, risk ratio 3.00
[0.31, 28.82]); and ‘renal failure acute’ (two and none, risk ratio 3.00 [0.31, 28.82]). The fatal serious harms most decreased by the HPV vaccines were:
‘completed suicide’ (four in the HPV vaccine group and eight in the control group, risk ratio 0.58 [0.15, 2.19]); and ‘road traffic accident’ (five and seven, risk
ratio 0.77 [0.24, 2.46]).
9
Page 585 of 637
10
Page 586 of 637
1.10. Serious harms judged as ‘definitely associated’* with chronic regional pain syndrome (CRPS): intention to treat
analysis
*1.10. Risk ratio for GlaxoSmithKline studies (i.e., HPV-0xx): 1.55 [1.09, 2.20]; risk ratio for Merck Sharp & Dohme studies (i.e., V50x-xxx): 1.47 [0.56, 3.89]. We
asked a physician with clinical expertise in CRPS to assess the reported MedDRA preferred terms as 'definitely,' 'probably,' 'probably not' or 'definitely not'
associated with CRPS. We sent an Excel sheet to the physician with all the reported MedDRA terms. The physician was blinded, as the Excel sheet contained no
outcome data. When the physician had assessed all the MedDRA terms, we synthesized the data for those MedDRA terms that the physician judged 'definitely'
associated with CRPS and compared it to the reported serious harms.
11
Page 587 of 637
asked a physician with clinical expertise in POTS to assess the reported MedDRA preferred terms as 'definitely,' 'probably,' 'probably not' or 'definitely not'
associated with POTS. We sent an Excel sheet to the physician with all the reported MedDRA terms. The physician was blinded, as the Excel sheet contained no
associated with POTS and compared it to the reported serious harms.
12
Page 588 of 637
13
Page 589 of 637
1.13. New onset diseases ('medically significant conditions' and 'new medical history'*): intention to treat analysis
*1.13. Risk ratio for 'medically significant conditions’ (GlaxoSmithKline): 0.98 [0.90, 1.06]; risk ratio for ‘new medical history’ (Merck Sharp & Dohme): 1.00 [0.97,
1.03]; risk ratio for the follow-up periods for the trials V501-005, V501-019 and V501-020 (Merck Sharp & Dohme): 0.98 [0.94, 1.01] (2,296 participants with new
medical history in the HPV vaccine group vs. 2,365 participants with new medical history in the control group. The trials V501-005, V501-019 and V501-020 split
estimate, we only included the new onset diseases reported in the vaccination period for the trials V501-005, V501-019 and V501-020).
14
Page 590 of 637
1.14. New onset diseases most associated with the HPV vaccines (‘medically significant conditions’*) - ‘back pain’:
1.28]; risk ratio for the follow-up periods for the trials V501-005, V501-019 and V501-020: 0.85 [0.60, 1.19]. The trials V501-005, V501-019 and V501-020 split
estimate, we only included the new onset diseases reported in the vaccination period for the trials V501-005, V501-019 and V501-020.
15
Page 591 of 637
1.15. New onset diseases most inversely associated with the HPV vaccines (‘new medical history’*) - ‘vaginal
*1.15. Risk ratio for 'medically significant conditions’ (GlaxoSmithKline): not applicable; risk ratio for ‘new medical history’ (Merck Sharp & Dohme): 0.87 [0.76,
1.00]; risk ratio for the follow-up period for the trial V501-019: 0.61 [0.26, 1.48]. The trial V501-019 split the reporting of new onset diseases into the vaccination
period and the follow-up period. To avoid double counting of participants in the total risk ratio estimate, we only included the new onset diseases reported in
the vaccination period for the trial V501-019.
16
Page 592 of 637
MedDRA system organ class ‘vascular disorders (10047065)': intention to treat analysis
1.03]; risk ratio for the follow-up periods for the trials V501-019 and V501-020: 0.93 [0.51, 1.73]. The trials V501-019 and V501-020 split the reporting of new
onset diseases into the vaccination period and the follow-up period. To avoid double counting of participants in the total risk ratio estimate, we only included
the new onset diseases reported in the vaccination period for the trials V501-019 and V501-020.
17
Page 593 of 637
1.17. General harms (‘solicited and unsolicited’ and ‘systemic adverse events’*): intention to treat analysis
*1.17. Risk ratio for ‘solicited and solicited’ (GlaxoSmithKline): 1.11 [1.06, 1.16]; risk ratio for ‘systemic adverse events’ (Merck Sharp & Dohme): 1.01 [0.98,
1.03]. The total numbers of participants with general harms in GlaxoSmithKline studies were reported as ‘solicited [SGAE] and unsolicited [UGAE]’ combined.
18
Page 594 of 637
1.18. General harms most associated with the HPV vaccines (‘solicited’ and ‘unsolicited’) - ‘fatigue’: intention to
treat analysis
*1.18. Risk ratio for ‘solicited’ (GlaxoSmithKline): 1.14 [1.09, 1.19]; risk ratio for ‘unsolicited’ (GlaxoSmithKline): 1.00 [0.15, 6.53]; risk ratio for ‘systemic adverse
events’ (Merck Sharp & Dohme): 0.92 [0.70, 1.20]. To avoid double counting of participants in the total risk ratio estimate, we excluded the ‘unsolicited’ adverse
events from total risk ratio estimate for studies that reported ‘solicited’ adverse events.
19
Page 595 of 637
1.19. General harms most associated with the HPV vaccines (‘solicited’ and ‘unsolicited’) - ‘headache’: intention to
treat analysis
20
Page 596 of 637
1.20. General harms most associated with the HPV vaccines (‘solicited’ and ‘unsolicited’) - ‘myalgia’: intention to
treat analysis
21
Page 597 of 637
2. Trial register entries
22
Page 598 of 637
irrespective of HPV type: intention to treat analysis
*2.2. Risk ratio for 'medically significant conditions’ (GlaxoSmithKline): not applicable; risk ratio for ‘new medical history’ (Merck Sharp & Dohme): not
applicable.
23
Page 599 of 637
type: intention to treat analysis
applicable.
24
Page 600 of 637
[VaIN3]) irrespective of HPV type: intention to treat analysis
applicable.
25
Page 601 of 637
HPV type: intention to treat analysis
applicable.
26
Page 602 of 637
applicable.
27
Page 603 of 637
28
Page 604 of 637
29
Page 605 of 637
30
Page 606 of 637
analysis
31
Page 607 of 637
32
Page 608 of 637
33
Page 609 of 637
*2.13. Risk ratio for 'medically significant conditions’ (GlaxoSmithKline): 1.02 [0.95, 1.10]; risk ratio for ‘new medical history’ (Merck Sharp & Dohme): not
applicable.
34
Page 610 of 637
2.14. New onset diseases most associated with the HPV vaccines (‘medically significant conditions’*) - ‘back pain’:
*2.14. Risk ratio for 'medically significant conditions’ (GlaxoSmithKline): not applicable; risk ratio for ‘new medical history’ (Merck Sharp & Dohme): 1.08 [0.77,
1.52].
35
Page 611 of 637
2.15. New onset diseases most inversely associated with the HPV vaccines (‘new medical history’) - ‘vaginal
applicable.
36
Page 612 of 637
2.16. New onset diseases ('medically significant conditions' and 'new medical history') reported within the MedDRA
applicable.
37
Page 613 of 637
2.17. General harms (‘solicited,’ ‘unsolicited’ and ‘systemic adverse events’*): intention to treat analysis
*2.17. Risk ratio for ‘solicited’ (GlaxoSmithKline): 1.07 [1.00, 1.15]; risk ratio for ‘systemic adverse events’ (Merck Sharp & Dohme): not applicable.
38
Page 614 of 637
2.18. General harms most associated with the HPV vaccines (‘solicited,’ ‘unsolicited’ and ‘systemic adverse events’*)
- ‘fatigue’: intention to treat analysis
*2.18. Risk ratio for ‘solicited’ (GlaxoSmithKline): 1.14 [1.08, 1.20]; risk ratio for ‘unsolicited’ (GlaxoSmithKline): not applicable; risk ratio for ‘systemic adverse
39
Page 615 of 637
- ‘headache’: intention to treat analysis
40
Page 616 of 637
- ‘myalgia’: intention to treat analysis
events’ (Merck Sharp & Dohme): not applicable. To avoid double counting of participants in the total risk ratio estimate, we excluded the ‘unsolicited’ adverse
41
Page 617 of 637
3. Journal publications
42
Page 618 of 637
applicable.
43
Page 619 of 637
type*: intention to treat analysis
44
Page 620 of 637
[VaIN3]) irrespective of HPV type*: intention to treat analysis
45
Page 621 of 637
*3.5. Risk ratio for GlaxoSmithKline studies (i.e., HPV-0xx): not applicable; risk ratio for Merck Sharp & Dohme studies (i.e., V50x-xxx): 0.82 [0.69, 0.96]. There
46
Page 622 of 637
47
Page 623 of 637
`
48
Page 624 of 637
49
Page 625 of 637
50
Page 626 of 637
analysis
51
Page 627 of 637
52
Page 628 of 637
53
Page 629 of 637
*3.13. Risk ratio for 'medically significant conditions’ (GlaxoSmithKline): 1.00 [0.92, 1.09]; risk ratio for ‘new medical history’ (Merck Sharp & Dohme): not
applicable.
54
Page 630 of 637
3.14. New onset diseases most associated with the HPV vaccines (‘medically significant conditions’) - ‘back pain’:
applicable.
55
Page 631 of 637
3.15. New onset diseases most inversely associated with the HPV vaccines (‘new medical history’) - ‘vaginal
applicable.
56
Page 632 of 637
3.16. New onset diseases ('medically significant conditions' and 'new medical history') reported within the MedDRA
applicable.
57
Page 633 of 637
3.17. General harms (‘solicited,’ ‘unsolicited’ and ‘systemic adverse events’*): intention to treat analysis
events’ (Merck Sharp & Dohme): 1.01 [0.97, 1.04]. To avoid double counting of participants in the total risk ratio estimate, we only included ‘solicited’ adverse
events if the journal publication also had reported ‘unsolicited’ adverse events.
58
Page 634 of 637
- ‘fatigue’: intention to treat analysis
59
Page 635 of 637
- ‘headache’: intention to treat analysis
60
Page 636 of 637
- ‘myalgia’: intention to treat analysis
61
Page 637 of 637

PHD Thesis - Benefits and Harms of The HPV Vaccines - Lars Jørgensen MD (Jcl649) - 12 November 2018

Uploaded by

Document Information

Original Title

Copyright

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

PHD Thesis - Benefits and Harms of The HPV Vaccines - Lars Jørgensen MD (Jcl649) - 12 November 2018

Uploaded by

Copyright:

GRADUATE SCHOOL OF HEALTH AND MEDICAL SCIENCES

Benefits and Harms of the Human Papillomavirus

THIS THESIS WAS SUBMITTED TO

Name of department Nordic Cochrane Centre

Institutnavn Nordisk Cochrane Center

Author Dr Lars Jørgensen MD

Title Benefits and harms of the human papillomavirus (HPV) vaccines

Titel Gavnlige og skadelige virkninger af HPV-vaccinerne

Principal supervisor Prof Peter C Gøtzsche DrMedSci

Co-supervisor Dr Tom Jefferson MD MSc MRCGP FFPHM

Submitted 12 November 2018

Word count 5,971

Preface and acknowledgements 4

Preface and acknowledgements

This thesis is a synopsis based on the following five research papers:

3. Jørgensen L, Doshi P, Gøtzsche PC and Jefferson T. Challenges of independent assessment of

The following three papers support the thesis:

Resumé på dansk [summary in Danish]

Human papillomavirus (HPV) infection

Regulatory data in systematic reviews

Cochrane’s risk of bias tool

Paper 1: Evaluation of Cochrane’s risk of bias tool

Paper 2: Index of the HPV vaccine study programmes

Paper 3: Challenges of independent HPV vaccine assessment

Paper 4: Systematic review of the HPV vaccines

Paper 5: Comparison of HPV vaccine study documents

Description of the research projects

Paper 1: Evaluation of Cochrane’s risk of bias tool

Paper 2: Index of the HPV vaccine study programmes

Paper 3: Challenges of independent HPV vaccine assessment

Paper 4: Systematic review of the HPV vaccines

Paper 5: Comparison of HPV vaccine study documents

Paper 1: Evaluation of Cochrane’s risk of bias tool

Paper 2: Index of the HPV vaccine study programmes

Paper 3: Challenges of independent HPV vaccine assessment

Paper 4: Systematic review of the HPV vaccines

Paper 5: Comparison of HPV vaccine study documents

Conclusions and perspectives for further research

21. Dehlendorff C, Sparén P, Baldur-Felskov B, Herweijer E, Arnheim-Dahlström L, Ploner A, et al. Ef-

RESEARCH Open Access

Evaluation of the Cochrane tool for

* Correspondence: lj@cochrane.dk; larsjorgensens@gmail.com

Background clinical trials and an observational study of how the tool is

as an independent bias domain. Adding a domain would

funded trials, such as selection of control groups. The Authors’ contributions

“(bias) AND (tool OR assess*) AND (Cochrane)”

(((systematic[sb]) AND (”0001/01/01”[PDAT]: ”2014/12/31”[PDAT])) AND (systematic[Title/Abstract] OR systematical[Title/Abstract] OR

53 Lundh Strengths: ”...our data suggest that industry sponsorship should be 75

Reference First author Selected commentaries from references

Title of PhD thesis:

Benefits and harms of the human papillomavirus (HPV) vaccines

This declaration concerns the following article:

The PhD student’s contribution to the article:

*Benchmark scale of the PhD student’s contribution to the article

Signature of the co-authors:

PhD student: Principal supervisor:

RESEARCH Open Access

Index of the human papillomavirus (HPV)

Background (HPV) vaccines as a basis for a systematic review of clin-

Page 100 of 637

Most industry studies had a study programme spe- Discussion

Page 101 of 637

Page 102 of 637

Page 103 of 637

Page 104 of 637

Page 105 of 637