Cancer Genetics 206 (2014) 441e448

REVIEW ARTICLE
Understanding the limitations of next generation
sequencing informatics, an approach to clinical
pipeline validation using artificial data sets
Robert Daber*, Shrey Sukhadia, Jennifer J.D. Morrissette
Center for Personalized Diagnostics, University of Pennsylvania School of Medicine, Philadelphia, PA

The advantages of massively parallel sequencing are quickly being realized through the adoption
of comprehensive genomic panels across the spectrum of genetic testing. Despite such wide-
spread utilization of next generation sequencing (NGS), a major bottleneck in the implementation
and capitalization of this technology remains in the data processing steps, or bioinformatics. Here
we describe our approach to defining the limitations of each step in the data processing pipeline
by utilizing artificial amplicon data sets to simulate a wide spectrum of genomic alterations.
Through this process, we identified limitations of insertion, deletion (indel), and single nucleotide
variant (SNV) detection using standard approaches and described novel strategies to improve
overall somatic mutation detection. Using these artificial data sets, we were able to demonstrate
that NGS assays can have robust mutation detection if the data can be processed in a way that
does not lead to large genomic alterations landing in the unmapped data (i.e., trash). By using
these pipeline modifications and a new variant caller, AbsoluteVar, we have been able to validate
SNV mutation detection to 100% sensitivity and specificity with an allele frequency as low 4% and
detection of indels as large as 90 bp. Clinical validation of NGS relies on the ability for mutation
detection across a wide array of genetic anomalies, and the utility of artificial data sets demon-
strates a mechanism to intelligently test a vast array of mutation types.
Keywords Next generation sequencing, bioinformatics, validation, sensitivity, artificial data set
ª 2014 Elsevier Inc. All rights reserved.

Rapid advancements in next generation sequencing (NGS)
have opened the door for unprecedented diagnostic capa-
bilities (1e4). While massively parallel sequencing makes it
feasible to rapidly sequence large genomic regions, the
overall utility is limited by the least dependable or reproduc-
ible steps in the assay (5,6). Although perceived as a single
test, an NGS assay is composed of three distinct, yet inter-
dependent modules consisting of library preparation,
sequencing, and bioinformatic analysis (Figure 1A). The
methods used for each of these modules have intrinsic lim-
itations, which critically impact the overall sensitivity and
specificity of a given test.
Most NGS validation strategies involve comparison of
mutations previously characterized by standard molecular
techniques to determine variant detection concordance in
Received October 1, 2013; received in revised form November
19, 2013; accepted November 20, 2013.
* Corresponding author.
E-mail address: Robert.Daber@uphs.upenn.edu
overlapping genes, followed by confirmation of the additional
variants identified in the NGS data with an alternative platform
(1,7). This is a common approach, due to clinical expertise in
these technologies and the availability of specimens with
characterized mutations. This approach has limited utility in
oncology specimens where tumor heterogeneity may result in
allelic burdens below 15e20%. In addition, methods such as
Sanger sequencing confirmation, often considered the “gold
standard,” are inadequate due to innate sensitivity limitations
(8). The aforementioned validation strategies also do not
address false-negative detection in all of the genomic posi-
tions that are included in the panel, yet not fully interrogated
by an alternative methodology. Specifically, if a mutation,
such as a large insertion, was not previously characterized by
an alternative method, failure of the NGS assay to detect this
mutation would be unknown, since only NGS-identified vari-
ants in novel regions are further confirmed.
One solution is a validation strategy that utilizes an alter-
native NGS methodology with overlapping coverage to
confirm false-positive and -negative results across all genes

2210-7762/$ - see front matter ª 2014 Elsevier Inc. All rights reserved.
http://dx.doi.org/10.1016/j.cancergen.2013.11.005
442
within the panel and to confirm variants seen in this lower
allelic burden range. This approach is useful to address false
variant calls that may result from either the library preparation
or sequencing methodology; however, most NGS assays
share similar informatics pipelines. This creates a situation
where variant-positive and -negative sites may be concordant
between two different NGS approaches, yet may collectively
represent false calls due to limitations in the informatics
alone. A classic example of this limitation is in the detection of
intermediate-sized insertions or deletions (indels) (9,10). This
limitation represents a significant deficiency in the validation
of NGS, since overall assay limits of detection (LOD) based
upon indel size cannot be adequately addressed. The
severity of this limitation is also amplified in many clinical labs
where the expertise of both the informatics groups and clinical
groups are mutually exclusive, often resulting in frustrating
and unsuccessful communication surrounding this bottleneck.
To bridge the gap between the expertise of these groups
and to demonstrate the impact each processing task has on
the raw data, we broke down each step in the informatics
pipeline to validate the informatics approach of our NGS
assay by defining the relationship between each processing
step and the sensitivity and specificity of variant detection.
We first created artificial data sets that modeled a range of
mutation types. These included single nucleotide variants
(SNVs), insertions, and deletions over a wide range of
genomic loci and at defined allelic frequencies ranging from
w1e50%, to determine the detection limitations for the bio-
informatic software solutions utilized. Since every read
generated from the sequencer has a unique read name, we
were able to monitor the fate of every read during pipeline
processing to determine at which steps artificial variant reads
were either improperly mapped, filtered out for quality, or
simply not identified during the variant calling process. This
approach allowed us to investigate the advantages and lim-
itations of various bioinformatic tools, different algorithms,
algorithm input parameters, and overall algorithm configu-
rations. Using this knowledge, we were then able to construct
novel algorithms and pipeline configurations to maximize
overall variant detection performance and to define actual
sensitivities and specificities of our analysis approach.
Materials and methods
Data from one de-identified data set was used to create
artificial fastq files used for validation. These data were
generated from a custom amplicon panel containing 382
amplicons spread over 33 genes. Target enrichment was
performed using a customized panel of amplicons (Truseq
Custom Amplicon; Ilumina, San Diego, CA). As per the
manufacturer’s protocol, a library was created and subse-
quently sequenced on the Ilumina MiSeq with 150 bp paired
end reads to an average depth of coverage of 2,700X. After
de-multiplexing the data for each sample, the fastq files were
processed through our laboratory’s informatics pipeline to
identify variants present in the normal data (Figure 1B).
Artificial variant files were generated by searching the
fastq files for specific reference sequences (Supplemental
Tables 1e3) and replacing the sequence text string with a
sequence containing one of the artificial variants. The locations
and mutations were selected at random, with the requirement
R. Daber et al.
that they needed to impact a range of different loci. For reads that
received artificial insertions, each read was subsequently trim-
med back to 150 bp after inserting the artificial sequence. To
simulate variant allele frequencies ranging from w1e50%,
different variant files were created by counting the total number of
matching target reads and then randomly replacing the wild type
sequence with the artificial sequence in the specified percent of
total reads. The method utilized replacement of sequence based
upon a 100% match strategy; therefore, since random errors
exist in the data, we also had to count the number of times a
sequence was replaced to get a final absolute artificial allele
frequency. In all situations, the actual artificial variant allele fre-
quency was lower than the targeted percentage.
After each artificial set of fastq files were generated, data
was processed through either our standard analysis pipeline or
through a modified version of the pipeline. This pipeline utilized
the onboard analysis workflows on the MiSeq (Illumina), as
well as different iterations of a custom analysis workflow. In
addition to in house scripts, the following software programs
were used: BTrimV64 (11) to trim amplicon primer sequences;
Novoalign V2.07.18 and V3.00.05 for read mapping (http://
novocraft.com); SamTools V0.1.18 to sort, merge, and index
bam files (12,13) (http://samtools.sourceforge.net/); Picard
V1.46 to fix read groups (http://picard.sourceforge.net/); Bed-
Tools V2.12.0 to remove off-target reads (14) (https://code.
google.com/p/bedtools/); VCFTools V0.1.9 to sort vcf files
(15) (http://vcftools.sourceforge.net/); SnpEff V3.0 (16) (http://
snpeff.sourceforge.net/) and Annovar V25th May 2012 for
variant annotation (17) (http://www.openbioinformatics.org/
annovar/); and GATK V1.6-13 for depth of coverage, read
quality trimming, and variant calling (18,19) (http://www.
broadinstitute.org/gatk/). Quality filtering of bam files before
variant calling was performed to clip reads to an average base
quality score of 22 and to remove reads with mapping qualities
below 40 and alignment scores below 95. AbsoluteVar, a new
variant caller, utilizes a base-counting strategy to determine
the number of base counts for each position in the target re-
gion, and generates a variant call file for each position where a
non-reference base is detected at a depth and/or allele fre-
quency specified by the user.
Results
The initial analytical pipeline was modeled after those
commonly used in the NGS community. This algorithm be-
gins with de-multiplexing of individual sample reads, followed
by mapping of reads to the reference genome, filtering of off-
target reads, data quality filtering, base pair depth of
coverage calculations across every base in the target re-
gions, and, finally, variant calling and annotation (Figure 1B).
Artificial fastq files were generated, which contained a range
of mutation types across the amplicon reads corresponding
to our panel (Table 1, see Materials and methods), and
processed through various iterations of the pipeline. Larger
deletions were not tested since large deletions would influ-
ence amplicon-based capture due to the primer locations.
Limitations of read alignment/mapping
Our validation began by exploring the impact of various data
alignment/mapping approaches and input parameters to
NGS validation and artificial data sets 443

Figure 1 (A) Modularity of next generation sequencing assays: library preparation, sequencing, and bioinformatics. (B) Data pro-
cessing pipeline highlighting the various steps through which data must proceed to generate a final report. Increased sensitivity for
mutation detection was achieved through development of two new algorithms, AbsoluteVar and GarbagePicker, and by calling large
indels on unfiltered data.

understand the interplay between successful alignment and workflow), reads with a homologous sequence on chromo-
the sensitivity and specificity of mutation detection. We some 2 were properly placed onto chromosome 9. Corrected
compared two strategies commonly employed in NGS soft- mapping of the off-target capture reads resulted in removal of
ware solutions: mapping to the entire genome or mapping to a false-positive variant reads on chromosome 9. False-positive
restricted genome representing the amplicon target regions variants, which were produced by mapping restricted to the
and potential off-target homologous sites. Mapping reads to amplicon panel, were identified in every sample and, therefore,
the amplicon target regions has been utilized by the onboard could be filtered as artifacts; however, mismapping resulted in
bioinformatics processing in the Illumina MiSeq as part of the inaccurate read depths at these loci, which diluted the allele
amplicon workflow to reduce data processing times. This frequency of true variants. With variant calling dependent upon
approach is limited by the need to know all potential off-target allele frequency, this dilution could result in false-negative calls
capture regions a priori. Comparison of these mapping stra- for real mutations within the targeted amplicon.
tegies identified several false-positive and -negative variant Using the whole genome for read mapping, all artificial reads
calls resulting from the restricted alignment search space. containing SNV insertions up to 34 bp and deletions up to 30 bp
When allowing read data to map to the entire genome with were successfully mapped to the correct genomic locations
both Novoalign (V2.07.18) or Illumina (MiSeq Resequencing using Novoalign (V2.07.18) as the aligner. Since the mapping
methodology aligns a single read at a time, there was no impact
Table 1 Artificial variants introduced into the raw data from the artificial allele frequency on mapping. For those read
Number of Allele frequency
pairs containing indels larger than w30 bp, two trends were
Artificial mutation loci/types Range of range for each
identified in the data based upon where the artificial indel
resided. We considered these either “orphaned” or “lost”
type tested sizes (bp) change (%)
alignment reads. The “orphaned” pattern occurred when an
SNV 25 1 1.6e50.1 artificial indel was found predominantly in one read for a given
Insertion 49 2e90 0.3e69.0 read pair and the indel-containing mate would fail to map, while
Deletion 40 1e37 0.5e44.0 its “normal” or mostly normal mate would map correctly. This
444 R. Daber et al.

Figure 2 Raw data visualized in the IGV, which highlights data patterns for a single amplicon. In normal samples sequenced by
paired-end reads, the distribution at the top of the window demonstrates longer bars with the overlapping segments between the
forward and reverse reads showing a higher depth of coverage. During data processing, reads containing an insertion (purple bar) in
only one read end up being filtered out, resulting in discordant read depths between the read pairs. (Color versions of these illustrations
are available on the journal’s website at www.cancergeneticsjournal.org.)

would occur when an insertion was present near the ends of an
amplicon and affected only one read of the read pair. The “lost”
pattern occurred when the artificial indel affected both reads,
leading to a general lack of mapping for both pairs. Since the
data utilized in this approach was generated from amplicons
with the same starts and stops to every read, indels affecting
only one mate, and therefore demonstrating discordant read
counts over the amplicon, could be visually detected by
reviewing the mapped data in the Integrative Genomics Viewer
(IGV; http://www.broadinstitute.org/igv/) (Figure 2); however,
indels affecting both reads resulted in a normal data pattern
(due to the lost, unmappable reads) and, therefore, could not be
visually detected. Since these reads would never map, this
demonstrated a hard limit of indel variant detection related to
the first step in the pipeline.
Review of the run data showed that each of the un-
mapped reads were present in the alignment file, but con-
tained a flag value (http://samtools.sourceforge.net/SAMv1.
pdf), which described that it was unmapped (http://picard.
sourceforge.net/explain-flags.html). To help identify ampli-
cons with a large fraction of unmapped reads, we developed
a tool called GarbagePicker. Since every amplicon contains
reads with the same start and stop primer sequence, we
were able to capture flag-value statistics (i.e., both mates
mapped, one mate unmapped, both mates unmapped) for
every amplicon read based upon searching for its primer
sequences in the post-alignment data. With this tool, we
were able to then identify amplicons where a given fraction of
the sequencing reads were flagged as unmapped and
manually investigate the sequence in the unmapped reads to
identify the mutation. After a baseline was established for the
number of reads that did not map for all amplicons in the
panel, thresholds were established to flag only amplicons
that showed an atypical mapping pattern (data not shown).
With this approach, manual review of misbehaving amplicons
detected the artificial mutations in every artificial data set.
After our initial pipeline validation, Novoalign was updated
(V3.0) to include a wider dynamic range for mapping quali-
ties. Using the same artificial variant files, the updated
version of Novoalign was able to successfully map all artifi-
cial reads with an insertion size up to 90 bp. Deletions <20
bp were always mapped properly, while deletions >20 bp
were often mapped as if the remaining bases after the
deletion were inserted. This marked a huge improvement in
the ability to detect large indels.
Quality filtering
Despite the ability to map indels as large as 90 bp, the data
quality processing steps implemented after mapping resulted
NGS validation and artificial data sets
Figure 3 Mutation detection sensitivity for various-sized insertions comparing insertion detection on bam files before (circles) and
after (squares) quality filtering: (A) <6 bp. (B) 7e16 bp. (C) 17e26 bp. (D) 21e90 bp. Insertions <6 bp are detected equally with the
pre- and post-quality filtered data, whereas insertions >16 bp are never detected with post-quality filtering.
in the inability to consistently detect indels >6 bp without manual
data inspection or the use of GarbagePicker (Figure 3). Since
GATK requires an indel fraction and minimum indel count
parameter, all artificial indels 6 bp were detected to the
specified allele frequency of 5%, with no false-positives detec-
ted. By tracing individual artificial variant reads using their
unique read names (identifiers), we found that they were
removed during the quality filter step used to clean up data that
has either low mapping quality or low alignment scores
(Figure 2). By altering the stringency on quality control filtering,
we were able to obtain higher sensitivity in detecting larger
indels; however, this decreased stringency had a negative
impact on false-positive SNV detection (data not shown).
As an alternative approach to sacrificing SNV specificity
for indel detection sensitivity, we altered our processing
pipeline to call large (>6 bp) insertions on the pre-filtered
data set and call small insertions and SNVs after quality
filtering. By incorporating this change (Figure 1B) into our
processing pipeline, we were able to successfully identify
insertions up to 90 bp and deletions up to 37 bp (the limit set
by the mapping step) with no impact on specificity in the data
analyzed (Figure 3). For lower limits of detection, 100%
sensitivity was obtained by going as low as a 7% allelic
burden for pure (i.e., not complex) insertions and deletions,
respectively. Complex rearrangements involving both the
deletion of the wild type sequence and insertion of a foreign
sequence at the same location were not tested. Despite the
detection of deletions >20 bp, only those deletions <20 bp
445
were properly identified by GATK as a deletion. The larger
deletions were often mis-identified as insertions, with the
undeleted portion of the read being “inserted” at the starting
point for the deletion. In some cases, mutant-containing
reads were mapped as both an insertion and a deletion,
which split the total mutation allele frequency over two
different variants. As a result, overall sensitivity was
decreased in these cases, where a variant was only identified
if either the deletion or insertion were present at 4% allele
frequency (Supplemental Figure 1).
Single nucleotide variants
To investigate the interplay of the data processing steps
with respect to SNV detection, 25 variants were introduced
into the data sets at allelic ratios ranging from 3e50%. Using
the standard informatics processing pipeline with GATK as the
variant caller, we were unable to identify a single variant below
6% allele frequency (Figure 4A). Even at allele frequencies
higher than this 6%, 100% sensitivity could not be achieved,
despite manual confirmation of the data in the bam file used
for variant calling. Using IGV to review the mapped data, we
were able to manually adjust the allele frequency threshold for
consensus variant visualization, which also confirmed the
presence of each of these artificial mutations. False-positive
calls were also made where the allele frequency was 100%,
but the total read depth was only a few reads.
446
Figure 4 SNV detection comparing AbsoluteVar (circles) to
GATK (triangles) at sites with (A) only two alleles present and (B)
at sites with two non-reference alleles present. Using Absolu-
teVar 100% sensitivity is achieved at allele frequencies >4%,
compared to <100% sensitivity when using GATK.
A major deficit in GATK was the inability to set input pa-
rameters to control the minimum depth of coverage or an
allelic fraction for variant calling. To address this issue, we
developed a novel algorithm, AbsoluteVar (see Materials and
methods), which provided the ability to define our thresholds.
Utilizing this tool in place of GATK, we analyzed the artificial
SNV bam files with variant detection cutoff parameters
specified at 4% allele frequency for any base and a minimum
depth of 100 reads, counting only bases with a quality score
>21. With this variant caller, we achieved 100% sensitivity
and specificity down to a 5% allele burden (Figure 4A). As we
approached the specified 4% cutoff, we saw a small
decrease in sensitivity that resulted from the somewhat
variable number of reads mapping from repeat analysis to
analysis. Since the mapping step is based upon random
processes of matching seed sequences to the reference
genome, we saw small variations in the number of reads
mapping in each repeat analysis that resulted in artificial
variant allelic burdens, which ranged from 3.95% to slightly
>4.0%, and were therefore not called with the 4% hard
cutoff. Pushing the variant calling threshold below 4% began
resulting in false-positive variant calls, and an average error
rate of 1% for SNVs was calculated across the panel.
R. Daber et al.
Using GATK and AbsoluteVar, we also investigated the
ability to detect a rare event where a normally polymorphic
site in the genome contained a germline homozygous variant
in addition to a somatic variant at a low allelic burden. This
created an artificial situation where the data set contained
two nucleotide changes at one location compared to the
reference genome. For example, in one test case, the artifi-
cial data presented with a germline homozygous state of CC
compared to the reference genome, GG, in addition to a
somatic variant, T. Comparing GATK to AbsoluteVar, we
were unable to detect the T variant call below the 7% allele
frequency with GATK, but were able to identify the variant at
all allelic burdens >4% using AbsoluteVar (Figure 4B).
Artifacts due to variant position in amplicon reads
The read data used for validation was generated using our
amplicon panels. As a result of using this technology, nearly
every read for a given loci contained identical start and stop
positions. Informatic variant callers identified the genomic
coordinate and variant allele present, but did not specify
where in an amplicon the variant was located.
During validation, we identified an indel at a position fall-
ing at the end of one amplicon and the middle of another. As
a result of the position, this change was identified as both an
SNV and an insertion (Figure 5). In the first data set, the
analytical pipeline identified a missense change in TET2
resulting in a T>C. In this data set, the variant was present at
the last base in all of the sequencing reads for the amplicon.
In the overlapping amplicon, the data demonstrated a 2-bp
insertion of CC at the same location, which resulted in a
frameshift mutation. Both data sets were concordant for a C
at the first variant call site; however, the absent sampling of
the adjacent nucleotide with the first amplicon resulted in a
false variant call.
Discussion
Next generation sequencing methods have the ability to
detect a wide array of mutation types, which is mostly limited
by the ability to properly process the raw data generated by
the sequencer. Through the creation of artificial variant files
with known mutation positions and allelic frequencies, we
were able to directly follow next generation sequencing reads
as they progressed through the bioinformatic analysis pipe-
line. This revealed the limitations of our initial pipeline and
prompted changes in how large and small indels are identi-
fied, and illustrated the need for the development of addi-
tional tools to achieve higher mutation detection sensitivity.
Initial limitations of indel detection arose from the inability
to map amplicon reads that contained these mutations. Large
insertion and deletion containing reads were unable to be
mapped with early versions of alignment software, while
recent updates have overcome those issues by extending the
range of mapping quality values assigned to each mapped
read. The initial limit of w35 bp was dramatically increased to
w90 bp with this update. This critical step presents the
community with a “clinical” informatics dilemma, since the first
step in processing the data takes raw sequences and maps
them to a specific location in the genome; however, the
premise is to find the mismatch in the patient’s genome that
NGS validation and artificial data sets 447

Figure 5 Raw data visualized in IGV for a variant captured by two different amplicons covering that locus. In the top section, the single
nucleotide change (T>C) is shown at the last base of the amplicon. A longer amplicon in the lower panel confirms the presence of a C at
that genomic position; however, it demonstrates that the mutation was not a single nucleotide event, but rather a two-base insertion.

explains the abnormal phenotype. It is therefore critical to
understand the limitations of the alignment tool used, along
with the specific parameters utilized with that tool.
This critical limitation is further highlighted in the pipeline
data quality filtering steps, where medium and large insertion
containing reads are filtered out of the data set even after
successful mapping. To overcome the limitations introduced
by this filtering step, while preserving the crucial ability to
filter out low-quality data and therefore detect low-level
SNVs, we moved large indel detection to the preequality
filtered bam file. This approach dramatically improved indel
detection without an increase in false-positives, since the
common sequencing artifacts with the Illumina sequencing
technology are single nucleotide changes. It is unclear how
these parameters would need to be set for other technolo-
gies that have different common sequencing artifacts.
When a large insertion or deletion was found in both read
pairs, neither read was able to map, which made detection
nearly impossible. The discrete nature of amplicon generated
data with identical ends makes it possible to “count” the
448
number of reads captured for each amplicon in the unpro-
cessed data. Utilizing this knowledge, we developed a tool
called GarbagePicker, which determined the number of
reads falling into both unmapped and mapped portions and
allowed identification of amplicons that showed abnormal
mapping patterns. Upon manual review of these “flagged”
amplicons, we were able to identify very large insertions in all
of the artificial data sets tested. This approach would not
work with NGS data generated by using a random sheering-
based library prep, but is very powerful for amplicon-based
assays.
Single nucleotide variant detection also proved to be a
challenge with our artificial data sets, since the major variant
calling tools were designed to identify heterozygous or ho-
mozygous changes in low read-depth data sets (i.e.,
constitutional changes). The inability to reliably identify vari-
ants with low allele frequencies prompted us to develop a
simple tool that utilized base counts at every position to
generate a variant call file. With this approach, we created a
program called AbsoluteVar, which enabled us to define
allele frequency, allele depth, and allele quality score cutoffs
to capture all genomic locations where a non-reference allele
was present to meet the specified criteria. This tool also
allowed us to identify sites with multiple non-reference alleles
present, even at low allele frequencies for those bases.
Validating the bioinformatics process of a clinical NGS
test is something that can be achieved without a significant
cost to the laboratory by using artificial data sets. It also al-
lows the laboratory to test mutation detection across many
different genomic loci, and with a range of mutation types
that are impossible to obtain from biological samples. This
approach may also be a useful mechanism of proficiency
testing to ensure high-quality mutation detection capabilities
and to define a laboratory’s limits of detection due to the
processing pipeline alone. Ultimate validation must still
include analysis of real samples. For many labs, however,
samples used for assay validation are only characterized by
a handful of loci, making false-negatives in novel regions
arising from inadequate data processing a real problem.
In conclusion, next generation sequencing is a powerful
approach that is poised to alter the paradigms in the practice
of all fields of medicine. The technologies are incredibly
robust and continue to improve. Even in the presence of
immature data processing pipelines, countless discoveries
and an unprecedented number of diagnoses have been
made. As the field of bioinformatics matures and begins
producing clinical-grade solutions, it is critical to validate the
informatics portion of an NGS assay to define the actual
limits of variant detection as it relates to single nucleotide
changes, insertions, and deletions before clinical imple-
mentation. In addition to initial validation of the informatics
pipeline, swapping data sets among various laboratories can
be a robust tool for quality assurance and proficiency testing.
Supplementary data
Supplementary data related to this article can be found at
http://dx.doi.org/10.1016/j.cancergen.2013.11.005.
R. Daber et al.
References
1. McCourt CM, McArt DG, Mills K, et al. Validation of next gen-
eration sequencing technologies in comparison to current diag-
nostic gold standards for BRAF, EGFR and KRAS mutational
analysis. PLoS One 2013;8:e69604.
2. Ross JS, Ali SM, Wang K, et al. Comprehensive genomic
profiling of epithelial ovarian cancer by next generation
sequencing-based diagnostic assay reveals new routes to tar-
geted therapies. Gynecol Oncol 2013;130:554e559.
3. Shanks ME, Downes SM, Copley RR, et al. Next-generation
sequencing (NGS) as a diagnostic tool for retinal degeneration
reveals a much higher detection rate in early-onset disease. Eur
J Hum Genet 2013;21:274e280.
4. Weiss MM, Van der Zwaag B, Jongbloed JD, et al. Best practice
guidelines for the use of next-generation sequencing applica-
tions in genome diagnostics: a national collaborative study of
Dutch genome diagnostic laboratories. Hum Mutat 2013;34:
1313e1321.
5. Wong LJ. Challenges of bringing next generation sequencing
technologies to clinical molecular diagnostic laboratories. Neu-
rotherapeutics 2013;10:262e272.
6. Fernald GH, Capriotti E, Daneshjou R, et al. Bioinformatics
challenges for personalized medicine. Bioinformatics 2011;27:
1741e1748.
7. Singh RR, Patel KP, Routbort MJ, et al. Clinical validation of a
next-generation sequencing screen for mutational hotspots in 46
cancer-related genes. J Mol Diagn 2013;15:607e622.
8. Tsiatis AC, Norris-Kirby A, Rich RG, et al. Comparison of Sanger
sequencing, pyrosequencing, and melting curve analysis for the
detection of KRAS mutations: diagnostic and clinical implica-
tions. J Mol Diagn 2010;12:425e432.
9. Yeo ZX, Chan M, Yap YS, et al. Improving indel detection
specificity of the Ion Torrent PGM benchtop sequencer. PLoS
One 2012;7:e45798.
10. Grimm D, Hagmann J, Koenig D, et al. Accurate indel prediction
using paired-end short reads. BMC Genomics 2013;14:132.
11. Kong Y. Btrim: a fast, lightweight adapter and quality trimming
program for next-generation sequencing technologies. Geno-
mics 2011;98:152e153.
12. Li H, Handsaker B, Wysoker A, et al. The Sequence Align-
ment/Map format and SAMtools. Bioinformatics 2009;25:
2078e2079.
13. Ramirez-Gonzalez RH, Bonnal R, Caccamo M, et al. Bio-sam-
tools: Ruby bindings for SAMtools, a library for accessing BAM
files containing high-throughput sequence alignments. Source
Code Biol Med 2012;7:6.
14. Quinlan AR, Hall IM. BEDTools: a flexible suite of utilities for
comparing genomic features. Bioinformatics 2010;26:841e842.
15. Danecek P, Auton A, Abecasis G, et al. The variant call format
and VCFtools. Bioinformatics 2011;27:2156e2158.
16. Cingolani P, Platts A, Wang Ie L, et al. A program for annotating
and predicting the effects of single nucleotide polymorphisms,
SnpEff: SNPs in the genome of Drosophila melanogaster strain
w1118; iso-2; iso-3. Fly (Austin) 2012;6:80e92.
17. Wang K, Li M, Hakonarson HANNOVAR. functional annotation
of genetic variants from high-throughput sequencing data.
Nucleic Acids Res 2010;38:e164.
18. McKenna A, Hanna M, Banks E, et al. The Genome Analysis
Toolkit: a MapReduce framework for analyzing next-generation
DNA sequencing data. Genome Res 2010;20:1297e1303.
19. DePristo MA, Banks E, Poplin R, et al. A framework for variation
discovery and genotyping using next-generation DNA
sequencing data. Nat Genet 2011;43:491e498.