Trends Trends in Analytical Chemistry, Vol. 25, No.

11, 2006

Problems with the ‘‘omics’’

Jackson O. Lay Jr., Sabine Borgmann, Rohana Liyanage, Charles L. Wilkins

‘‘omics’’ studies involve the measurement of large numbers of parameters, certain biological state of a cell or organ-
typically genes (genomics), proteins (proteomics), lipids (lipidomics) or ism. The development of modern instru-
metabolites (metabolomics). Values associated with each of the measured mental analytical methods in the
parameters are searched to find examples that correlate with biological twentieth century had a profound impact
endpoints, often disease or cancer. Although the number of parameters being upon biomarker research. Routine ana-
measured has increased dramatically with ‘‘omics’’ studies, the number of lytical measurement of chemical markers
biological and methodological replicates has not. As with comparable clas- from biological systems became possible,
sical biomedical studies, there exist limitations arising from whether or not and the search for chemical biomarkers of
the analytical methods are adequate for making the measurements needed diseases began in earnest. As new
and whether or not the measurements are implemented properly. In addit- chemical-measurement techniques were
ion, because of the large number of measurements and the limited number of developed, they each were applied, in turn,
test subjects, unique problems arise in ‘‘omics’’ studies involving statistics to the search for new disease-related bio-
and bias. Even though improvements in technology may well minimize markers. These were not yet ‘‘omics’’
measurement problems, the inherent difficulties associated with measuring studies because the number of markers in
so many parameters from a limited number of test subjects will remain. This each study could not be considered
review focuses on the four main problems with the ‘‘omics’’: bias, statistics, ‘‘omic’’ or ‘‘massive’’.
methodology, and method misuse. Although we give suggestions to minimize The primary issues with early
the impacts of these problems, some problems may not be solved unless the biomarker-discovery research were mea-
number of measurements is more consistent with the number of possible surement feasibility or difficulty. Mea-
biological replicates. surements were difficult and the biological
ª 2006 Elsevier Ltd. All rights reserved. matrix was complex. Difficulties included
mixture complexity, analyte polarity,
Keywords: Bias; Genomics; Lipidomics; Metabolomics; Methodology; ‘‘Omics’’;
sensitivity and disease specificity. Could
Proteomics; Statistics
the measurement be done? Sometimes
method validation and method suitability
1. Introduction also became issues. Measurements were
Jackson O. Lay Jr.,
feasible but sometimes difficult to imple-
Rohana Liyanage
Arkansas Statewide Mass The term ‘‘omic’’ is derived from the Latin ment or reproduce.
Spectrometry Facility, suffix ‘‘ome’’ meaning mass or many. In recent years, as analytical chemists
University of Arkansas, Thus ‘‘omics’’ studies are like other studies have developed methods believed suitable
Fayetteville, AR 72701, except that they involve a mass (large for the analysis of a very large number of
USA
number) of measurements per endpoint analytes from a single sample, it has
Sabine Borgmann, rather than one or a few. In the current become popular to attempt the measure-
Charles L. Wilkins* context, the measurements are of chemi- ment of many thousands of potential
Department of Chemistry and cal markers indicative of biological events. biomarkers rather than only a few.
Biochemistry, The original concept of using chemicals as This approach gives rise to the modern
University of Arkansas,
markers to monitor biological endpoints is ‘‘omics’’ experiments. Considering these
Fayetteville, AR 72701,
USA difficult to attribute to any one individual. approaches generally decreased the num-
Considering that Hippocrates proposed the ber of replicates possible while massively
presence of ‘‘fruity odors’’ in human increasing the number of parameters tes-
breath as a diagnostic marker for diabetes ted, it is not clear why this approach was so
in the fourth century B.C., it could be widely accepted – except perhaps simply
argued that the concept of chemical bio- because it became possible. For these new
*
markers has been around for over two approaches, the statistical issues may be
Corresponding author.
millennia. problematic, even if the measurement
Tel.: +1 479 575 3160;
Fax: +1 479 575 4049; The term ‘‘biomarker’’ is used here for techniques themselves are perfected.
E-mail: cwilkins@uark.edu molecules that are characteristic of a Today, investigators consider searching

1046 0165-9936/$ - see front matter ª 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.trac.2006.10.007
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006
entire classes of possible markers within an organism,
with the aim of finding biomarkers or their combinations,
from which subtle changes or complex interactions can be
used to make difficult associations with biological events.
The goal of ‘‘omic’’ approaches is to acquire compre-
hensive, integrated understanding of biology by studying
all biological processes to identify the different players
(e.g., genes, RNA, proteins and metabolites) rather than
each of those individually. ‘‘Omic’’ approaches currently
attempt to address specific biological questions, often
without need for prior understanding of a biological
basis. With the development of current and future
technologies, ‘‘omic’’ research might aim to explain
more complex systemic questions and become a tool in
diagnostics and drug development. To date, we are still
far from putting these demanding goals into practice due
to the complexity of the task.
Fig. 1 simplifies the interaction of the main ‘‘omics’’
actors (genome, proteome, transcriptome and metabo-
lome) within the organism. Fig. 2 displays and simplifies
the quantitative dimensions of the different ‘‘omic’’ ele-
ments within the context of scale and time resolution.
When the targets are proteins, genes and metabolites,
the corresponding approaches are called proteomics,
genomics and metabolomics (or metabonomics). These
and other ‘‘omics’’ retain the analytical requirements
and problems associated with older or classical analytical
“ome”
genome
regulation
feedback regulation
transcriptome
regulation
proteome/metabolome
characteristics:
allosteric interaction
covalent modification
enzyme levels
linked metabolic pathways
compartmentalization
metabolic specialization of organs
resulting “phenome” (phenotype)
Figure 1. Complexity of the different ‘‘omic’’ levels is highly impacted by genetic predisposition, environmental factors and regulatory interac-
tions within and between the ‘‘omes’’.
Trends
approaches and add additional challenges due to mix-
ture complexity and the number of biomarkers measured
from a single sample. The resulting use of very large data
sets coupled with a search for biomarkers showing subtle
or complex relationships adds an additional layer of
complexity and introduces new problems. Many of these
problems can be solved, but this will require considerable
interaction between methods developers, users, statisti-
cians, and bioinformatics experts. The purpose of this
review is to present representative ‘‘problems with the
‘‘omics’’’’ with the aim of provoking a discussion
amongst these various groups and others.
The problems currently encountered with ‘‘omics’’
studies are reflected in the provocative title of a recent
article in The Scientist – ‘‘Gene Association Studies Typ-
ically Wrong’’ [1]. Although the title makes specific
reference to genomics, the problems encountered in
these studies have parallels in all the ‘‘omics’’, as the
author notes. After the fact, it has been possible to
demonstrate that the first published study linking genes
to diseases is often not supported by subsequent studies.
For example, 1992 research associating a 2–4-fold
increased risk of schizophrenia in persons carrying two
copies of a specific mutation [2] was not supported by
some 50 follow-up studies, which failed to confirm the
initial findings [3]. If this happened rarely, it would be
easy to attribute it to the users, rather than to a problem
parameters influencing the “ome”
person‘s genetic predisposition, environmental factors, interactions
person‘s genetic predisposition, environmental factors, interactions
regulation (activation, inactivation of e.g. proteins, expression capacity)
status of metabolism (catabolism, anabolism; rate of compound absorption,
distribution, metabolism, and excretion)
age, sex, ethnicity
stage of cell cycle
interactions
temperature
redox state
culture conditions
external stimuli (nutrition, life style, stress, pharmaceuticals, radiation, ..)
“normal” function vs. dysfunction (organism with good health vs.
disease)
cumulative effects of a multiplicity of genes, various signaling and
metabolic pathways
http://www.elsevier.com/locate/trac 1047
Trends
Figure 2. Complexity of the different ‘‘omic’’ levels with respect to quantitative dimensions and time-scales of processes within an organism
(time-scale for proteome adapted from [41]).
with the general approach. Unfortunately this is not
unusual and may even be the norm. Indeed, in one
evaluation of the probability of reproducing genomics
studies finding associations of genes with complex dis-
eases, a review of 370 studies suggested that, more often
than not, subsequent research does not confirm the
initial finding [4,5].
For the purposes of this article, the ‘‘problems with the
‘‘omics’’’’ are divided into four general categories: bias,
statistical, methodological, and, ‘‘fitness of use’’. Most of
the problems with one ‘‘omics’’ are common to all sim-
ilar ‘‘omics’’ applications. Examples are based on repre-
sentation in the literature and their utility for illustrating
a specific problem. Due to space limitations, only a few
examples can be presented here, but the problems are
general across all ‘‘omics’’.
2. Bias
One of the emerging difficulties with the ‘‘omics’’,
including the flawed genomics studies referred to above,
is the problem of bias. Ransohoff has defined bias as ‘‘the
systematic erroneous association of some characteristic
with a group in a way that distorts a comparison with
another group’’ [6]. Bias is not limited to genomics but
also occurs in proteomics and other ‘‘omics’’. This
problem is sufficiently severe in proteomics that one
1048 http://www.elsevier.com/locate/trac
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006
researcher noted that bias was now a threat to the
validity of cancer molecular biomarker research [6].
Some prominent results in cancer research have been
disputed or not reproduced, and bias has been increas-
ingly recognized as a significant cause of this problem.
Part of the overall problem has to do with the approach to
discovery using the ‘‘omics’’ and the high stakes involved
in the finding of new ‘‘omics’’-related biomarkers. ‘‘omic’’
measurements are usually made without a specific
hypothesis in mind. In this case, it may be somewhat
easier to introduce bias and then to essentially ‘‘find what
you are looking for’’ because these experiments are so
loosely defined at the start. With larger data sets, it may
be inherently easier to find a correlation irrespective of
real cause and effect. False positives probably outnumber
true positives for reasons outlined below. The need to
discover non-invasive biomarker tests for diagnosis of
diseases, where no current non-invasive tests are avail-
able, places considerable pressure on investigators to
publish work quickly because there is some potential to
save lives or to make profits.
Consider the example of ovarian cancer. For this dis-
ease, there is currently no non-invasive screening test
and the development of a new biomarker for patient
screening would be extremely important and useful.
Recently a ‘‘pattern-recognition’’ serum proteomics
approach was used to search for a marker for ovarian
cancer, which the authors reported had nearly 100%
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006
success at discriminating between women with and
without the disease [7,8]. On this basis, a manufacturer
began the process of developing a blood test kit for
patient screening. However, a number of other investi-
gators challenged these studies, based both on the
methodology used and also on whether or not the pro-
posed biomarkers were plausible from a biological point
of view. Doubts about the validity of the first outcomes
were sufficient for the FDA subsequently to intervene [9].
Some have proposed that the underlying problem in
these specific studies was bias [6].
Bias is difficult to address in the design of experiments,
in part because it is often unintentional. Although some
sources of bias are obvious, many are not. Unfortu-
nately, bias cannot be minimized by simply increasing
the sample size. Similarly, there is no relationship
between reproducibility and bias, nor is there a statistical
method that can solve problems caused by bias in study
design.
Bias can be minimized only by careful control and
planning of the experiments. This can include sample
blinding and randomization. Both are powerful methods
for reducing bias. Because bias can also be introduced by
instrumental drift, randomization of the analyses them-
selves (in order and in time) is also important.
While many of these controls are familiar to those
involved in method-validation studies or approvals by
regulatory agencies, they have not been universally
taught or applied, especially in the ‘‘omics’’. An impor-
tant tool for monitoring ‘‘omics’’ studies for bias (and
other problems) is the development of standards for
reporting how analyses are conducted. Microarray-
experiment guidelines, such as the Minimum Informa-
tion About a Microarray Experiment (MIAME), address
reporting of technical details [10]. Similar criteria for
reporting proteomics data are being developed [11].
3. Statistics
It has been suggested that there is a widespread mis-
conception amongst users of ‘‘omics’’ that large
numbers of measurements could somehow make up
for small numbers of samples [12]. However, the
reality is that the introduction of multiple measure-
ments in no way minimizes the need to use statisti-
cally appropriate numbers of samples. Replicates must
be adequate both with respect to ‘‘measurement pro-
cess’’ and ‘‘biology’’. Some have argued that biological
replicates are even more important than methodolog-
ical or technical replicates [12]. Even in well-controlled
systems, such as inbred mice, there is usually signifi-
cant biological variability. The minimum number of
replicates therefore has to be selected to satisfy statis-
tical criteria driven by:
Trends
 the reliability of the method;
 the background variability within an organism and
between organisms;
 the number of experimental states being investigated;
 the number of parameters being measured; and,
 even the likelihood that the hypothesis is a reasonable
one.
This last factor, sometimes called prior probability, is
often not considered, but it is extremely important.
Because of the large number of parameters being mea-
sured, and the typically limited number of biological
replicates, the probability (P) of false-positive associations
is very high. A major reason for this is because of the
propensity for declaring statistical significance based on
the value alone, particularly any P value below 0.05
[13]. However, the false-positive-report probability
(FPRP) depends not only on the P value but also on the
prior probability that the association between the ‘‘omic’’
variant and the disease is real and also the statistical
power of the test. One approach to understanding the role
of false-positive findings is to estimate the FPRP so as to
use it as a measure of significance. When the FPRP value
is high, positive associations should be given little weight.
Even though both false-negative and false-positive
finds detract from the quality of studies, much of the
reproducibility problem mentioned above can be attrib-
uted to false-positive findings. One estimate of the prob-
ability of false-positive findings in association between
genetic variants and disease places the probability as
high as 0.95 [14]. In this case, the false positives so
outnumber the true positives that the data are probably
not even useful for screening.
In the absence of bias, three factors can contribute to a
false-positive yet statistically significant findings [15].
One is the absolute magnitude of the P value. Another is
the statistical power of the method and the third is
something called ‘‘the fraction of tested hypotheses that
is true’’. Each of these contributes to increasing the
FPRP, and thus the likelihood that an apparently valid
prediction is false. It is not uncommon in ‘‘omics’’ studies
to set a value of a (type-1 error) at 0.05 and call any
association between the ‘‘omics’’-detected biomarker and
disease with a P value below a statistically significant.
Fig. 3 shows the association between prior probability
and false-positive probability for P just below a = 0.05
for four different statistical powers. The false-positive
probability can be minimized by reducing P substantially
when the prior probability is high. Increasing the
numbers of test and control samples can also reduce the
false-positive rate, but again only when there are high
prior probabilities. Increasing the sample size provides
only marginal benefit when the prior probability is low.
Clearly, if all of these factors are not taken into account,
the probability of a false positive can be very high even
with significant statistical power, as Fig. 3 shows. The
http://www.elsevier.com/locate/trac 1049
Trends
Figure 3. Effect of changes in prior probability and statistical power on a false-positive-report probability (FPRP) when a = 0.05. FPRP and P is at
or just below a. (Reprinted from [13]).
current common practice, corresponding to use of a
universal statistical criterion for statistical significance,
based on rejecting the null hypothesis at a value of a of
0.5, will not work across the range of prior probabilities
for true association, even using the maximum statistical
power of 1. Ultimately, for the testing of highly unlikely
hypotheses, even an infinitely large sample size cannot
by itself substantially reduce the FPRP [13].
The proposed FPRP approach requires a realistic esti-
mate of the prior probability, specification of necessary
odds ratios for specific applications, definition (in
advance) of criteria for an acceptable FPRP value, and
then, after a study, determination of whether the find-
ings are ‘‘noteworthy’’ based on pre-established and
presumably defensible criteria. Often there are difficulties
in establishing (or estimating) these values. A complete
mathematical treatment of FPRP is beyond the scope of
this work, but can be found in an article by Wacholder
et al. [13].
Other possible methods for reducing the number of
false positives range from Bonferroni corrections [16],
the use of other false-discovery-rate methods [17,18] to
a large reduction of a from 0.05 to 0.005 or 0.0005
[14].
Because there is no single generally accepted approach
to the statistical design of ‘‘omics’’ experiments, there is
clearly a problem with substantial numbers of irrepro-
ducible findings, false positives, and otherwise statisti-
cally unsound data. The best approach would seem to be
inclusion of a biostatistician in the design of ‘‘omics’’
studies from their outset. It is essential to report sufficient
experimental details, so that the statistical significance,
1050 http://www.elsevier.com/locate/trac
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006
including the likelihood of false positives, can be judged
by others after experiments are completed.
4. Methodology
To investigate the different processes within an organ-
ism, a tissue, a cell, or a cell compartment, there exist
arsenals of technologies. Table 1 provides a selection and
a brief evaluation of several current techniques. For
further reading, we recommend several reviews [19–25]
that emphasize and partly explain why ‘‘omics’’ do not
meet current needs and expectations. In another,
‘‘omics’’ is addressed in the context of understanding
disease and drug development [26]. The status of current
technologies used in genomics and proteomics, and
possible future approaches, is comprehensively reviewed
by Baak et al. [27].
Here, we focus on the challenges and the pitfalls facing
researchers in this field. A typical workflow of an ‘‘omic’’
approach consists of several steps:
(1) definition of starting conditions as well as analyti-
cal/biological questions and study design;
(2) sample preparation;
(3) separation of analytes (if necessary);
(4) analysis and quantification of analytes;
(5) validation;
(6) documentation; and,
(7) bioinformatics (chemometrics).
Table 2 summarizes some general pitfalls, issues to
consider and suggestions related to designing ‘‘omic’’
studies (and experiments, in general).
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006 Trends

Table 1. Selected current methodologies employed in genomics, transcriptomics, proteomics, and metabolomics (partially adapted/modified
from [24,27,42])

Methodologies Biomarker target Merits Limitations

employed in

Genomics
Karyotyping Structural changes in Analysis of whole chromosome Labor intensive
chromosomes
Fluorescence in situ Homologous sequences in Gene specific Limited number of probes per labeling
hybridization (FISH) chromosomes reaction
Comparative genome Deletions or amplifications Analysis of whole genome; array Resolution limited to 10 Mbp (for
hybridization (CGH) CGH available array CGH 0.5 Mbp)
Spectral karyotyping Structural changes in Analysis of whole genome Labor-intensive fluorescence labeling
(SKY) chromosomes
Microsatellite instability Stability of chromosomes, Allele specific Large numbers of microsatellites
(MSI) efficiency of DNA repair required
Single nucleotide Genomic DNA, Archived patient genomic DNA High-density SNP mapping required;
polymorphism (SNP) polymorphism can be used SNP maps not yet completed (for
current results, refer to http://
snp.cshl.org/) [43]

Transcriptomics
sRNA interference Small interfering RNA Gene specific; analysis of whole A lot of false-negative and false-
(RNAi) interaction with targeted genome positive results
mRNA molecules (RNA
silencing)
DNA microarray [44,45] mRNAs Established technique; Many Fresh tissue obligatory for cDNA
technology (‘‘gene- thousands of gene-specific isolation, closed transcription profiling
expression profiling’’) mRNAs; ÔsignatureÕ analysis; (pre-requisite: knowledge of the
high-throughput sequence of the genome studied);
reproducibility [46]
Serial analysis of gene RNAs Open transcription profiling; Expensive, labor intensive, qualitative
expression (SAGE) absolute quantitation of gene
expression; large SAGE database
available
Massively parallel Nucleic acid fragments Open transcription profiling; Expensive, labor intensive
signature sequencing convenient handling of complex
(MPSS) mixture of nucleic acid
fragments, throughput
Differential display Differentially expressed Open transcription profiling Expensive, labor intensive; false
genes positives and redundancies observed
Tissue microarray DNA, RNA, in situ Morphologic context; high Formalin-fixed samples
hybridization throughput; compatible to
archived samples

Proteomics [21,24]
Two-dimensional gels Complex protein mixtures Established method; dynamic
range of 102–104 for protein
expression; sensitivity (10 6);
post-translational modifications
(PTMs) evident; data
incorporated in current literature
and databases
Yeast two-hybrid Protein-ligand interaction Established basic technique;
automated assay design
available; fast; protein function
determined in vivo
Protein microarrays Complex protein mixtures Fast, simple and reproducible
[47,48] assay designs; high throughput
Systematic limitation to access very
large, very small, low-abundant [31],
and membrane-bound proteins; only
2000 spots per gel; requires
experienced personal; time-
consuming; limited automation [31]
potential
Only binding properties evaluated (no
biochemical function); false-positive or
false-negative results observed
(additional confirmation necessary); no
structural information; labor intensive
New technology; limited dynamic
range (62D gels); no standardized
formats for assay validation (e.g., cross-
reactivity) and data-processing limited
resources of antibodies and
recombinant proteins

(continued on next page)

http://www.elsevier.com/locate/trac 1051
Trends Trends in Analytical Chemistry, Vol. 25, No. 11, 2006

Table 1 (continued)

Methodologies Biomarker target Merits Limitations

employed in
Fluorescence
microscopy (FRET, FLIP,
FRAP, FLIM, BRET,
PRIM)
Mass spectrometry
(Peptide Mass
Fingerprinting)[49]
Chromatography
coupled with MS
(e.g., LC-MS [51])
Stable isotope methods
coupled with MS [52]
Surface-enhanced laser
desorption/ionization
(SELDI)
Tissue microarray
Nuclear magnetic
resonance (NMR)
Microfluidics [53] and
lab-on-a-chip[54]
Protein-ligand interaction
Enzymatic protein fragments
from a single protein.
Enzymatic protein fragments,
multiple proteins.
Pooled enzymatic fragments
from differentially labeled
targets.
Affinity-selected proteins
from complex mixtures
Protein (e.g.,
immunohistochemistry)
Proteins
Proteins
Spatial and time resolution; Labor intensive
in vivo applications
Established method; high Limited dynamic range, highest
15
specificity and sensitivity (10 ) specificity requires tandem MS (MS2)
High sensitivity, moderate to Complex separations, complex data,
high specificity. false identifications possible
Accurate quantification of Dynamic range; tagging required
differential expression;
Fast; small samples required; False positives; poor reproducibility
easy sample preparation; already
used for clinical applications
Morphologic context; high Formalin-fixed samples
throughput; compatible with
archived samples
Detailed 3D structural Poor sensitivity; low throughput
information
Small, portable; simple Experimental techniques
operation; high throughput; cost-
effective

Metabolomics [55–58]
Cell-based assays of
secreted metabolites
Metabolites from body
fluids (‘‘microdialysis’’
[59–61])
Liquid chromatography
(LC)-NMR [61]
Chromatographic
(LC, GC, CE [63,64])
approaches with MS,
FTMS [65,66]
Secreted metabolites Many established methods for
analysis of some metabolites
Metabolites in the Near real-time metabolic
extracellular space in living profiling; clinical microdialysis is
tissue already available
Metabolites High throughput, complete
sample recovery, little sample
preparation
Metabolites Higher sensitivity and throughput
than NMR
Metabolites High mass accuracy and
resolution; broad range of
analytes
Standardized assay and data-
processing format is missing; limited
validity of in vitro functional assays for
in vivo predictions
Invasive technique; dependent on
osmosis
Expensive; much lower sensitivity and
throughput than MS techniques
Expensive equipment; limited
throughput; possible analyte
interactions without chromatographic
introduction

The first step in the process includes proper design of
the study. From seven large studies aimed at predicting
prognosis of cancer patients by employing DNA-micro-
array techniques, Michiels and co-workers [28] found
that the results obtained in these studies depended on
how the patients were selected in the training set.
Because of problems with this aspect of study design, the
outcomes of these studies were biased.
We covered difficulties associated with statistical
problems in study design and the effects of bias in pre-
vious sections. After study design is complete, the
experimental processes can begin. Consider a proteomics
study as an example. Generating a precise, reproducible
representation of the proteome intrinsically demands
very accurate control of many parameters. These include
variables such as cell-culture conditions, sample prepa-
ration, sample handling, and even protein separation.
Unfortunately, many parameters may also affect either
protein expression or protein measurements. Effects on
the measured proteome are thus not limited to those
caused by the biomarkers in question. Many of these
proteomics-influencing parameters are difficult to
recognize or control.
Assuming that the experimental parameters gov-
erning proteome expression are well controlled, there is
still a need to measure it. Technically, it is challenging
to maintain the quantitative integrity of the measure-
ment of a complex sample during sample preparation,

1052 http://www.elsevier.com/locate/trac
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006 Trends

Table 2. Examples of pitfalls in ‘‘omic’’ approaches

Common pitfalls Issues to consider Suggestions

Definition of starting Number of samples is too Start with reasonable goals for a Develop sound experimental
conditions and low for number of features study; select approaches that are design [25]; analyze homogeneous
analytical question analyzed (Ôovertraining of able to meet those objectives groups of samples; perform large-
the initial learning setÕ) (e.g., select a reasonable number scale analysis on a large number of
of samples to be investigated) samples; use a hypothesis-driven
rather than a discovery-oriented
approach
Sample preparation Changing the ratio(s) of RNA and proteins can easily Perform adequate sampling (e.g.,
analyte(s) during sample degrade over time if the samples homogenous groups of samples);
preparation are not handled and stored optimize and standardize sample
properly handling and storage; select
sample types that are stable,
reliable, ethically inoffensive, and
repeatedly accessible
Separation of analytes (if Often incapable of Choose techniques with Use the ÔbestÕ techniques currently
necessary) resolving the whole appropriate resolving power and available for specific question of
‘‘ome’’ dynamic range for analytes of study
interest
Data analysis, Automated data analysis Avoid being ‘‘over-optimistic’’ Check raw data of positive ‘‘hits’’
quantification and might give false-positive about positive results for plausibility; validate computer
interpretation or false-negative results; algorithms for automated data
data are not quantitative handling
Validation, Done insufficiently Key step in proving the utility of Validate and document study in
documentation the study accordance with good laboratory
practice (GLP) guidelines
[21,67,68]; verify results by an
independent method and/or
independent sample set (repeated
random sampling); examine inter-
laboratory variation to ensure that
novel techniques have adequate
accuracy and precision; establish
recommendations for novel
techniques when they are ready to
be used by a broader scientific
community
Bioinformatics, Done insufficiently ‘‘Omic’’ data need to be Employ chemometric methods to
chemometrics [12] produced in a format that will facilitate the outcome of ‘‘omic’’
meet the requirements of large- studies; extract relevant
scale modeling tools (e.g., information by optimizing
definition, quantity, function, experiments, processing data, and
localization, dynamics, and calibrating, organizing, and
interaction of the component of performing quality controls of the
interest) [41] ‘‘omic’’ studies; (see also [12])

separation, and quantification. Indeed, for reasons of
magnitude, the measurement is difficult and has little
precedent in other traditional instrumental analytical
fields. First, most instrumental analytical measure-
ments do not involve large numbers of analytes. Typ-
ical instrumental methodologies are designed to
measure relatively small numbers of analytes quanti-
tatively, often under 50. Second, usual analytical
approaches require either identity or quantity be
determined, but not both. Analysis of pollutants, toxi-
cants, or typical analytical test subjects normally
requires that the analyte has characteristics defined by
specificity criteria that assure, with some defined level
of certainty, that signal arises from a pre-determined
target. In other words, the method employs selection
criteria to allow quantification of predetermined
targets. However, with proteomics, the target identities
and their quantities are determined simultaneously and
they may not correspond to previously expected com-
pounds for which standards and controls have been
developed. Moreover, with proteomics, all of this is
done at a level of complexity not attempted before.
Given these considerations, it is now difficult to defend
the initial optimism regarding the ability to quantify
and to identify proteins from the proteome simulta-
neously with any degree of accuracy. This may require

http://www.elsevier.com/locate/trac 1053
Trends
repeated analysis of multiple samples derived from the
control and the experimental sources.
In addition to the problems associated with the size
and the scope involved in ‘‘omics’’ measurements,
there are intrinsic challenges associated with each new
sample. These challenges are related to the great
complexity of the sample and the complex or incom-
pletely known biochemistry of the disease state (see
also Figs. 1 and 2). Focusing on the proteome, a
sound selection or ‘‘snapshots’’ (several proteomes) of
a given sample needs to be collected for the results to
be biologically and/or clinically relevant. Proteins are
quite diverse in their structures, turnover rates, com-
partmentalization, dynamic range of abundance (in
humans cells, up to 107–108, and, in plasma, up to
1012) [29,30], and function. More than 200 covalent
post-translational modifications of proteins have been
found and identified. There is little doubt that others
remain undetected. Taking the dynamic range of sev-
eral proteomics methods into account (102–104), it is
clearly not even possible to sample proteins of widely
varying copy numbers using a single proteomics
measurement or approach. The fact is that current
technologies are simply inadequate for covering a
whole proteome, even if statistics and sampling were
not an issue. Although progress is being made on low-
abundance-protein detection (e.g., in new 2D protein
maps, as pointed out by Ahmed and Rice in a recent
review [31]), we are nevertheless still far from even
the possibility of producing a complete proteome (or
metabolome maps) for any organism for reasons of
both complexity and dynamic range. It might well be
that low-abundant proteins are those of greatest
importance in regulatory-based diseases, but they are,
of course, also the most difficult to analyze.
Another aspect worth considering is the variation of
biomarkers in different compartments (e.g., individual
cells, sub-populations of cells, tissues, organs, and
organisms). There is a lack of understanding of the
baseline or ‘‘normal’’ value for biological variation and
for variation within different compartments, so baseline
criteria are missing in many modern experimental
designs. The importance of baseline variability can be
illustrated using a study by Colman-Lerner et al. using
pathway-induced yeasts [32]. Their work concentrated
on cell-to-cell variability. About half the variation they
observed was related to differences measured in associ-
ation with the cell-cycle stage of the individual cells [32].
Such large differences confound the detection of other
changes, especially small ones. Small individual ‘‘omics’’
differences have importance in disease states and need to
be measured, yet they exist superimposed on a variation
not associated with the target disease. Church stated in
an editorial about the Personal Genome Project [33] that
genomic variation between ‘‘normal’’ and cancer cells
might consist of about 10 6 per base pair [34] but only
1054 http://www.elsevier.com/locate/trac
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006
some of these differences cause the cancer. Other differ-
ences are incidental.
Such baseline, inter-individual, and intra-individual
variability not only confounds studies of group re-
sponse, but ultimately may reveal clues about individ-
ual responses if the measurements can eventually be
made. Revealing more insights into intra-species and
inter-species variations is not only interesting from the
scientific point of view, but might also promote the
movement towards ‘‘personalized’’ biomarker identifi-
cation and drug treatment (briefly, tailor-made medi-
cine). A recent review about pharmacogenomics related
to acute lymphoblastic leukemia revealed insights into
its usefulness for cancer therapy [35]. Clayton and co-
workers introduced a rat model with an interesting
pharmaco-metabolomic approach for phenotyping and
‘‘personalized’’ drug treatment [36]. They propose to
monitor the metabolites of pre-dose biofluids of animals
in order to take metabolic and other individual pre-
dispositions into account, and to use the results to
predict inter-subject variations in effects of drug
metabolism.
5. Fitness of use
For disease states showing large changes in significant
numbers of abundant biomarkers, successful ‘‘omics’’
studies should still be possible, even with current tech-
nological limitations. Unfortunately, problems with ‘‘fit-
ness of use’’ are often encountered in the ‘‘omics’’ and
have limited the successes (e.g., in 2000, Rhodes and co-
workers reported that immunohistochemical studies of
tumors in different laboratories were not useful due to
poorly defined analytical techniques [37]). Reports are
accumulating that the reproducibility and significance of
reported proteomic biomarkers are questionable [6,38–
40]. Standards are now being established requiring
reporting of all aspects of proteomic and genomic studies
(beyond what would be required in other fields) because
of prior fitness-of-use and validation problems. The fail-
ure of many studies probably correlates with failure to
consider the analytical need to define quality standards,
including method validation and standardization. Vali-
dation of the data obtained by ‘‘omic’’ studies is a highly
critical feature and requires much more effort than
proof-of-principle experiments. Reproducibility needs to
be estimated by using adequately independent samples of
appropriate size. Hypotheses derived by ‘‘omic’’
approaches need to be verified by additional, functional
studies in order to differentiate false positives from bio-
markers that are clinically relevant. However, if the ratio
of false to true positives is very large, it is unlikely that
even this approach will work. The ‘‘omics’’ should not be
used as a discovery tool producing false and true posi-
tives for subsequent screening unless estimates of the
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006
FPRP suggest the true positives can be detected amongst
the false ones.
It is important to demonstrate intra-laboratory and
inter-laboratory reproducibility and standardization of
new ‘‘omic’’ tools as well as to maintain high quality
standards using blind control and test samples. Valida-
tion includes the selection of ‘‘hit’’ criteria. Which degree
of differences in a comparative analysis reflects a bio-
logically relevant meaning? How likely is it that a dif-
ferentially expressed protein is correctly identified? Could
it be that it is a modified version of the identified protein?
6. Conclusion
Putting theory into practice using ‘‘omic’’ approaches is
difficult. It is not feasible to measure an entire proteome,
genome or metabolome. However, that is not the root of
most problems with the ‘‘omics’’. Indeed, the very notion
that measuring every possible parameter is desirable is
probably one of the biggest misconceptions about the
‘‘omics’’. ‘‘Omic’’-wide measurements may violate sta-
tistical norms and have little precedent with respect to
feasibility in analytical chemistry literature.
Fig. 4 suggests one approach. An initial sampling
strategy is proposed and reviewed for statistical sound-
ness, based on what is known or can be assumed about
the problem. One obvious option for improving the
‘‘omics’’ is to decrease the number of parameters that are
being measured in each sample and to include more
replicates. This is certainly compatible with the current
limitations of measurement technology.
ideal workflow of “omic” approaches
analytical/biological questions
proposed sampling & starting conditions
biostatistics
- interpret data with significant uncertainty and false
+
development of study design
documentation
sample preparation
separation of analytes (if necessary)
analysis and quantification of analytes
method validation
- terization by Mass Spectrometry’’. The authors thank
+
biostatistics & chemometrics
+ -
conclusion biological validation
+ -
Figure 4. Schematic of an idealized work flow for an ‘‘omics’’
experiment.
Trends
If the number of biological and methodological repli-
cates appears appropriate, the study is more fully
designed with respect to sample preparation, analyte
separation, quantitation, and validation approaches.
To avoid bias, all criteria need to be developed before
experiments begin. Method validation on a test set or
standards is critical. If the analytical methodology can-
not be validated, then the study needs to be redesigned.
Once the initial analytical and statistical preconditions
have been met, large-scale measurements begin.
After completion, the data needs to be submitted to
statistical and chemometric treatment. This evaluation
may uncover flaws or limitations that require rethinking
any presumptions about the original biological question
or methodology.
If the statistical and chemometric treatments suggest a
meaningful correlation between the ‘‘omics’’ and the
biological endpoint, then a biological validation step
should be sought to minimize the chances of eliminating
false positives.
During each step, the entire process should be doc-
umented. Studies that do not reach a biologically
validated final conclusion may merit publication if
sufficient details can be documented to aid in the
design of new studies. Studies that appear validated
also need complete documentation, especially for eval-
uation if instances arise where different studies need to
be closely compared.
Integrating validation of analytical methods and
statistical criteria into all phases of the ‘‘omics’’ will
not be easy or inexpensive, but ultimately it is likely to
produce more reliable data. It appears that the scien-
tific community is willing to put significant resources
into generating large data sets by ‘‘omics’’ approaches,
despite the aforementioned difficulties. It remains to be
seen whether it will be possible to learn from prior
‘‘problems with the ‘‘omics’’’’ and ‘‘do it right’’ so as to
leave a legacy of reliable and useable data for future
researchers or whether production of difficult-to-
positives will continue.
Acknowledgement
Part of this work was financially supported by the
Arkansas Biosciences Institute grant ‘‘Protein Charac-
Douglas D. Rhoads and Ina Radtke for helpful discus-
sions.
References
[1] J. Lucentini, The Scientist 18 (2004) 20.
[2] M.A. Crocq, R. Mant, P. Asherson, J. Williams, Y. Hode,
A. Mayerova, D. Collier, L. Lannfelt, P. Sokoloff, J.C. Schwartz,
http://www.elsevier.com/locate/trac 1055
Trends
M. Gill, J.P. Macher, P. McGuffin, M.J. Owen, J. Med. Genet. 29
(1992) 858.
[3] E.G. Jonsson, R. Kaiser, J. Brockmoller, V.L. Nimgaonkar,
M.A. Crocq, Psychiatr. Genet. 14 (2004) 9.
[4] K.E. Lohmueller, C.L. Pearce, M. Pike, E.S. Lander, J.N. Hirsch-
horn, Nat. Genet. 33 (2003) 177.
[5] J.P.A. Ioannidis, E.E. Ntzani, T.A. Trikalinos, D.G. Contopoulos-
Ioannidis, Nat. Genet. 29 (2001) 306.
[6] D.F. Ransohoff, Nat. Rev. Cancer 5 (2005) 142.
[7] E. Petricoin III, A. Ardekani, B. Hitt, P. Levine, V. Fusaro,
S. Steinberg, G. Mills, C. Simone, D. Fishman, E. Kohn, Lancet 359
(2002) 572.
[8] W. Zhu, X. Wang, Y. Ma, M. Rao, J. Glimm, J.S. Kovach, Proc.
Natl. Acad. Sci. USA 100 (2003) 14666.
[9] L.A. Wagner, J. Natl. Canc. Inst. 96 (2004) 500.
[10] A. Brazma, P. Hingamp, J. Quackenbush, G. Sherlock,
P. Spellman, C. Stoeckert, J. Aach, W. Ansorge, C.A. Ball,
H.C. Causton, T. Gaasterland, P. Glenisson, F.C.P. Holstege,
I.F. Kim, V. Markowitz, J.C. Matese, H. Parkinson, A. Robinson,
U. Sarkans, S. Schulze-Kremer, J. Stewart, R. Taylor, J. Vilo,
M. Vingron, Nat. Genet. 29 (2001) 365.
[11] S. Carr, R. Aebersold, M. Baldwin, A. Burlingame, K. Clauser,
A. Nesvizhskii, Mol. Cell. Proteom. 3 (2004) 351.
[12] D.M. Rocke, Semin. Cell Dev. Biol. 15 (2004) 703.
[13] S. Wacholder, S. Chanock, G.-C. Maontserrat, L. El Ghormli,
N. Rothman, J. Natl. Canc. Inst. 96 (2004) 434.
[14] H.M. Colhoun, P.M. McKeigue, G. Davey Smith, Lancet 361
(2003) 865.
[15] J.A. Sterne, G. Davey Smith, Br. Med. J. 322 (2001) 226.
[16] N. Risch, K. Merikangas, Science (Washington, DC) 273 (1996)
1516.
[17] Y. Benjamini, Y. Hochberg, J. R. Stat. Soc., Ser. B 57 (1995) 289.
[18] C. Sabatti, S. Service, N. Freimer, Genetics 164 (2003) 829.
[19] C. Gerner, Comb. Chem. High Throughput Screen. 7 (2004) 1.
[20] J.L. Harry, M.R. Wilkins, B.R. Herbert, N.H. Packer, A.A. Gooley,
K.L. Williams, Electrophoresis 21 (2000) 1071.
[21] F. Vitzthum, F. Behrens, N.L. Anderson, J.H. Shaw, J. Proteome
Res. 4 (2005) 1086.
[22] G.L.G. Miklos, R. Maleszka, Proteomics 1 (2001) 30.
[23] M.F. Lopez, Electrophoresis 21 (2000) 1082.
[24] P.R. Graves, T.A.J. Haystead, Microbiol. Mol. Biol. Rev. 66 (2002)
39.
[25] W.S. Hancock, S.L. Wu, P. Shieh, Proteomics 2 (2002) 352.
[26] J.A. Bilello, Curr. Mol. Med. 5 (2005) 39.
[27] J.P.A. Baak, E.A.M. Janssen, K. Soreide, R. Heikkilae, Ann. Oncol.
16 (2005) 30.
[28] S. Michiels, S. Koscielny, C. Hill, Lancet 365 (2005) 488.
[29] N.L. Anderson, N.G. Anderson, Electrophoresis 19 (1998) 1853.
[30] G.L. Corthals, V.C. Wasinger, D.F. Hochstrasser, J.C. Sanchez,
Electrophoresis 21 (2000) 1104.
[31] N. Ahmed, G.E. Rice, J. Chromatogr., B 815 (2005) 39.
[32] A. Colman-Lerner, A. Gordon, E. Serra, T. Chin, O. Resnekov,
D. Endy, C.G. Pesce, R. Brent, Nature (London) 437 (2005) 699.
[33] G.M. Church, Mol. Syst. Biol. (2005) doi:10.1038/msb4100040.
[34] Z.H. Wang, D. Shen, D.W. Parsons, A. Bardelli, J. Sager, S. Szabo,
J. Ptak, N. Silliman, B.A. Peters, M.S. van der Heijden,
G. Parmigiani, H. Yan, T.L. Wang, G. Riggins, S.M. Powell,
J.K.V. Willson, S. Markowitz, K.W. Kinzler, B. Vogelstein,
V.E. Velculescu, Science (Washington, DC) 304 (2004) 1164.
1056 http://www.elsevier.com/locate/trac
Trends in Analytical Chemistry, Vol. 25, No. 11, 2006
[35] M.H. Cheok, W.E. Evans, Natl. Rev. Cancer 6 (2006) 117.
[36] T.A. Clayton, J.C. Lindon, O. Cloarec, H. Antti, C. Charuel,
G. Hanton, J.P. Provost, J.L. Le Net, D. Baker, R.J. Walley,
J.R. Everett, J.K. Nicholson, Nature (London) 440 (2006)
1073.
[37] A. Rhodes, B. Jasani, A.J. Balaton, K.D. Miller, J. Clin. Pathol. 53
(2000) 292.
[38] K.R. Coombes, J.R.S. Morris, J.H. Hu, S.R. Edmonson,
K.A. Baggerly, Nat. Biotechnol. 23 (2005) 291.
[39] E.P. Diamandis, Expert Rev. Mol. Diagn. 4 (2004) 575.
[40] K.A. Baggerly, J.S. Morris, S.R. Edmonson, K.R. Coombes, J. Natl.
Cancer Inst. 97 (2005) 307.
[41] S. Souchelnytskyi, Proteomics 5 (2005) 4123.
[42] A.Q. Dove, Nat. Biotechnol. 17 (1999) 233.
[43] R. Sachidanandam, D. Weissman, S.C. Schmidt, J.M. Kakol,
L.D. Stein, G. Marth, S. Sherry, J.C. Mullikin, B.J. Mortimore,
D.L. Willey, S.E. Hunt, C.G. Cole, P.C. Coggill, C.M. Rice,
Z.M. Ning, J. Rogers, D.R. Bentley, P.Y. Kwok, E.R. Mardis,
R.T. Yeh, B. Schultz, L. Cook, R. Davenport, M. Dante, L. Fulton,
L. Hillier, R.H. Waterston, J.D. McPherson, B. Gilman,
S. Schaffner, W.J. Van Etten, D. Reich, J. Higgins, M.J. Daly,
B. Blumenstiel, J. Baldwin, N.S. Stange-Thomann, M.C. Zody,
L. Linton, E.S. Lander, D. Altshuler, Nature (London) 409 (2001)
928.
[44] M. Schena, D. Shalon, R.W. Davis, P.O. Brown, Science (Wash-
ington, DC) 270 (1995) 467.
[45] M. Schena, R.A. Heller, T.P. Theriault, K. Konrad, E. Lachenmeier,
R.W. Davis, Trends Biotechnol. 16 (1998) 301.
[46] E. Marshal, Science (Washington, DC) 306 (2004) 630.
[47] Y. Hu, M. Uttamchandani, S.Q. Yao, Comb. Chem. High
Throughput Screen. 9 (2006) 203.
[48] D. Stoll, M.F. Templin, J. Bachmann, T.O. Joos, Curr. Opin. Drug
Discov. Dev. 8 (2005) 239.
[49] J. Reinders, U. Lewandrowski, J. Moebius, Y. Wagner,
A. Sickmann, Proteomics 4 (2004) 3686.
[51] Y. Shi, R. Xiang, C. Horvath, J.A. Wilkins, J. Chromatogr., A 1053
(2004) 27.
[52] L.V. Schneider, M.R. Hall, Drug Discov. Today 10 (2005) 353.
[53] W.C. Sung, H. Makamba, S.H. Chen, Electrophoresis 26 (2005)
1783.
[54] R.B.M. Schasfoort, Expert Rev. Proteomics 1 (2004) 123.
[55] O. Fiehn, Compar. Funct. Genom. 2 (2001) 155.
[56] W.B. Dunn, D.I. Ellis, Trends Anal. Chem. 24 (2005) 285.
[57] J. van der Greef, A.K. Smilde, J. Chemometr. 19 (2005) 376.
[58] J.L. Griffin, Philos. Trans. R. Soc., Ser. B 361 (2006) 147.
[59] N. Plock, C. Kloft, Eur. J. Pharm. Sci. 25 (2005) 1.
[60] M.J. Cano-Cebrian, T. Zornoza, A. Polache, L. Granero, Curr. Drug
Metab. 6 (2005) 83.
[61] F. Magkos, L.S. Sidossis, Curr. Opin. Clin. Nutr. Metab. Care 8
(2005) 501.
[63] W. Kolch, C. Neususs, M. Peizing, H. Mischak, Mass Spectrom.
Rev. 24 (2005) 959.
[64] D.C. Simpson, R.D. Smith, Electrophoresis 26 (2005) 1291.
[65] J.J. Jones, S. Mariccor, A.B. Batoy, C.L. Wilkins, Comput. Biol.
Chem. 29 (2005) 294.
[66] J.J. Jones, S. Borgmann, C.L. Wilkins, R.M. OÕBrien, Anal. Chem.
78 (2006) 3062.
[67] J.P.A. Baak, J. Pathol. 198 (2002) 277.
[68] P.A. Hall, J.J. Going, Histopathology 35 (1999) 489.