Article

pubs.acs.org/ac

Bootstrap Approach To Compare the Slopes of Two Calibrations

When Few Standards Are Available
Graciela Estévez-Pérez,† Jose M. Andrade,*,‡ and Rand R. Wilcox§
†
Department of Mathematics and ‡Group of Applied Analytical Chemistry, Department of Analytical Chemistry, University of A
Coruña, Campus da Zapateira, 15071 A Coruña, Galicia, Spain
§
Department of Psychology, University of Southern California, Los Angeles, California 90089, United States
*
S Supporting Information

ABSTRACT: Comparing the slopes of aqueous-based and

standard addition calibration procedures is almost a daily task
in analytical laboratories. As usual protocols imply very few
standards, sound statistical inference and conclusions are hard
to obtain for current classical tests (e.g., the t-test), which may
greatly affect decision-making. Thus, there is a need for robust
statistics that are not distorted by small samples of
experimental values obtained from analytical studies. Several
promising alternatives based on bootstrapping are studied in
this paper under the typical constraints common in laboratory
work. The impact of number of standards, homoscedasticity or
heteroscedasticity, three variance patterns, and three error
distributions on least-squares fits were considered (in total,
144 simulation scenarios). The Student’s t-test is the most valuable procedure when the normality assumption is true and
homoscedasticity is present, although it can be highly affected by outliers. A wild bootstrap method leads to average rejection
percentages that are closer to the nominal level in almost every situation, and it is recommended for laboratories working with a
small number of standards. Finally, it was seen that the Theil−Sen percentile bootstrap statistic is very robust but its rejection
percentages depart from the nominal ones (<5%), so its use is not recommended when the number of standards is very small.
Finally, a tutorial and free software are given to encourage analytical laboratories to apply bootstrap principles to compare the
slopes of two calibration lines.

A nalytical laboratories must decide daily whether a series of

sample aliquots can be quantified appropriately by
“traditional” standardization [external standard calibration,
direct calibration (DC), or aqueous calibrations are common
synonyms]. This is addressed usually after preliminary trials
where the slope of the (first order) regression line obtained for
a traditional standardization framework is compared against
that derived from a series of additions of a standard to a set of
sample aliquots (SAM, standard addition method). Should their
slopes be comparable, the SAM method can be avoided, and so
the laboratory workload will be simplified, the overall costs
reduced, and the laboratory turnaround improved.
The general framework for straight-line standardization and
comparison of the slopes of two regression lines were
reviewed1−3 recently, with special emphasis on common
laboratory practices and some misconceptions.4 How to
correctly evaluate the statistical equality of the residual
variances of the regression models constituted a difficulty.
A literature survey found that this issue has surprisingly not
been considered broadly during the last years in the analytical
chemistry literature,4 despite its importance in daily laboratory
work worldwide. As an example, the ANCOVA approach
(analysis of covariance, which circumvents elegantly some
difficulties when the F-test is considered to compare the
variances of two regression straight lines before Student’s t-test
is calculated), is not generally known or, at least, is seldom
applied.4 However, it is a routine method in biology,5
psychology, and many other scientific disciplines, although
most applications focus on study of the regression coef-
ficients.6,7 Indeed, outside the analytical chemistry field, several
studies dealt with this topic and related aspects; some were
reviewed elsewhere.4 Unfortunately, in many scientific fields it
is hard or even impossible (or too costly) to prepare numerous
specimens to deploy good statistical calibrations. Therefore,
applying classical statistics to draw conclusions is not without
concern.
Of particular relevance for the present paper is the work from
Ng and Wilcox,8 which extends a previous study where
bootstrap was presented as a reliable alternative to other
approaches.9 In the former, the effects of nonnormality and
heteroscedasticity were studied when testing the hypothesis
that regression lines associated with two independent groups
Received: October 28, 2015
Accepted: January 11, 2016

© XXXX American Chemical Society A DOI: 10.1021/acs.analchem.5b04004

Anal. Chem. XXXX, XXX, XXX−XXX
Analytical Chemistry
have the same slopes. Problems associated with Student’s t-test
were underlined, and promising alternatives based on boot-
strapping studied. However, their simulations did not consider
less than 20 standards, a value that may be almost impracticable
for many analysts, not to mention the huge difficulties and high
cost that may arise in the industrial arena to undertake such
prolonged calibration times [e.g., in gas chromatography to
analyze PAHs (polycyclic aromatic hydrocarbons), more than
30 h of instrumental time would be required, just to get the
calibration data].
Bootstrap methods are statistical techniques that do not
assume a probability distribution function for the population;
they provide a nonparametric approach to testing hypotheses
that deals effectively with nonnormal distributions and
heteroscedasticity. In the event sampling is from a normal
distribution, bootstrap methods perform nearly as well as
standard methods that assume homoscedasticity. The practical
advantage of bootstrap methods is that they continue to give
accurate results for a much broader range of situations.10 In
essence, bootstrap methods perform simulations on the
observed data to determine confidence intervals or the
distribution of some appropriate test statistic. Conceptually,
the data are resampled with replacement many times in order
to generate empirical estimates of the statistics and use them to
make inferences about the populations. We will not present its
basics here, although two user-oriented references are highly
advisable.11,12 Here, we will only mention that bootstrapping is
particularly useful when (i) the theoretical distribution of a
statistic of interest is complicated or unknown (bootstrapping is
distribution-independent) and when (ii) the sample size is
insufficient to get a straightforward and sound statistical
inference (this situation is very common in analytical chemistry,
and therefore, it is required to estimate statistics that are not
distorted by the specific values derived from a given study).
Bootstrap (resampling) methods are gaining interest within
analytical chemistry and have already been proposed to
compare analytical procedures 13 and ordinates of two
regression lines;12 to calculate the probability of correct
assignments in KNN classification;14 to improve variable
selection for manufacturing processes, combined with permu-
tation tests and PLS-VIP (partial least-squares variable
importance in projection);15,16 to calculate sample-specific
prediction intervals;17 and to evaluate the reliability of
classifications when evaluating counterfeit banknotes,18 just to
cite some recent examples.
The aim of the present study was 2-fold. First, we aim to
evaluate the behavior of nonparametric approaches discussed
by Ng and Wilcox8 in typical situations that emerge in analytical
laboratories, specifically a low number of standards, which, in a
way, might be considered as an “extreme situation” in statistical
terms. In particular, we are interested in evaluating the most
promising options of that previous study, namely, the wild
bootstrap quasi t-statistic and the Theil−Sen percentile
bootstrap (a procedure related to the well-known non-
parametric Theil’s method to calculate a regression line based
on the median). Second, a pragmatic approach is presented so
that laboratories can apply the most satisfactory methods
derived from this study to their own data sets.
As a step forward, the methods recommended by Ng and
Wilcox8 will be checked under nonnormality (in particular,
simulating the presence of an undetected outlier, as may
happen in laboratories) and heteroscedasticity contexts, when
the sample sizes (number of calibrators) are small, on the order
B DOI: 10.1021/acs.analchem.5b04004
Article
of {3, 5, 7, 9}. The traditional Student’s t-test will also be
studied for benchmarking. It is also worth noting that in the
present study the explanatory variables will not be random,
which is the case in analytical laboratories. Please note that we
tried to keep a pedagogic approach, as far as possible. Some
statements might appear relatively trivial (it was considered
important to reinforce them) whereas the most complex ones
are not presented in detail (instead, interested readers are
referred to seminal papers).
The design of the study and the simulations undergone will
be described in the Experimental Section. Results gathered
from them will be discussed in the section Results and
Discussion, and finally, practical examples will be presented
under Worked Examples. A tutorial on how to use the free
software is included in Supporting Information.
■
EXPERIMENTAL SECTION
Here we explain how the experiments were designed. The
ANCOVA model constitutes a convenient way to compare two
regression lines in a more general context than the classical
Student’s t-test. A simple introduction was given elsewhere (ref
4 and other references cited therein). In brief, ANCOVA
combines several calibrations (standardizations) simultaneously
in a unique multiple regression model (like the one in eq 1)
and it allows for testing different issues about its coefficients;
for instance, if the slopes of the regression lines are comparable
(with the alternative hypothesis that the regression lines were
not derived from populations with equal slopes).
Throughout this paper, data sets were generated from the
regression model depicted in eq 1:
Yi = β0 + β1xi + β2Gi + β3xiGi + τ(xi)εi (1)
with i = 1, ..., n. βj (j = 0, ..., 3) are the unknown coefficients
(calculated by a least-squares fit), xi is the explanatory variable
(e.g., concentration), Gi is a dummy variable for two groups (Gi
takes a value of 0 or 1 for each corresponding calibration), τ
(variance pattern) is a function of xi used to model
heteroscedasticity, and εi is the error term. The term τ is
used in eq 1 to associate different variances with the error (ε)
term (for instance, to make it dependent on the x values). Note
that the regression model is Yi = β0 + β1xi + τ(xi)εi when Gi = 0
and Yi = (β0 + β2) + (β1 + β3)xi + τ(xi)εi when Gi = 1 (i.e.,
when each calibration is considered separately). To assess
whether the slopes of the groups (calibration lines) differ, the
hypothesis given by eq 2 is tested:
for H0 , β3 = 0; for H1 , β3 ≠ 0 (2)
Interest is in analyzing, through a Monte Carlo simulation
study, the performance of several methods to test eq 2 when
the normality and homoscedasticity assumptions are violated.
Unlike Ng and Wilcox,8 the values of the explanatory variable
are fixed by the researcher, and two typical situations for the
explanatory variable were considered. In situation 1 (S1), the
explanatory variable takes values on the same interval in both
calibrations (noted by 1 and 2), a common, pragmatic working
design in laboratories; that is, [I1, L1] = [I2, L2] = [0, 5]. Here
n1, n2 ∈ {3, 5, 7, 9}. When n1 = n2 (n = number of standards),
the values of the explanatory variable x are equal; otherwise,
they are different. In situation 2 (S2), the explanatory variables
take different values on different intervals [I1, L1] = [0, 5]; [I2,
L2] = [0, 10]. Here n1, n2 ∈ {3, 5, 7, 9}.
Anal. Chem. XXXX, XXX, XXX−XXX
Analytical Chemistry Article

To simulate heteroscedasticity, we focused on two scenarios.
First, within-groups heteroscedasticity (WG) occurs (this
means that the variance of ε depends only on x) without
differences between the groups (BG homoscedasticity). This
corresponds to common experience in, for example, spectro-
As a final note here, the statistics to be studied are T
(Student’s t), Qt (HC4-based quasi-t test), WB (HC4-based
wild bootstrap quasi-t test), MP (modified percentile boot-
strap), and TS (Theil−Sen percentile bootstrap). Technical,
full descriptions of these statistics were given previously,8 and
interested readers are kindly forwarded to that seminal work.

■
scopic laboratories, where currently standard additions method
(SAM) and direct method (DM) standardizations exhibit
similar standard errors of the fits. A second scenario considers RESULTS AND DISCUSSION
that heteroscedasticity occurs due to both WG and different In the following, the simulations will be resumed and discussed
variances among the standardizations (i.e., BG heteroscedas- separately for each scenario. First, a scenario where the
ticity, meaning that the variance of ε depends on G). This variability depends on x (but not on the calibration; i.e., WG)
might correspond to a situation where the added analyte and will be explored for the two situations S1 and S2, the three
the sample-inherent one do not hold the same behavior. In our types of residual variances, and three types of error distribution.
opinion, this should not occur in laboratories because it would Scenario 1, Situation 1: Within-Group Heteroscedas-
denote a suboptimal analytical procedure, but it is worth ticity and Between-Group Homoscedasticity, Same
considering. Calibration Ranges. This corresponds to a common design
Three variance patterns (VP) for τ(xi) were considered. in laboratories (termed scenario 1 in the Experimental
They are thought to represent typical laboratory conditions and Section): sample sizes and values of the explanatory variables
are depicted in Table S1 (Supporting Information), along with are equal in both calibrations.
a comparison with the variance patterns employed in the Table 1 reveals that whenever homoscedasticity occurs
original study.8 Numerical values in the mathematical (VP1) and the residuals follow a normal distribution, the best
expressions given in Table S1 for the variance patterns, τ(xi),
are a consequence of applying the original functions presented Table 1. Comparison of Statistics under Variance Pattern 1
in ref 8 to the range of experimental values in x (in this study, and Three Types of Error Distribution
without loss of generality, [0, 5] or [0, 10]). normal error +
Variance patterns 1, 2, and 3 represent three disparate normal error skewed errors outlier
conditions. In VP1, the residuals of the regression fits follow type I type I type I
normal homoscedastic distributions regardless of x. In VP2, the error error error
method p-value rates p-value rates p-value rates
residuals are normal but their variance increases when the
explanatory variable departs from the central value (to both n=3
higher and lower values). This corresponds to a common T 0.5026 5.40 0.4710 4.40 0.5474 7.60
situation. Indeed, it resembles the typical Hotelling’s confidence Qt 0.5579 3.30 0.5285 2.60 0.5889 0.00
region around the calibration line. Finally, VP3 “opposes” VP2, WB 0.5231 3.10 0.4905 2.70 0.5184 5.70
as the variance decreases when x values move away from the MP 0.7105 0.00 0.6905 0.00 0.6916 0.00
central value. The latter two situations are visualized in the TS 0.5451 0.00 0.5802 0.00 0.6287 0.00
n=5
graphical insets of Table S1.
T 0.5016 5.50 0.4775 4.80 0.5726 2.90
With regard to the error distribution, for each variance
Qt 0.5395 5.00 0.5235 2.90 0.6327 2.00
pattern, error terms (εi) were generated randomly from the so-
WB 0.5032 5.10 0.4592 7.30 0.6117 1.90
called g-and-h distribution, which is used currently to generate
MP 0.5584 0.00 0.5294 0.00 0.6204 0.00
models with different distributions. Three distributions were
TS 0.5810 0.90 0.5851 1.40 0.5919 0.80
considered for εi in this work: standard normal (g = 0, h = 0),
n=7
asymmetric light-tailed (g = 0.5, h = 0), and standard normal (g
T 0.5158 4.50 0.4917 4.20 0.5627 2.30
= 0, h = 0) with an outlier. It is acknowledged that in a perfect Qt 0.5373 4.50 0.5218 1.80 0.6360 0.90
situation the latter situation should not occur; however, it is WB 0.5181 4.40 0.4868 5.70 0.6072 0.80
also true that routine work in stressing circumstances, MP 0.5232 1.80 0.4985 1.30 0.5883 0.70
combined with a very low number of standards, might well TS 0.5601 2.00 0.5732 1.10 0.5963 1.20
make it difficult to correctly detect outliers. n=9
The two situations for the explanatory variable (S1 and S2) T 0.5079 5.00 0.4909 3.00 0.5725 1.60
with various sample sizes {3, 5, 7, 9} were combined with all Qt 0.5271 4.80 0.5110 1.30 0.6443 0.40
three variance patterns (τ) for each of the three εi distributions. WB 0.5075 4.50 0.4751 5.80 0.6280 0.50
Hence, a total of 144 simulation scenarios are considered. The MP 0.5087 5.30 0.4826 4.50 0.5959 0.90
probability of a type I error was based on 1000 replications and TS 0.5569 2.20 0.5860 1.30 0.5773 0.70
was estimated with α̂ , the proportion of p values less than or
equal to 0.05 (at the α = 0.05 level) (proportion of times that statistic is the traditional Student’s t-test, as expected. In our
H0 is rejected under H0). context, “best” means that the p-value and the percentage of
The studies were carried out by considering two rejections approach the nominal values (i.e., 0.5 and 5%,
experimental laboratory setups, typical of DM and SAM respectively). This was so regardless of the sample size.
comparisons. In setup A, the regression lines are identical. Nevertheless, WB yielded very good results. When the
For instance, without loss of generality, β0 = 0.15, β1 = 0.1, and distribution of the residuals is asymmetric, T, WB, and MP had
β2 = β3 = 0. In setup B, the regression lines have the same p-values close to the nominal ones (although MP only with the
slopes but different intercepts. For instance, without loss of largest number of standards). When an outlier is considered,
generality, β0 = 0.15, β1 = 0.1, β2 = −0.15, and β3 = 0. the most robust statistic was TS, although it is not the most
C DOI: 10.1021/acs.analchem.5b04004
Anal. Chem. XXXX, XXX, XXX−XXX
Analytical Chemistry Article

Table 2. Comparison of Statistics under Within-Group Heteroscedasticity Conditions (Variance Patterns 2 and 3) and Three
Types of Error Distribution
variance pattern 2 (3)
normal error skewed errors normal error + outlier
method p-value type I error rates p-value type I error rates p-value type I error rates
n=3
T 0.376 (0.616) 12.5 (2.7) 0.334 (0.581) 9.7 (2.5) 0.573 (0.685) 4.2 (0.4)
Qt 0.429 (0.667) 8.6 (1.9) 0.392 (0.632) 6.7 (1.7) 0.621 (0.729) 2.5 (0.4)
WB 0.492 (0.535) 1.3 (3.5) 0.453 (0.532) 2.8 (3.4) 0.646 (0.594) 0.0 (2.8)
MP 0.646 (0.766) 0.0 (0.0) 0.627 (0.741) 0.0 (0.0) 0.737 (0.786) 0.0 (0.0)
TS 0.533 (0.600) 0.0 (0.0) 0.531 (0.601) 0.0 (0.0) 0.606 (0.643) 0.0 (0.0)
n=5
T 0.395 (0.559) 12.9 (2.1) 0.396 (0.544) 10.1 (1.6) 0.532 (0.641) 4.3 (1.2)
Qt 0.481 (0.559) 6.4 (3.2) 0.489 (0.570) 3.2 (0.8) 0.633 (0.659) 1.9 (1.7)
WB 0.472 (0.486) 5.3 (6.6) 0.457 (0.482) 4.8 (5.0) 0.661 (0.629) 0.6 (0.8)
MP 0.490 (0.585) 0.0 (0.0) 0.489 (0.570) 0.0 (0.0) 0.620 (0.656) 0.0 (0.0)
TS 0.546 (0.599) 1.1 (0.7) 0.583 (0.606) 0.7 (0.5) 0.581 (0.617) 1.1 (0.6)
n=7
T 0.409 (0.565) 11.9 (1.9) 0.401 (0.539) 9.7 (2.7) 0.528 (0.610) 3.8 (1.1)
Qt 0.494 (0.551) 7.0 (2.5) 0.485 (0.535) 3.9 (2.1) 0.642 (0.636) 0.6 (1.2)
WB 0.487 (0.508) 4.3 (4.0) 0.469 (0.468) 6.3 (4.5) 0.633 (0.601) 0.2 (1.4)
MP 0.466 (0.544) 6.3 (0.9) 0.448 (0.514) 7.4 (1.5) 0.592 (0.600) 1.3 (1.0)
TS 0.547 (0.586) 3.4 (1.0) 0.545 (0.586) 1.6 (0.9) 0.562 (0.582) 1.6 (0.6)
n=9
T 0.419 (0.558) 10.8 (2.4) 0.409 (0.535) 9.6 (2.3) 0.541 (0.591) 3.5 (1.1)
Qt 0.501 (0.532) 6.8 (4.2) 0.483 (0.526) 3.2 (1.5) 0.659 (0.619) 0.2 (0.6)
WB 0.491 (0.501) 4.8 (5.2) 0.468 (0.488) 6.7 (6.2) 0.650 (0.602) 0.1 (1.5)
MP 0.467 (0.521) 7.5 (2.8) 0.453 (0.505) 7.7 (3.1) 0.604 (0.578) 1.1 (1.7)
TS 0.542 (0.590) 3.2 (1.0) 0.551 (0.573) 2.4 (1.6) 0.564 (0.573) 1.5 (0.5)

Figure 1. Relative performance of statistics, illustrated through averaged p-values calculated considering the three distributions of errors when sample
sizes and values of the explanatory variable are equal in both calibrations.

precise because it yields p-values not very close to the nominal

one. Therefore, if homoscedasticity occurs, the classical
D DOI: 10.1021/acs.analchem.5b04004
Anal. Chem. XXXX, XXX, XXX−XXX
Analytical Chemistry
Figure 2. Relative performance of statistics: average type I error probabilities considering the three error distributions when different types of WG
heteroscedasticity (variance patterns: VP1, blue; VP2, red; and VP3, green) and samples sizes are considered (“reject” stands for rejection).
Student’s t-test can be used regardless of the error distribution,
which is a highly advantageous result given the ubiquitous and
widespread use of this test.
On the contrary, when heteroscedasticity occursin
particular, when the variance increases with x (VP2)Table
2 shows that the t-test leads clearly to higher percentages of
rejection than the nominal level (for both normal and
asymmetric distributions). Qt and MP worked satisfactorily
only when the number of standards is at least 5. The best test
statistics were WB and TS (in particular, the former). When an
outlier is present, TS is still the most robust option (although
with p-values > 0.55) because the p-values and/or percentages
of rejections remain quite stable (e.g., compare columns 2nd
and 8th). The other statistics decrease the percentage of
rejections and, a bit surprinsingly, the T statistic is the best.
Table 2 presents the results for heteroscedasticity associated
with a reduction of the variance of residues when the
explanatory variable departs from the central value (VP3). In
general the p-values clearly overpass the nominal one, both for
normal and asymmetric errors, but for the WB statistic. When
the fewest standards are considered (n = 3), results are really
poor for T, Qt, and MP. When an outlier is present, the other
tests improve their behavior but they do not outperform it.
Therefore, WB should be the choice here.
An interesting conclusion from the simulations above is that
the behavior of the statistics depends much more on the type of
heteroscedasticity than on the distribution of errors (in which
cases the behaviors were quite homogeneous). Thus, it is
E DOI: 10.1021/acs.analchem.5b04004
Article
possible to calculate an “average behavior” of the statistics for
each variance pattern considering the three error distributions.
This is presented in Figure 1 and in Figure S1 (Supporting
Information). They show that, roughly, the T statistic has very
good behavior only with homoscedastic data (VP1). With
heteroscedasticity (VP2 and VP3), WB is the best option
because despite the fact that the other statistics improve when
the number of standards increase, they did not outperform it.
In the same way, the situation where the explanatory variable
does not take the same values in the two calibrations, although
it does expand on the same range, was studied. It was observed
that T behaves very well under homoscedasticity conditions,
regardless of the number of standards and error distribution.
However, the statistic degrades under heteroscedasticity as the
p-values are lower and higher than the nominal one (50%) for
VP2 (increasing variance) and VP3 (decreasing variance),
respectively. Also, T is very sensitive to the existence of a
unique outlier, which makes the p-values increase remarkably.
The average p-values for each statistic, when VPs and error
distributions and number of standards (sample size) are
considered, can be seen in Figure S2 (Supporting Information).
Type I error probability for the WB situation is close to the
nominal level (5%, when 95% confidence level is considered)
for both normal and asymmetric errors, regardless of the type
of heteroscedasticity and number of standards. It is, no doubt,
the best statistic under these conditions. Qt and MP vary quite
a lot depending on sample size, sometimes with good results.
Finally, TS seems the most robust statistic, even though the
Anal. Chem. XXXX, XXX, XXX−XXX
Analytical Chemistry
average p-values exceed systematically the nominal one (=0.5,
or 50%, Figure S2). Figure 2 depicts the average percentage of
rejections (average type I error estimated for the three error
distributions, nominal value = 5%).
A final remark is in order here, because in our studies TS was
far less satisfactory than for Ng and Wilcox.8 Further
simulations with more standards were considered (n = 10,
20, 40, 60) and the statistic yielded average results close to the
nominal ones, comparable to those of WB. In some cases TS
overpassed WB, but we feel that TS is not suitable for typical
laboratory situations where very few standards are used.
Scenario 1, Situation 2: Within-Group Heteroscedas-
ticity and Between-Group Homoscedasticity, Different
Calibration Ranges. In the Experimental Section , situation
S2 was described as that when the explicative variable has
different ranges in both calibrations. Results were totally similar
to those above, so they are not presented here for brevity.
Scenario 2: Both Within-Group and Between-Group
Heteroscedasticity. Out of the many possibilities that might
be simulated, three rather “extreme” situations were considered:
normal error in group 1 plus asymmetric error in group 2;
normal error in group 1 plus normal error with an outlier in
group 2; and asymmetric error in group 1 plus normal error
with an outlier in group 2.
Figure S3 shows that the most useful statistics were T, WB,
and TS. Their behaviors agree largely with those in the two
previous scenarios. T is the most valuable statistic when
homoscedasticity is present, although it becomes very affected
by an outlier. In contrast, WB leads to average rejection
percentages that are very close to the nominal ones in almost
every situation. Hence this would be the recommended choice
for laboratories working with a small number of standards.
Finally, TS is very robust but its rejection percentages depart
from the nominal ones (<5%).
Statistical Power. In the analysis of a statistical test, it is
common practice to focus on type I error (α error, or
significance level); that is, the probability of rejecting the null
hypothesis when, indeed, it is true. Nevertheless, the probability
of a type II error (β), or the probability of accepting the null
hypothesis when it is false, should not be neglected. Evaluation
of this probability is made by studying the statistical power of
the test (1 − β), that is, the test’s ability to detect a specific
alternative hypothesis (the probability of correctly rejecting the
null hypothesis when it is false or a specified alternative is true).
Although the statistical power is also known as sensitivity, it
should not be confused with analytical sensitivity, which is the
slope of the regression line. Note that the higher the power, the
lower the type II error will be.
The two most satisfactory statistics according to the previous
simulations, T and WB, were subjected to a final analysis to
determine their statistical power. For this, a situation was
considered first where the true regression lines can differ just in
their slopes (β0 = 0.15, β1 = 0.1, β2 = 0.0) and normality of their
residuals and homoscedasticity are fulfilled by design. Then,
their power was evaluated as a function of both the number of
standards (sample size) and the values of β3. It is worth noting
that a “perfect” test should have power = 1 and that power
tends to 0.05 (type I error) when the null hypothesis (β3 = 0) is
true; otherwise the power should be evaluated for different
values of β3. In particular, for the test posed in eq 2, we
investigated whether the methods considered in this paper can
detect a difference between the two slopes (if it exists) by
rejecting the null hypothesis. Such ability in general depends on
F DOI: 10.1021/acs.analchem.5b04004
Article
sample sizes and β3 values. In effect, the more standards we
have and the larger the value of β3 is, the greater the statistical
power should be.
Theoretically, when homoscedasticity holds, T should be
more powerful than WB, ex ante. In effect, it was found that
both T and WB have very similar power, although for smaller
sample sizes, T approaches 1 faster. As expected, both statistics
clearly increase their power when the number of standards
increases. Hence, WB performed very satisfactorily, even when
compared with the optimal approach (under these conditions),
T.
When heteroscedasticity and nonnormal residuals were
considered [we selected increasing variances (VP2) and
asymmetric error distribution (g = 0.5, h = 0) because in our
experience this seems a common situation in laboratories], the
T statistic had very unsatisfactory results, as expected, and only
the WB statistic must be considered. However, its statistical
power converged to 1 at a slower pace than when
homoscedasticity and normal errors occurred: it was relatively
low for small values of β3 and/or for very small sample sizes
(n1= n2 = 3). However, when the sample size of a calibration is
equal or greater than 5, the statistical power increases to 1
quickly. This is a highly relevant result because, in analytical
laboratories, to prepare a series of five standards is a de facto
standard, and so this practice now has an additional benefit.
■ WORKED EXAMPLES
In Table S2 (Supporting Information), experimental data from
four real case studies are presented. They show how awkward
decision-making can be when classical tests are used with very
few data points and/or when the structural characteristics are
not carefully reviewed (which must be done before every
calculation).
Example 1 was intended to check whether an ionization
suppressor improved some spectrometric measurements in
flame atomic absorption spectrometry4 and represents a
measurement protocol where the standards are not replicated.
Examples 2−4 correspond to laboratory studies carried out
with a graphite furnace atomic spectrophotometer to assess the
need for SAM calibration when different instrumental setups
were considered. The standards were replicated three times
(replicates mean full preparation).
In the first example, heteroscedasticity cannot be checked
because of a lack of replicates. Although normality of the
residuals is not fulfilled (p-value for Shapiro−Wilks’ test =
0.04), T and WB procedures yield the same conclusion: the
slopes are comparable (p-values for T and WB were 0.25 and
0.51, respectively).
In example 3, homoscedasticity and normality are fulfilled
(nonsignificative tests at 5% level) and both procedures lead to
the same decision; that is, the slopes are comparable (p-values
for T and WB were 0.24 and 0.59, respectively).
In example 4, neither heteroscedasticity nor normality
occurs, and the T and WB tests showed that the slopes are
not different (p-values for T and WB were 0.44 and 0.62,
respectively).
Finally, in example 2, neither homoscedasticity nor normality
occurs (p-values for Shapiro−Wilks and Breusch−Pagan tests
not significant 95% confidence level), and the WB test accepts
that the slopes are equal, while the Student’s one rejects this (p-
values of 0.11 and 0.01, respectively).
These results clearly show that the behavior of the t-test
depends hugely on the particular data under study and that
Anal. Chem. XXXX, XXX, XXX−XXX
Analytical Chemistry
although it is quite reliable (according to the simulations
described earlier), it may lead to different conclusions than the
robust tests (in particular, the WB one). In particular, one
should be aware that if the data present heteroscedasticity, its
correct performance is by no means guaranteed.
These examples were used to develop the practical examples
displayed in the tutorial on practical use of the WB statistic
(including how to get and install the free software). The
tutorial can be found at the Supporting Information.
■
CONCLUSIONS
The slopes of two regression lines were compared in 144
simulation scenarios. Ranges of experimental predictors, sample
sizes, three variance patterns, and three error distributions were
combined to assess how they influence the performance of
Student’s t-test and four nonparametric tests.
A first interesting conclusion from the simulations is that
behavior of the statistics depends much more on type of
heteroscedasticity than on distribution of errors (for which the
behaviors of different setups were very similar). A second fact
derived from the simulations is that the statistics behaved the
same, regardless of whether the explicative variable takes the
same values in the two calibrations or not.
When homoscedasticity occurs, the classical Student’s t-test
can be used regardless of error distribution, which is a highly
advantageous result given the ubiquitous and widespread use of
this test. However, when heteroscedasticity occurs (no matter
whether the variance increases or decreases), the t-test leads
clearly to higher percentages of rejection than the nominal one
(for both normal and asymmetric error distributions). The best
statistics were WB and TS, in particular WB, although when an
outlier was present TS was the most robust option (although it
had p-values >0.55, and so its use is discouraged for a low
number of standards).
In the most complex situation, heteroscedasticity was
induced both within and between calibrations. Once more,
the most useful statistics were T, WB, and TS. Their behaviors
agree largely with those in the previous scenarios. T is the most
valuable statistic when homoscedasticity is present, although it
becomes strongly affected by an outlier. On the contrary, WB
lead to average rejection percentages that are very close to the
nominal ones in almost every situation. TS is very robust, but
its rejection percentages depart from the nominal ones.
As a final conclusion, the wild bootstrap (WB) statistic seems
a very convenient and useful choice to compare regression
straight lines in laboratories working with a small number of
standards. In addition, its power is comparable to Student’s t-
test even in cases where optimal homoscedasticy occurs, and it
does not need previous statistical tests to assess normality or
homoscedasticity.
■
ASSOCIATED CONTENT
*
S Supporting Information
The Supporting Information is available free of charge on the
ACS Publications website at DOI: 10.1021/acs.anal-
chem.5b04004.
Two tables, listing variance patterns associated with
residuals and practical examples to compare the behavior
of T and WB tests; three figures, showing relative
performance of statistics when sample sizes and values of
explanatory variable are equal in both calibrations and
average p-values and relative performance of statistics
G DOI: 10.1021/acs.analchem.5b04004
Article
when different types of WG heteroscedasticity are
considered; and an tutorial on use of the WB Statistic
(PDF)
■
AUTHOR INFORMATION
Corresponding Author
*E-mail andrade@udc.es; fax +34-981-167065.
Notes
The authors declare no competing financial interest.
■ ACKNOWLEDGMENTS
The Galician Government, “Xunta of Galicia”, is acknowledged
for its support to the QANAP group (Programa de
Consolidación y Estructuración de Unidades de Investigación
Competitiva, GRC2013-047). The financial support of the
Spanish Government (Ministerio de Economiá y Compet-
itividad) and Xunta de Galicia (research projects MTM2014-
52876-R and CN2012/130) is also acknowledged.
■
REFERENCES
(1) Ortiz, M. C.; Sánchez, S.; Sarabia, L. Quality of analytical
measurements: univariate regression. In Comprehensive Chemometrics:
Chemical and Biochemical Data Analysis, Vol. 1; Brown, S. D.; Tauler,
R.; Walczack, B., Eds.; Elsevier: Amsterdam, 2009; pp 127−169; DOI:
10.1016/B978-044452701-1.00091-0.
(2) Thompson, M.; Lowthian, P. J. Notes on Statistics and Data
Quality for Analytical Chemists; Imperial College Press: London, 2011.
(3) Andrade-Garda, J. M.; Carlosena-Zubieta, V.; Soto-Ferreiro, R.;
Terán-Baamonde, J.; Thompson, M. Classical Linear Regression by
the Least Squares Method. In Basic Chemometric Techniques in Atomic
Spectroscopy; Royal Society of Chemistry: London, 2013; Chapt. 2, pp
52−122; DOI: 10.1039/9781849739344-00052.
(4) Andrade, J. M.; Estévez-Pérez, G. Anal. Chim. Acta 2014, 838, 1−
12.
(5) Draper, N. R.; Smith, H. Applied Regression Analysis; John Wiley
& Sons: New York, 1998; DOI: 10.1002/9781118625590.
(6) Smithson, M. Frontiers in Psychology 2012, 3, No. 231.
(7) DeShon, R. P.; Alexander, R. A. Psychological methods 1996, 1 (3),
261−277.
(8) Ng, M.; Wilcox, R. R. Br. J. Math. Stat. Psych. 2010, 63, 319−340.
(9) Wilcox, R. R. Br. J. Math. Stat. Psych. 1997, 50, 309−317.
(10) Wilcox, R. Introduction to Robust Estimation and Hypothesis
Testing, 3rd ed.; Elsevier, New York, 2012.
(11) Mooney, C. Z.; Duval, R. D. Bootstrapping: A Nonparametric
Approach to Statistical Inference. Sage Publications: Newbury Park, CA,
1993.
(12) Wehrens, R.; Putter, H.; Buydens, L. M. C. Chemom. Intell. Lab.
Syst. 2000, 54, 35−52.
(13) Hartmann, C.; Smeyers-Verbeke, J.; Penninckx, W.; Vander
Heyden, Y.; Vankeerberghen, P.; Massart, D. L. Anal. Chem. 1995, 67,
4491−4499.
(14) Villa, J. L.; Boqué, R.; Ferré, J. Chemom. Intell. Lab. Syst. 2008,
94 (1), 51−59.
(15) Afanador, N. L.; Tran, T. N.; Buydens, L. M. C. Anal. Chim.
Acta 2013, 768, 49−56.
(16) Afanador, N. L.; Tran, T. N.; Buydens, L. M. C. Chemom. Intell.
Lab. Syst. 2014, 137, 162−172.
(17) Pereira, A. C.; Reis, M. S.; Saraiva, P. M.; Marques, J. C.
Chemom. Intell. Lab. Syst. 2011, 105 (1), 43−55.
(18) de Almeida, M. R.; Correa, D. N.; Rocha, W. F. C.; Scafi, F. J.
O.; Poppi, R. J. Microchem. J. 2013, 109, 170−177.
Anal. Chem. XXXX, XXX, XXX−XXX