See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/304581212

Refined Latinized Stratified Sampling: A robust sequential sample size extension

methodology for high-dimensional Latin hypercube and stratified designs

Article · June 2016

CITATIONS READS

2 122

1 author:

Michael Shields
Johns Hopkins University
52 PUBLICATIONS   257 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Higher-order methods in nonlinear stochastic dynamics View project

Quantifying the effects of small data uncertainty in computational modeling View project

All content following this page was uploaded by Michael Shields on 29 June 2016.

The user has requested enhancement of the downloaded file.

 1 Refined Latinized Stratified Sampling: A robust sequential sample size
2 extension methodology for high-dimensional Latin hypercube and stratified
3 designs

4 Michael D. Shields1∗
5
1 Dept. of Civil Eng., Johns Hopkins University
6
∗ Corresponding Author: michael.shields@jhu.edu
7

8 Abstract

A robust sequential sampling method, Refined Latinized Stratified Sampling, for simulation-based
uncertainty quantification and reliability analysis is proposed. The method combines the benefits of
the two leading approaches, Hierarchical Latin Hypercube Sampling (HLHS) and Refined Stratified
Sampling, to produce a method that significantly reduces the variance of statistical estimators for
very high-dimensional problems. The method works by hierarchically creating sample designs
that are both Latin and stratified. The intermediate sample designs are then produced using the
Refined Stratified Sampling method. This causes statistical estimates to converge at rates that are
equal to or better than HLHS while affording maximal flexibility in sample size extension (one-
at-a-time or n-at-a-time sampling are possible) that does not exist in HLHS - which grows the
sample size exponentially. The properties of the method are highlighted for several very high-
dimensional problems demonstrating the method has the distinct benefit of rapid convergence for
transformations of all kinds.
9 Keywords: uncertainty quantification, Monte Carlo simulation, stratified sampling, Latin
10 hypercube sampling, sample size extension

11 1. Introduction

12 Simulation-based uncertainty quantification and reliability analysis are becoming increasingly

13 widespread in computational engineering analysis and design. With this growth, there has been
14 a strong demand to develop efficient sampling methods that enable probabilistic analysis of larger
15 and more complex engineering systems including those that involve very high-dimensional random
16 vectors [1] and random fields [2]. Additionally, these methods are important for the development of

Preprint submitted to International Journal for Uncertainty Quantification June 29, 2016
17 surrogate models such as those that employ non-intrusive generalized polynomial chaos expansions
18 [3, 4] and machine learning [5]. One desirable feature of these methods is adaptivity and flexi-
19 bility that can only be achieved through a truly sequential sample design. This flexibility allows
20 computational analysts to stop sampling when desired confidence has been achieved (or resources
21 depleted) and extend the sample when it has not.
22 Sample size extension has historically come at the expense of simulation efficiency as variance
23 reduction techniques often place constraints on the sample design that do not allow (or at least
24 restrict) sample size extension. While quasi-Monte Carlo methods (e.g. using deterministic low-
25 discrepancy sequences) [6] have been widely employed for truly sequential (one-at-a-time to n-at-a-
26 time) sample size extension, very few methods employing random samples with variance reduction
27 have been used (importance sampling is one example but it has been mostly used for probability
28 of failure analysis [7]). Recently, Shields et al. [8] proposed the Refined Stratified Sampling (RSS)
29 methodology (and the closely related Targeted Random Sampling for reliability analysis [9]) which
30 is one of only a few truly sequential sample size extension methods with variance reduction. Its
31 usefulness, however, was shown to be limited primarily to low-dimensional applications. Over
32 the past decade, another family of sample size extension methods - Hierarchical Latin Hypercube
33 Sampling (HLHS) [10, 11, 12] - has gained widespread popularity because it enables extension of
34 Latin hypercube sample (LHS) designs, which have strong variance reduction properties [13, 14, 15].
35 The primary limitation of the HLHS method is that it is not a truly sequential method. It produces
36 exponentially increasing sample size which can have large computational impact given that, at each
37 extension, the number of simulations must be at least doubled. Other, similar methods for extending
38 LHS designs have been proposed but these methods either do not involve refinement of the design
39 [16] or do not allow the user to retain all original samples [17].
40 Another approach that is gaining popularity is one in which a surrogate model is embedded
41 in the sample design process. A comparison of several such methods for UQ has been conducted
42 by Maljovec et al. [18]. A significant take-away from this article is that, even when samples are
43 informed by a complementary model, it is difficult to achieve the same level of convergence and
44 stability obtained through LHS. Similarly, the notion of surrogate informed sampling has been
45 applied for reliability analysis by Sundar and Shields [19].
46 In this work, a new method - Refined Latinized Stratified Sampling (RLSS) is proposed that

2
47 combines the RSS and HLHS methods into a robust sequential sample size extension approach
48 that is useful for high and low-dimensional transformations of all sort. Specifically, by constructing
49 a sample design that is both Latin (at given intervals) and stratified the method is space-filling
50 and effectively reduces variance associated with both additive/main effects and variable interac-
51 tions [20]. By contrast, classical LHS (and HLHS) reduces variance associated with main effects
52 alone [14, 20]. Several very high-dimensional problems are considered and the influence of main
53 effects and interactions are studied to elucidate the method’s performance in comparison to exist-
54 ing methods. The performance of higher-order statistical estimators and reliability/probability of
55 failure estimates that are difficult to obtain are also studied.
56 The new RLSS methodology, while simple conceptually and quite easy to implement, requires
57 a basic understanding of some existing methodologies. Most notably, RLSS builds upon the exist-
58 ing RSS methodology [8] by combining it with concepts from both Hierarchical Latin hypercube
59 sampling (HLHS) [10, 11, 12] and the newly developed Latinized Stratified Sampling (LSS) [20].
60 For this reason, these methods are discussed in some detail in Sections 2-4 prior to introducing the
61 RLSS method itself.
62 For clarity, the notation used throughout the paper is summarized in Appendix A.

63 2. Sample Size Extension Methods

64 Two leading sample size extension methods, Refined Stratified Sampling and Hierarchical Latin
65 hypercube sampling, are reviewed in this section. Algorithms for their implementation are provided
66 along with a brief discussion of their relative benefits and drawbacks.

67 2.1. Refined Stratified Sampling

68 The Refined Stratified Sampling (RSS) method [8] enables a straightforward extension of a
69 stratified sample design through division of existing strata. This stratum division is in contrast
70 with the historical precedent of simply adding samples to existing strata and, for symmetric stratum
71 division, is proven to reduce the variance of statistical estimators when compared to the previous
72 methods [8]. Moreover, it has been demonstrated that, if the strata division can be somehow
73 optimized there exists a potential for very large variance reduction [8]. The optimization of stratum
74 division, however, remains an open topic for future research.

3
75 The RSS method is conceptually simple and easy to implement. The basic algorithm proceeds
76 as follows and is provided in short form in Algorithm 1. A simple 2D demonstration is shown in
77 Figure 1.
Consider the random vector X = {X1 , X2 , . . . , Xn } having n independent components. Let us
divide the sample space S into a set of M disjoint and space-filling strata (Ω(k) ; k = 1, 2, . . . , M
with ∪M
k=1 Ω
(k) = S and Ω(p) ∩ Ω(q) = ∅; p 6= q) such that each stratum contains exactly one sample

point (M = N ). Assuming a rectilinear stratification (non-rectilinear strata have not yet been
considered but represent a potentially important extension to achieve the aforementioned stratum
optimality), the strata are defined as a set of orthotopes (hyper-rectangles) in the n-dimensional
(k) (k) (k)
probability space [0, 1]n defined by a corner ordinate nearest the origin χ(k) = {χ1 , χ2 , . . . , χn }
(k) (k) (k)
and side lengths [ξ1 , ξ2 , . . . , ξn ] such that the strata probabilities (weights) can be expressed
as:
n M
(k)
Y X
w(k) = ξi ; s.t. w(k) = 1. (1)
i=1 k=1

78 RSS adds samples to this stratified design by dividing in half the largest stratum by probability
79 weight (i.e. divide Ω(k) s.t. w(k) > w(j) ∀j 6= k). In the case where multiple largest Ω(k) exist, one
80 is randomly divided. The selected stratum is divided along the dimension i∗ corresponding to the
(k) (k)
81 maximum side length, Λi∗ = max(ξi ). Again, if multiple maximum side lengths are identified, a
i
82 component is randomly selected from among them. A new sample is added by randomly sampling
83 from the newly defined empty stratum and the weights are recomputed. This process is repeated
84 until the user-prescribed stopping criterion is met.

85 2.2. Hierarchical Latin Hypercube Sampling

86 In the context of Latin hypercube sampling, sample size extension is typically performed through
87 hierarchical methods [10, 11, 12]. The hierarchical methods start with a Latin hypercube design
88 and add samples through successive refinements of the uni-variate strata. Note that herein, the
89 methods in [10, 11, 12] are all referred to as hierarchical Latin hypercube sampling (HLHS) given
90 the similar manner in which each method constructs a sequential “hierarchy” of samples through
91 stratum refinement although only the method in [12] is explicitly identified as HLHS. Although
92 similar, the methods differ in their details in meaningful ways. Specifically, the method in [10] adds
93 the samples randomly while the methods in [11, 12] allow for some measure of control over the
94 sample correlations using either superposition of a new LHS sample set with correlation control

4
 Algorithm 1 Refined Stratified Sampling
Qn (k)
1: Define a stratified design with M strata Ω(k) ; k = 1, . . . , M having probability weights w(k) = i=1 ξi .
2: Draw a random sample x(k) from each stratum Ω(k) ; k = 1, . . . , M .
3: Select a stratum Ω(k) to divide according to the following criteria:

• If a single stratum Ω(k) s.t. ω (k) > ω (j) ∀j 6= k exists, divide this stratum.

• If Ns strata Ω(l) s.t. w(l) = max(w(j) ); l = 1, 2, . . . , Ns exist, randomly select stratum Ω(k) to divide with proba-
j
1
bility P [Ω(k) = Ω(l) ] = .
Ns

4: Divide stratum Ω(k) in half according to the following criteria:

(k) (k)
• Determine the maximum side length of Ω(k) , Λi∗ = max(ξi )
i

(k) (k) (k)

• Divide stratum Ω(k) along component i∗ corresponding to Λi∗ . If Nc components exists such that ξi = Λ i∗ ; i =
1
1, . . . , Nc , randomly select the component i∗ to divide with probability P [i∗ = i] = .
Nc

5: Keeping all existing samples x(k) ; k = 1, 2, . . . , N , randomly sample in the newly defined empty stratum.
(k)
6: Compute weights w(k) = n
Q
i=1 ξi ∀k.
7: Repeat steps 3-6 for every new sample extension.

w1

(x1,y1)  
w2
(x2,y2)  
Y   (1,1)  
w1
(x3,y3)  
(x1,y1)   2. Randomly divide (x1,y1)   w3
6. Divide largest
probability space stratum Ωk* to obtain
w2 strata Ωk; k=1,2,3,4
into strata Ωk; k=1,2
(x2,y2)  
(0,0)   X  
w1 w1
1. Randomly  sample   (x3,y3)   8. Randomly divide
(x1,y1)   4. Randomly divide (x1,y1)   w3 stratum Ωk* to
stratum Ωk* to obtain obtain strata Ωk;
w2 w4 w2
strata Ωk; i=1,2,3 (x4,y4)   k=1,2,3,4,5
(x2,y2)   (x2,y2)  

3. Randomly sample w1 7. Randomly sample

in empty stratum (x1,y1)   in empty stratum
Ωk and assign Ωk and assign new
New  stratum.   weights: wk=P(Ωk) w2 weights: wk=P(Ωk)
(x3,y3)  
w3 (x2,y2)  
Sample  carried  from  previous  step.  
New  sample.  
5. Randomly sample
in empty stratum
X  ~  Probability  space  for  RV1  
Ωk and assign new
Y  ~  Probability  space  for  RV2  
weights: wk=P(Ωk) 9. Randomly sample in empty stratum
Ωk and assign new weights:
wk=P(Ωk)

Figure 1: Refined Stratified Sampling: Flowchart of sample size extension procedure for two random variables.

95 [21] and subsequent stratum splitting [11] or through stratum splitting first and optimization of
96 sample placement [12]. A basic HLHS algorithm (without correlation control) is described in the
97 following, shown in abridged form in Algorithm 2, and depicted for a level-two refinement in Figure
98 2. The method described here most resembles that of [10] and is used throughout.
99 Consider a Latin hypercube design having N samples in n dimensions with strata defined in each

5
 1
100 vector component Ωij , i = 1, . . . , n; j = 1, . . . , N having equal size ξij = . Utilizing HLHS, each
N
101 stratum Ωij is divided into t + 1 smaller strata Ωijl ; i = 1, . . . , n; j = 1, . . . , N ; l = 1, . . . , t + 1 of
1
102 equal size ξijl = , where t is referred to as the refinement factor. Added samples are drawn
(t + 1)N
103 by randomly selecting points from the newly defined empty strata and, as with conventional LHS,
104 randomly grouping the vector components. In this way, HLHS produces Nnew = tN new samples
105 at each extension yielding a total of Ntot = N (t + 1)r samples through r sample size extensions.
106 Note again that the methods in [10, 11, 12] differ in some details from this implementation. Details
107 of these variations are not presented here.

Algorithm 2 Hierarchical Latin Hypercube Sampling

1: Define a Latin hypercube design with N samples and strata in each vector component Xi ; i = 1, . . . , n given by Ωij ; i =
1, . . . , n; j = 1, . . . , N
2: Divide each Ωik into t + 1 equal strata Ωijl ; l = 1, . . . , t + 1
3: Randomly draw samples of each vector component Xi in each empty Ωijl ; i = 1, . . . , n; j = 1, . . . , N ; l = 1, . . . , t + 1.
4: Randomly pair the newly sampled components to construct samples of the n-dimensional random vector.
5: Repeat steps 2-4 for every new sample extension.

Figure 2: Schematic of a Hierarchical Latin hypercube sampling method with refinement factor t = 2. Image from
[12].

108 2.3. Advantages and drawbacks of RSS and HLHS

109 The RSS and HLHS methods currently represent the state-of-the-art in sample size extension
110 for Monte Carlo simulation. Yet, each method has its distinct strengths and limitations. HLHS,
111 on the one hand, utilizes the sampling method, LHS, that provides superior variance reduction for
112 many transformations. Specifically, LHS has been proven to filter the additive effects (main effects)
113 of a transformation because it samples uniformly from all marginals [14]. Stratified sampling, on
114 the other hand, is more effective at reducing variance associated with interaction terms and is

6
115 often preferable for low-dimensional problems [8, 20]. Considering that most transformations have
116 significant additive effects as stated by the Hierarchical Ordering Principle [22] (although these
117 additive effects diminish in importance for higher-order statistics where the transformation is raised
118 to a non-unit power - See Section 5), LHS generally produces a significant variance reduction. LHS
119 is also generally more effective than SS for high dimensional problems - owing again to the fact
120 that it stratifies the marginal distributions.
121 The LHS design, however, places significant limitations on how sample size can be extended.
122 Since it is necessary to stratify each dimension equally, it requires to at least double the sample size
123 at each extension yielding an exponentially growing sample size. When computational constraints
124 are present (as is often the case), this can be a major limitation. Imagine an analysis using 1000
125 Latin hypercube samples that nearly exhausts the available computational resources. In order to
126 extend the sample size using HLHS, it is necessary to conduct at least another 1000 analyses (an
127 additional 2000 are necessary using the method in [12]. This motivation gave rise to the RSS
128 methodology where, because samples can be unevenly weighted, any number of samples can be
129 added in a given extension. Using RSS it is possible, for example, to add only 100 analyses in the
130 previous example. RSS, however, is significantly constrained in that it is not well-suited to high
131 dimensional problems; offering little advantage for dimensions n > log2 (N ) [8]. This motivates the
132 present work.
133 Prior to introducing the new high dimensional RSS methodology, it is necessary in the following
134 section to describe some intermediate developments that enable the high-dimensional extension.

135 3. Hierarchical Latinized Stratified Sampling

136 In this section, we review a method referred to herein as Latinized Stratified Sampling (LSS)
137 [20], that represents a hybrid approach combining both Latin hypercube and stratified sampling
138 into a unified methodology. A new methodology is then presented to enable sample size extension
139 for LSS, referred to as Hierarchical Latinized Stratified Sampling (HLSS) that merges concepts
140 from the HLHS and LSS methodologies.

141 3.1. “Latinized” Stratified Sampling

142 Latinized Stratified Sampling, introduced by Shields and Zhang [20], simultaneously imposes
143 the conditions of one-dimensional and n-dimensional stratification on the sample design. Similar

7
144 methodologies have been presented previously in the form of Orthogonal Array-Based Latin hyper-
145 cubes [23], Binning Optimal Symmetric Latin hypercube sampling [24, 25], and “Su-Do-Ku”-based
146 space-filling designs [26, 27]. Some subtle differences exist however since LSS relaxes some of the
147 conditions of these methods in the interest of conceptual simplicity and ease of implementation.
148 LSS, for example, is not necessarily binning “optimal” as defined by Dalbey and Karystinos [24, 25].
149 Binning optimality requires a cubic stratification and, as can be seen from Figure 3(a), the strata in
150 an LSS design need not be cubes but must be hyperrectangles. Nor do the LSS samples necessarily
151 constitute an orthogonal array or a Su-Do-Ku based design [20] (the left design in Figure 3(a) is
152 not an OA or Su-Do-Ku design while the right design is both). In fact, the LSS method generates
153 OA designs only when all strata have identical dimensions. In fact, LSS can be viewed as a sort of
154 generalization of the above-mentioned designs in the sense that each of these designs represents a
155 subset of LSS designs. Also note that other sampling methods, such as Centroidal Voronoi tessel-
156 lations, have been Latinized as well [28, 29]. The LSS methodology is presented briefly herein, in
157 shortened form in Algorithm 3, and is presented graphically in Figure 4.
158 An LSS design is constructed by first simultaneously defining a Latin hypercube design and
159 an n-dimensional stratified design on the unit hypercube [0, 1]n . To achieve this simultaneous
160 stratification two simple compatibility conditions must be enforced between the stratified design
161 and the LHS design.

162 1. All SS strata must coincide with a LHS stratum boundary.

163 2. All SS strata must be equally weighted hyperrectangles.

164 It is clear to see that if condition 1 is not met, then samples from the LHS design cannot be uniquely
165 assigned to SS strata. If the second condition is not met, then the resulting stratified design yields
166 LHS cells that cannot be sampled while maintaining consistency with both the stratified and LHS
167 designs. This can be seen in Figure 3 where admissible LSS designs are provided in Figure 3(a) and
168 inadmissible designs show in Figure 3(b). For the inadmissible designs, samples cannot be drawn
169 from the shaded cells without violating either the SS design or the LHS design.
170 Having defined both the LHS and SS designs, the samples are drawn by pairing the LHS samples
171 such that one sample is drawn from each stratum. As shown in Figure 4, this can be conducted by
172 sequentially adding a LHS sample to each stratum. Notice that, upon adding a sample to a given

8
 x2 x2

x1 x1
(a) Admissible LSS designs

x2 x2

x1 x1
(b) Inadmissible LSS designs

Figure 3: Admissible and inadmissible LSS designs. Shaded regions show LHS cells where samples cannot be drawn
without violating either the LHS or the SS design.

173 stratum, some LHS cells within other strata cannot be sampled (shaded cells in Figure 4). This is
174 distinguished from the inconsistency shown in Figure 3(b) in that the inaccessible cells in Figure 4
175 only result from the conditional selection based on the previous samples. Continuing through the
176 sample selection, an LHS sample is drawn from those admissible cells within each stratum.

Algorithm 3 Latinized Stratified Sampling

1: Define a Latin hypercube design with N samples and 1D strata Ωij ; i = 1, . . . , n; j = 1, . . . , N . Do not group the
individual components.
2: Define an n-dimensional stratification with strata Ω(k) ; k = 1, . . . , N such that all strata are equally weighted hyperrect-
angles and all strata boundaries coincide with a LHS stratification boundary.
3: For each stratum Ω(k) ; k = 1, . . . , N , randomly select a point xi from each LHS component (without replacement) such
that the constructed sample x(k) = {x1 , x2 , . . . , xn } lies within the stratum.

177 3.2. Sample size extension

178 Utilizing the hierarchical sample size extension concepts presented in Section 2.2, an LSS design
179 can be similarly extended. We refer to this approach as Hierarchical Latinized Stratified Sampling

9
 x2 x2 x2

x1 x1 x1

x2 x2 x2

x1 x1 x1

x2 x2 x2

x1 x1 x1

Figure 4: Schematic description of the Latinized stratified sampling (“Su-Do-Ku Sampling”) method for 9 samples
in 2-dimensions.

180 (HLSS) and its details are discussed in the following with a shortened procedure provided in Algo-
181 rithm 4.
182 First, define an initial LSS design constructed according to Section 3.1 (Algorithm 3). An
183 initial design of four samples in two dimensions is shown in Figure 5(a). Next, define a t-level
184 refinement of the LHS design according to the HLHS procedure in Algorithm 2 as shown in Figure
185 5(b). Divide all SS strata t times along LHS stratum boundaries in the dimension of largest side

10
 (k)
186 length, Λi∗ as in the RSS procedure and shown in Figure 5(c). These two steps define the t-level
187 refinement of the LSS design according to which the samples must be grouped. For each stratum
188 Ω(k) , identify the candidate LHS strata that intersect the stratum in each dimension i = 1, . . . , n.
(k) (k) (k) (k)
189 Denote each set of candidate LHS sampling strata by Θi = {Ωij ∈ [χi , χi + ξi ]}. The
(k)
190 intersection of these strata ∩ni=1 Θi form the LHS sampling cells in each stratum for the HLSS
191 design as shown by the highlighted cells in Figure 5(c). Samples cannot be simply drawn from
192 these candidate cells at this point because there exist certain cells where sampling will result in
193 a violation of either the LHS or the stratified design. Sampling from cell (2,4) in Figure 5(c), for
194 example, will prohibit sampling from the upper left stratum. To avoid this pitfall, it is necessary to
195 sample first from those cells that place such constraints on the design as follows. For each stratum,
196 Ω(k) , count the number of candidate LHS strata in each dimension i = 1, . . . , n and denote this by
(k)
197 ni . For each dimension, i = 1, . . . , n, determine the minimum number of candidate LHS sampling
 
(k)
198 strata from among the strata, mi = min ni . Samples are then selected in dimension i from
k
(k)
199 those strata possessing ni = mi . Note that if mi = 1, then the corresponding stratum Ω(k) has
200 only a single candidate LHS stratum from which it must select a sample in dimension i. This is
201 the case, for example, in the upper left stratum in Figure 5(c) where the component x1 must be
202 drawn from stratum Ω12 . Applying this sampling rule to all dimensions yields the two new samples
(k)
203 shown in Figure 5(d). If mi 6= 1, samples are randomly drawn without replacement from Θi
(k)
204 with ni = mi . This is illustrated by the selection of samples from the top-right and bottom-right
205 strata in Figures 5(d)-5(f) where m1 = 2. First, a sample is randomly drawn from the shaded cells
206 in the top-right stratum which yields only one remaining candidate stratum from which to drawn
207 in the bottom-right stratum (Figures 5(e)-5(f)).

Algorithm 4 Hierarchical Latinizized Stratified Sampling

1: Define an LSS according to Algorithm 3
2: Apply a t-level refinement of the LHS according to the HLHS procedure in Algorithm 2
3: Divide all SS strata Ω(k) along an LHS boundary in the dimension of largest side length Λ(k) (k)
i∗ = max(ξi ).
i
4: For each dimension i of stratum Ω(k) , determine the set of LHS strata that intersect Ω(k) , Θ(k)
i
(k) (k) (k)
= {Ωij ∈ [χi , χi +ξi ]}
(k)
and let ni , i = 1, . . . , n denote the number of such candidate strata in each dimension i.
 
5: Compute mi = min ni(k)
k
6: Sample the ith dimension of each stratum with n(k)
i = mi as follows:

• If mi = 1, sample from Ωij .

(k)
• If mI > 1, sample randomly from Θi .

7: Repeat steps 4-6 until all strata are sampled in all directions.

11
 x2 x2 x2

x1 x1 x1
(a) (b) (c)

x2 x2 x2

x1 x1 x1
(d) (e) (f)

Figure 5: Schematic description of the Hierarchical Latinized stratified sampling (“Su-Do-Ku Sampling”) method for
9 samples in 2-dimensions.

208 4. Refined Latinized Stratified Sampling

209 The newly proposed Refined Latinized Stratified Sampling (RLSS) builds upon all of the pre-
210 viously defined methods to facilitate a truly sequential sample construction under the constraints
211 that it is impossible to achieve a sequential LHS given the need to maintain equal sample weights
212 (more specifically, even distribution of samples in all margins). To address this constraint, the
213 RLSS method builds successive Hierarchical Latinized Stratified Samples and allows intermediate
214 samples to be drawn according to an unevenly weighted stratified design. The methodology is
215 described in detail in the following and provided in short form in Algorithm 5.
216 Beginning with a LSS of arbitrary size N and dimension n, extend the sample using the hier-
217 archical method in Section 3.2 (Algorithm 4). The new HLSS samples define the candidates from
218 which the new samples will be drawn. Specifically, each new sample is selected at random from

12
219 among these candidates and the associated stratum refinement (from the HLSS) is applied. It is
220 important to note that with this sample selection/stratum refinement, the sample is no longer a
221 LHS in general and the sample weights are not equal. Stratum refinement requires that the weights
222 be recomputed according to the conventional RSS methodology using Eq. (1). Note also that the
223 samples are selected at random. Other means of sample selection are considered in Section 5.3
224 but do not provide any additional savings and, more importantly, can introduce noticeable bias in
225 the statistical estimators. Samples are continuously added in this way until a desired number of
226 samples is added, a user-defined convergence criterion is satisfied, or until a new HLSS is required
227 to continue the extension.

Algorithm 5 Refined Latinized Stratified Sampling

1: Define an LSS according to Algorithm 3
2: Refine the LSS according to the HLSS method in Algorithm 4
3: Apply a single stratum refinement from the HLSS design at random.
(k)
Draw the corresponding HLSS sample and compute the stratum weights w(k) = n
Q
4: i=1 ξi .
5: Repeat steps 3-4 until the desired number of samples or until a new HLSS is required. Then return to step 2.

228 The RLSS methodology affords maximal flexibility in sample size extension with continuous
229 convergence of statistical estimators in most cases (depending on the order of the statistic and the
230 nature of the transformation). Although it necessitates a deviation from Latin in the intermediate
231 samples between LSS designs, this is shown to have influence only on the convergence of low-order
232 statistics for strongly additive transformations. Thus, in most cases, the convergence rate of LHS
233 is maintained.

234 5. Exploration of Sample Properties

235 In this section, a variety of analytical 100-dimensional transformations designed to highlight the
236 strengths and limitations of the proposed methodology are considered. The functions vary from
237 strongly additive to those having dominant interactive components as discussed in detail in Section
238 5.1. The behavior for alternate (non-random) sample selection methods is then explored along with
239 the implications of the RLSS method for reliability analysis and the dimensional dependence of the
240 method.

241 5.1. RLSS for transformations of different types

As previously mentioned, LHS reduces variance by filtering the main effects [14] while SS re-
duces variance associated with variable interactions and, to a lesser extent, main effects [20]. Here

13
 two different 100-dimensional functions with varying degrees of main and interaction effects are
considered to observe the performance of RLSS in comparison to HLHS. The influence of main ef-
fects is assessed through the estimation of global sensitivity indices [30], Si , using the methodology
proposed by Saltelli et al. [31]. Specifically, define the cumulative main effect sensitivity as:
X Var(Y |Xi )
SC = Si ; Si = (2)
Var(Y )
i

242 These main effect sensitivities provide a normalized metric for the influence of main effects such that
243 SC = 1 corresponds to a purely additive transformation of the variables (with no variable interac-
244 tions) and SC = 0 corresponds to pure interaction effects (with no additive effects). Alternatively,
245 the cumulative interaction sensitivity can be defined by SI = 1 − SC .

246 5.1.1. Rosenbrock Function

The Rosenbrock function is a common benchmark function for high-dimensional optimization
and uncertainty quantification defined as follows [32]:
n−1
X
fR (x) = 100(x2i − xi+1 )2 + (xi − 1)2 (3)
i=1

247 Consider that x is a 100-dimensional random vector, the convergence of statistical estimators for
248 various moments of fR (x) is investigated here. Three different input distributions are considered
249 as summarized in Table 1 in order to understand the influence of the main effects and variable
250 interactions on statistical convergence. Figure 6 shows the standard error as a function of sample
251 size for estimates of the mean, standard deviation, skewness, and kurtosis for each of the three
252 distributions for both HLHS and RLSS. Notice that, in nearly all cases, the two converge at
253 identical rates. Some variation is observed due to statistical variability from the 100 repeated trials
254 - especially among the higher order moments. Only when main effects are very strong (i.e. first
255 order statistics in the N(0,0.25) and U(-1,1) cases - see Table 1), do the RLSS estimates begin to
256 deviate in their convergence from HLHS. Careful inspection of these cases shows that the standard
257 error of the mean estimate using RLSS achieves minima when the design is Latin (i.e. when the
258 design corresponds with an HLSS). After these minima, when the Latin property of the design is
259 lost and the additive effects are not effectively filtered, there is an initial increase in the variance
260 followed by a sharp decrease as the sample size approaches the next Latin design. This effect is
261 not observed in cases where main effects are not dominant, as is the case in all estimates of higher-

14
262 order statistics (see from Table 1 that raising the transformation to a power diminishes the additive
263 effects and enhances the interaction effects).

P
Table 1: Cumulative main effect global sensitivity indices (SC = i Si ). Note that sensitivities presented here are
for a 10-dimensional functions although the 100-dimensional problems have similar values.
Y Distribution SC (Y − E[Y ]) SC ((Y − E[Y ])2 ) SC ((Y − E[Y ])3 ) SC ((Y − E[Y ])4 )
U(0,1) 0.39 0.10 0.19 0.06
fR (x) U(-1,1) 0.80 0.18 0.31 0.11
N(0,0.25) 0.89 0.49 0.44 0.39
Exp(1) 1 0.30 0.51 0.23
fA (x) U(0,1) 1 0.04 0.63 0.03
U(-1,1) 1 0.04 0.62 0.02
N(0,1) 1 0.12 0.59 0.07

264 To add a sense of scale to this assessment, Figure 7 shows banded estimates of the statistics as
265 a function of sample size considering normal random variables. All bands are shown as mean ±
266 one standard deviation. Notice that the bands coincide very closely with the associated bands from
267 HLHS (‘o’ in Figure 7) and that the effect of variance increase in the intermediate states for the
268 mean value is nearly imperceptible since the mean converges rapidly. For the higher-order statistics
269 where the scale of variance is significant, there is no variance increase at intermediate states given
270 the diminished main effects. Consequently the variance increase observed in mean value estimation
271 has little impact on the performance of the RLSS method - even when additive effects are strong.
272 Having established that convergence of the proposed method is comparable to HLHS, its ben-
273 efits can be fully realized. The plots in Figures 6 and 7 show convergence for the RLSS method
274 as continuous lines and HLHS only at discrete points. These discrete points represent the only
275 achievable sample sizes for the HLHS method considering an initial sample size of N = 2 and a
276 refinement factor t = 1. Consider that the threshold for acceptable statistical convergence corre-
277 sponds to obtaining an estimate with coefficient of variation less than . Table 2 shows the number
278 of simulations required for the two methods to obtain given levels of convergence in each statis-
279 tic. The sequential nature of the RLSS provides potentially major savings in computational effort,
280 often reducing the number of simulations nearly in half compared to HLHS. This is attractive for
281 both small sample uncertainty analyses where low-order statistics are estimated from only a few
282 simulations and large sample analyses where higher-order statistics can be estimated.

15
 10 3 10 3

Standard Error of Standard Deviation

N(0,0.25) N(0,0.25)
HLHS HLHS
U(0,1) U(0,1)
Standard Error of Mean

10 2 HLHS HLHS
U(-1,1) U(-1,1)
10 2
HLHS HLHS

10 1 1
2

10 1 1
0 2
10

10 -1 10 0
10 0 10 1 10 2 10 3 10 4 10 0 10 1 10 2 10 3 10 4
Number of Samples Number of Samples
(a) (b)

10 0 10 1
N(0,0.25) N(0,0.25)
HLHS HLHS
Standard Error of Skewness

Standard Error of Kurtosis

U(0,1) U(0,1)
HLHS HLHS
U(-1,1) U(-1,1)
10 0
HLHS HLHS

-1
10
1
2 1
10 -1 2

10 -2 10 -2
10 1 10 2 10 3 10 4 10 1 10 2 10 3 10 4
Number of Samples Number of Samples
(c) (d)

Figure 6: Rosenbrock Function: Convergence of statistical estimators as a function of sample size and variable
distribution for RLSS and HLHS: (a) Mean, (b) Standard Deviation, (c) Skewness, and (d) Kurtosis. The continuous
lines denote the RLSS approach while the markers denote achievable HLHS designs.

283 5.1.2. Additive Function

The previous section argued that the observed variance increase for intermediate designs when
the transformation is dominated by additive components is largely inconsequential. In this section,
this result is explored for the extreme case of a purely additive function:
n
X
fA (x) = xi . (4)
i=1

284 Again, let n = 100 to illustrate for a very high-dimensional application. Cumulative main effect
285 sensitivities are given in Table 1 for different distributions of x. Notice that the main effects are

16
 1050 0.9 1.5

1000 0.8
950 1
0.7
900
0.6

Skewness
0.5

Kurtosis
850
0.5
800
0.4 0
750
0.3
700 -0.5
650 0.2

600 0.1 -1
10 1 10 2 10 3 10 4 10 2 10 3 10 4 10 2 10 3 10 4
Number of Samples Number of Samples Number of Samples

(a) (b) (c)

Figure 7: Rosenbrock Function: Mean ± one standard deviation confidence bands for statistical estimators as a
function of sample size: (a) Mean and mean ± one standard deviation, (b) Skewness, and (c) Kurtosis.

Table 2: Rosenbrock Function: Number of simulations required to achieve accuracy in statistics within a given
coefficient of variation,  for RLSS and HLHS. (— means this level of convergence was not achieved in 8192 samples)
N(0,0.25) U(0,1) U(-1,1)
 RLSS HLHS RLSS HLHS RLSS HLHS
Mean 0.05 7 8 5 8 4 4
0.01 62 128 76 128 59 64
Std. Dev. 0.1 55 64 56 64 56 64
0.05 230 256 237 256 219 256
0.01 6447 8192 6474 8192 6692 8192
Skewness 0.5 106 128 485 1024 455 512
0.25 491 1024 2031 2048 1410 2048
0.1 4053 4096 — — — —
Kurtosis 0.1 689 2048 334 256 447 512
0.05 4599 4096 1392 2048 1586 4096

286 almost entirely eliminated for even power moments.

287 In this case, the additive effects control the convergence strongly for the mean value but again,
288 have little influence on the convergence of higher-order statistics. Figure 8 shows the standard
289 error as a function of sample size for each of the first four moments. HLHS and the HLSS states of
290 the RLSS method are exceptionally effective for mean value estimation. The intermediate states
291 of the RLSS method, however, show an increase in estimate variance that can be several orders of
292 magnitude. This is only of marginal concern though because the Latin states converge so rapidly
293 that an accurate estimate of the mean is obtained very quickly for additive transformations. The
294 variance loss is not observed in higher-order moment estimation while the convergence rate is the
295 same for the RLSS method and HLHS. Note that variations between the convergence rates in
296 Figures 8(c) and 8(d) is most likely caused by statistical variation as the estimates are produced

17
297 from only 100 repeated trials (a small number for higher-order statistics).

10 1 10 1

Standard Error of Standard Deviation

0
10 1 2
Standard Error of Mean

-1
10
10 0
-2
10
Exp(1)
HLHS
10 -3
U(0,1) N(0,0.25)
-4 HLHS 10 -1 HLHS
10 N(0,1) 1
U(0,1)
1
2
1
HLHS HLHS
-5
10 U(-1,1) U(-1,1)
HLHS HLHS
10 -6 10 -2
10 0 10 1 10 2 10 3 10 4 10 0 10 1 10 2 10 3 10 4
Number of Samples Number of Samples
(a) (b)

10 0 10 0
Exp(1) N(0,0.25)
HLHS HLHS
Standard Error of Skewness

Standard Error of Kurtosis

U(0,1) U(0,1)
HLHS HLHS
N(0,1) U(-1,1)
HLHS HLHS
U(-1,1) 1
-1 HLHS -1 2
10 10
1

10 -2 10 -2
10 1 10 2 10 3 10 4 10 1 10 2 10 3 10 4
Number of Samples Number of Samples
(c) (d)

Figure 8: Additive Function: Convergence of statistical estimators as a function of sample size and variable distribu-
tion for RLSS and HLHS: (a) Mean, (b) Standard Deviation, (c) Skewness, and (d) Kurtosis. The continuous lines
denote the RLSS approach while the markers denote achievable HLHS designs.

298 5.2. Effect of sample selection criterion

299 The proposed RLSS method prescribes that samples are added (and strata refined) randomly
300 in any given extension. Two alternative non-random ‘maximin’ sample extension algorithms have
301 been considered as follows:

302 • 1D Maximin Selection: The next sample is selected from the HLSS that maximizes the
303 minimum distance in a single dimension. That is, each sample in the HLSS design has n

18
304 nearest neighbors - one in each dimension. This method selects the sample whose farthest
305 nearest neighbor is farthest away.

306 • L1 Maximin Selection: The next sample is selected from the HLSS that maximizes the mini-
307 mum distance with its nearest neighbor using a standard L1 -norm.

308 Note that many other such designs could be considered. In both cases, the overall convergence
309 rate of the statistical estimators remains unchanged as shown for the mean and standard deviation
310 of the Rosenbrock function with N(0,0.25) random variables in Figure 9. Any difference between
311 the convergences in Figure 9 is attributed to statistical variation. Although not shown, the higher-
order statistics and other examples show the same trend. The maximin selection criteria also do

10 2 10 2

Standard Error of Standard Deviation

Random Random
1D Maximin 1D Maximin
L1 Maximin L1 Maximin
Standard Error of Mean

10 1 10 1
1 1
2 2

10 0 10 0
10 0 10 1 10 2 10 3 10 0 10 1 10 2 10 3
Number of Samples Number of Samples
(a) (b)

Figure 9: Rosenbrock Function: Convergence of statistical estimators as a function of sample size for different RLSS
sample selection methods considering N(0,0.25) random variables: (a) Mean and (b) Standard Deviation.
312

313 not alter the variance increase during intermediate RLSS designs. In this sense, the convergence
314 is insensitive to the sample selection criteria during extension. Instead, it is the underlying HLSS
315 design that governs the convergence with variance loss in mean value estimation due to additive
316 effects at intermediate samples that does not appear reducible by changing the sample selection
317 criterion.
318 However, the non-random selection criteria may bias the statistical estimates. As shown in
319 Figure 10, the 1D maximin selection yields biased solutions at intermediate RLSS samples because
320 the criterion causes samples near the boundaries of the domain [0, 1]n to be sampled early while
321 those in the central strata follow. Considering this potential for bias combined with the observed

19
 2300

2200

2100

2000

1900

1800

1700
10 1 10 2 10 3
Number of Samples

Figure 10: Rosenbrock Function: Mean ± one standard deviation with one standard deviation bounds considering
U(0,1) random variables with the 1D maximin sample selection criteria. Intermediate samples between HLSS states
are biased by the sample selection routine.

322 lack of improvement in convergence using maximin design, the RLSS method proposed herein
323 recommends random sample selection.

324 5.3. Reliability Analysis

In this section, HLHS and RLSS are applied to estimate probability of failure, pf - which can
be expressed as the first moment (mean) of a subspace Ωf (referred to as the failure domain)
associated with undesirable performance (or failure) as:
Z ∞
pf = I(x ∈ Ωf )f (x)dx (5)
−∞

where I(·) is the indicator function and f (x) is the joint density of the random vector x. Classical
Monte Carlo and stratified estimators of pf can be constructed as:
N
1 X
p̂f = I(xi ∈ Ωf ) (6)
N
i=1

and
N (k)
X w i
p̄f = I(xi ∈ Ωf ) (7)
Mk
i=1
(k)
325 where wi is the probability weight of stratum Ω(k) from which sample i is drawn and Mk is the
326 total number of samples in stratum Ω(k) .
327 Consider the additive function given in Eq. (4) with n = 100 and xi ∼ N (0, 1). Figure 11
328 provides convergence plots for the RLSS estimate of pf = P {fA (x) > 30} using Eq. (7) and the

20
 -1 ×10 -3
Standard Error of Probability of Failure 10 5

Probability of Failure
10 -2
2

0
10 -3 1
2
-1

-2
-4
10 -3
0 1 2 3 4
10 10 10 10 10 10 2 10 3 10 4
Number of Samples Number of Samples
(a) (b)

Figure 11: Additive Function: Convergence of RLSS and HLHS for probability of failure estimation. (a) Standard
error as a function of sample size. (b) Mean ± one standard deviation pf estimates.

329 HLHS estimate of pf using Eq. (6). Figure 11(a) shows the standard error for the pf estimates
330 while Figure 11(b) shows mean ± one standard deviation estimates for pf . The true probability of
331 failure evaluated analytically is pf = 0.00135. Notice that the convergence does not demonstrate
332 a variance increase at intermediate RLSS states. Although not shown, the same trend of continu-
333 ously decreasing variance is observed for different distribution types as well as those cases of the
334 Rosenbrock function previously considered. This indicates that the method can be very effective
335 for reliability analysis - especially when paired with importance sampling or methods such as that
336 proposed by Olsson et al. [33]. This is an ongoing topic of research.

337 5.4. Dimensional dependence

338 The methodology presented resolves the primary pitfall of the original RSS methodology; that
339 it is useful only for low-dimensional problems [8]. In the previous sections, it has been shown
340 that RLSS performs as well or better than HLHS at allowable HLHS sample sizes for very high
341 dimensional problems with the added benefit of truly sequential sample size extension. The benefits
342 for low-dimensional problems are even more significant. Consider again the Rosenbrock function in
343 Eq. (3). Figure 12 shows convergence of the mean and standard deviation estimates for 2, 5, 10, and
344 100 dimensional cases with Xi ∼ U (0, 1) and Xi ∼ N (0, 0.25). There is a dramatic improvement in
345 performance for the RLSS method compared to HLHS for low-dimensional applications. This holds
346 true even for cases where additive components dominate (Xi ∼ N (0, 0.25) - see Table 1) although

21
347 the improvement is more significant for transformations with large interaction effects (Xi ∼ U (0, 1)
348 - see Table 1). Note that, although not shown in Figure 12, the same trend is observed for higher-
349 order statistics as well.

3 3
10 10

Standard Error of Standard Deviation

2 2
10 1 10 1
Standard Error of Mean

2 2

10 1 10 1

10 0 2D-RLSS 10 0 2D-RLSS
2D-HLHS 2D-HLHS
5D-RLSS 5D-RLSS
10 -1 5D-HLHS 1 10 -1 5D-HLHS 1
10D-RLSS 1 10D-RLSS 1

-2 10D-HLHS -2 10D-HLHS
10 100D-RLSS
10 100D-RLSS
100D-HLHS 100D-HLHS
10 -3 10 -3
10 0 10 1 10 2 10 3 10 4 10 0 10 1 10 2 10 3 10 4
Number of Samples Number of Samples
(a)

3 3
10 10
Standard Error of Standard Deviation

2D-RLSS
2D-HLHS
10 2 1 5D-RLSS
Standard Error of Mean

2
10 2 1 5D-HLHS
1 2 10D-RLSS
10
10D-HLHS
100D-RLSS
0 2D-RLSS 1 100D-HLHS
10 10
2D-HLHS
5D-RLSS
10 -1 5D-HLHS 1
1
10D-RLSS 10 0 1
-2 10D-HLHS
10 100D-RLSS
1

100D-HLHS
10 -3 10 -1
10 0 10 1 10 2 10 3 10 4 10 0 10 1 10 2 10 3 10 4
Number of Samples Number of Samples
(b)

Figure 12: Rosenbrock Function: Standard error convergence of the mean and standard deviation as a function of
sample size for problems of different dimension considering (a) U (0, 1) random variables and (b) N (0, 0.25) random
variables.

350 6. Conclusions

351 A robust sequential sampling method, Refined Latinized Stratified Sampling, has been pro-
352 posed for reducing the computational expense of simulation-based uncertainty quantification stud-
353 ies. The method combines the benefits of the existing Refined Stratified Sampling and Hierarchical
354 Latin hypercube sampling methods to produce a method that significantly reduces the variance

22
355 of statistical estimators for very high-dimensional problems. By producing sample designs that
356 are simultaneously Latin and stratified at given intervals, the method is shown to filter both ad-
357 ditive and interaction effects. The method is truly sequential in nature, allowing addition of a
358 single or multiple samples in a given extension through refinement of the strata. The reduced
359 variance combined with the sequential extension allows for maximal flexibility in sample design
360 and ultimately a significant reduction in sample size for a given accuracy threshold. The perfor-
361 mance of the method has been studied for high-dimensional functions of different type (strongly
362 additive and non-additive) - demonstrating its robust performance for any general transforma-
363 tion. Matlab code for the methods presented herein can be found on the author’s personal website
364 https://sites.google.com/site/jhusurg/.

365 Appendix A. Notation:

366 M = number of strata in stratified design

367 Mk = number of samples drawn from stratum Ω(k)
368 n = dimension of a random vector
(k)
369 ni = in HLSS, the number of candidate LHS sampling strata in dimension i that intersect
370 with stratum Ω(k)
371 N = number of samples
372 Nc = number of vector components meeting a given criterion
373 Ns = number of strata meeting a given criterion
374 Nnew = tN = number of new samples in an HLHS sample size extension
375 Ntot = N (t + 1)r = total number of samples after r extensions of a HLHS
376 t = refinement factor for HLHS
(k)
w(k) = ni=1 ξi = probability weight of stratum k
Q
377

378 S = sample space

(k)
379 Θi = in HLSS, the set of candidate LHS sampling strata in dimension i that intersect with
380 stratum Ω(k)
(k) (k)
381 Λi∗ = max(ξi ) = maximum side length of stratum k
i
382 χ(k) = corner ordinate of stratum k nearest the origin
1
383 ξij = = size of LHS stratum j in dimension i
N
1
384 ξijl = = size of the lth new LHS stratum created by splitting stratum j in dimension
(t + 1)N
385 i

23
 (k)
386 ξi = side length of stratum k in dimension i
387 Ωij = j th one dimensional LHS stratum in dimension i
388 Ωijl = lth new LHS stratum created by splitting LHS stratum j in dimension i
389 Ω(k) = k th stratum of a stratified design

390 References

391 [1] J. Warner, M. Grigoriu, W. Aquino, Stochastic reduced order models for random vectors:
392 Application to random eigenvalue problems, Probabilistic Engineering Mechanics 31 (2013)
393 1–11.

394 [2] V. Christou, P. Bocchini, M. Miranda, Optimal representation of multi-dimensional random

395 fields with a moderate number of samples: Application to stochastic mechanics, Probabilistic
396 Engineering Mechanics.

397 [3] D. Ghiocel, R. Ghanem, Stochastic finite-element analysis of seismic soil-structure interaction,
398 Journal of Engineering Mechanics 128 (2002) 66–77.

399 [4] G. Blatman, B. Sudret, An adaptive algorithm to build up sparse polynomial chaos expansions
400 for stochastic finite element analysis, Probabilistic Engineering Mechanics 25 (2010) 183–197.

401 [5] D. Giovanis, V. Papadopoulos, Spectral repressentation-based neural network assisted stochas-
402 tic structural mechanics, Engineering Structures 84 (2015) 382–394.

403 [6] H. Niederreiter, QuasiMonte Carlo Methods, Wiley Online Library, 2010.

404 [7] K. R. Dalbey, L. P. Swiler, Gaussian Process Adaptive Importance Sampling, International
405 Journal for Uncertainty Quantification 4 (2) (2014) 133–149.

406 [8] M. Shields, K. Teferra, A. Hapij, R. Daddazio, Refined stratified sampling for efficient monte
407 carlo based uncertainty quantification, Reliability Engineering and System Safety 142 (2015)
408 310–325.

409 [9] M. D. Shields, V. Sundar, Targeted Random Sampling: A new approach for efficient reliability
410 estimation for complex systems, International Journal of Reliability and Safety 9 (2/3) (2015)
411 174–190. doi:http://dx.doi.org/10.1504/IJRS.2015.072718.

24
412 [10] C. Tong, Refinement strategies for stratified sampling methods, Reliability Engineering and
413 System Safety 91 (2006) 1257–1265.

414 [11] C. Sallaberry, J. Helton, S. Hora, Extension of latin hypercube samples with correlated vari-
415 ables, Reliability Engineering and System Safety 93 (2008) 1047–1059.

416 [12] M. Vorechovsky, Hierarchical refinement of latin hypercube samples, Computer-Aided Civil
417 and Infrastructure Engineering 30 (2015) 394–411.

418 [13] M. McKay, R. Beckman, W. Conover, A comparison of three methods of selecting values of
419 input variables in the analysis of output from a computer code, Technometrics 21 (2) (1979)
420 239–245.

421 [14] M. Stein, Large sample properties of simulations using latin hypercube sampling, Technomet-
422 rics 29 (2) (1987) 143–151.

423 [15] J. Helton, F. Davis, Latin hypercube sampling and the propagation of uncertainty in analyses
424 of complex systems, Reliability Engineering and System Safety 81 (2003) 23–69.

425 [16] M. McKay, Evaluating prediction uncertainty, Tech. Rep. NUREG/CR-6311, Los Alamos Na-
426 tional Laboratory (1995).

427 [17] Z.-Z. Liu, W. Li, Y. M., Two general extension algorithms of latin hypercube sampling, Math-
428 ematical Problems in Engineering (2015) 450492.

429 [18] D. Maljovec, B. Wang, A. Kupresanin, G. Johannesson, V. Pascucci, P.-T. Bremer, Adaptive
430 sampling with topological scores, International Journal for Uncertainty Quantification 3 (2)
431 (2013) 119–141.

432 [19] V. Sundar, M. D. Shields, Surrogate-enhanced stochastic search algorithms to identify

433 implicitly defined functions for reliability analysis, Structural Safety 62 (2016) 1–11.
434 doi:10.1016/j.strusafe.2016.05.001.

435 [20] M. Shields, J. Zhang, The generalization of latin hypercube sampling, Reliability Engineering
436 and System Safety.

25
437 [21] R. L. Iman, W. J. Conover, A distribution-free approach to inducing rank correlation among
438 input variables, Communications in Statistics - Simulation and Computation 11 (3) (1982)
439 311–334. doi:10.1080/03610918208812265.

440 [22] C. Wu, M. Hamada, Experiments: Planning, Analysis, and Parameter Design Optimization,
441 Wiley, New York, 2000.

442 [23] B. Tang, Orthogonal array-based latin hypercubes, Journal of the American Statistical Asso-
443 ciation 88 (1993) 1392–1397.

444 [24] K. Dalbey, G. Karystinos, Fast generation of space-filling latin hypercube sample designs, in:
445 Proceedings of the 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Confer-
446 ence, 2010.

447 [25] K. Dalbey, G. Karystinos, Generating a maximally spaced set of bins to fill for high dimensional
448 space-filling latin hypercube sampling, International Journal for Uncertainty Quantification
449 1 (3) (2011) 241–255.

450 [26] X. Xu, B. E. N. Haaland, P. Z. G. Qian, Sudoku-based space-filling designs., Biometrika 98 (3)
451 (2011) 711–720.

452 [27] X. Xu, P. Z. G. Qian, Q. Liu, Samurai Sudoku-Based Space-Filling Designs for Multi-Source
453 Inference, Available at SSRN 2505445.

454 [28] V. J. Romero, J. V. Burkardt, M. D. Gunzburger, J. S. Peterson, Comparison of

455 pure and “Latinized” centroidal Voronoi tessellation against various other statistical sam-
456 pling methods, Reliability Engineering & System Safety 91 (10-11) (2006) 1266–1280.
457 doi:10.1016/j.ress.2005.11.023.

458 [29] Y. Saka, M. Gunzburger, J. Burkardt, Latinized, improved LHS, and CVT point sets in hy-
459 percubes, International Journal of Numerical Analysis and Modeling 4 (3-4) (2007) 729–743.

460 [30] A. Saltelli, M. Ratto, T. Andres, F. Campolongo, J. Cariboni, D. Gatelli, M. Saisana, S. Taran-
461 tola, Global Sensitivity Analysis: The Primer, Wiley, 2008.

462 [31] A. Saltelli, Making best use of model evaluations to compute sensitivity indices, Computer
463 Physics Communications 145 (2002) 280–297.

26
 464 [32] H. Rosenbrock, An automatic method for finding the greatest ro least value of a function, The
465 Computer Journal 3 (3) (1960) 175–184.

466 [33] A. Olsson, G. Sandberg, O. Dahlblom, On latin hypercube sampling for structural reliability
467 analysis, Structural Safety 25 (2003) 47–68.

27

View publication stats