CHAPTER 12

STATISTICAL METHODS FOR

OPTIMIZATION IN DISCRETE PROBLEMS
Organization of chapter in ISSO
Basic problem in multiple comparisons
Finite number of elements in search domain O
Tukey-Kramer test
Many-to-one tests for sharper analysis
Measurement noise variance known
Measurement noise variance unknown (estimated)
Ranking and selection methods

Slides for Introduction to Stochastic Search
and Optimization (ISSO) by J. C. Spall
12-2
Background
Statistical methods used here to solve optimization
problem
Not just for evaluation purposes
Extending standard pairwise t-test to multiple
comparisons
Let u e O {u
1
, u
2
, , u
K
} be finite search space (K
possible options)
Optimization problem is to find the j such that u
-
= u
j

Only have noisy measurements of L(u
i
)
12-3
Applications with Monte Carlo
Simulations
Suppose wish to evaluate K possible options in a
real system
Too difficult to use real system to evaluate options
Suppose run Monte Carlo simulation(s) for each of
the K options
Compare options based on a performance measure
(or loss function) L(u) representing average (mean)
performance
u represents options that can be varied
Monte Carlo simulations produce noisy
measurement of loss function L at each option
12-4
Statistical Hypothesis Testing
Null hypothesis: All options in O {u
1
, u
2
, , u
K
}
are effectively the same in the sense that L(u
1
) =
L(u
2
) = = L(u
K
)
Challenge in multiple comparisons: alternative
hypothesis is not unique
Contrasts with standard pairwise t-test
Analogous to standard t-test, hypothesis testing
based on collecting sample values of L(u
1
), L(u
2
),
and L(u
K
), forming sample means
, ,...,
1 2 K
L L L
12-5
TukeyKramer Test
Tukey (1953) and Kramer (1956) independently
developed popular multiple comparisons analogue to
standard t-test
Recall null hypothesis that all options in O {u
1
, u
2
, ,
u
K
} are effectively the same in the sense that L(u
1
) =
L(u
2
) = = L(u
K
)
TukeyKramer test forms multiple acceptance
intervals for K(K1)/2 differences o
ij

Intervals require sample variance calculation based on
samples at all K options
Null hypothesis is accepted if evidence suggests all
differences o
ij
lie in their respective intervals
Null hypothesis is rejected if evidence suggests at least
one o
ij
lies outside its respective interval

i j
L L
12-6
Example: Widths of 95% Acceptance
Intervals Increasing with K in TukeyKramer
Test (n
1
=n
2
==n
K
=10)
12-7
Example of TukeyKramer Test
(Example 12.2 in ISSO)
Goal: With K = 4, test null hypothesis L(u
1
) = L(u
2
) =
L(u
3
) = L(u
4
) based on 10 measurements at each u
i

All (six) differences o
ij


must lie in acceptance
intervals [1.23, 1.23]
Find that o
34
= 1.72
Have o
34
e [1.23, 1.23]
Since at least one o
ij
is not in acceptance interval, reject
null hypothesis
Conclude at least one u
i
likely better than others
Further analysis required to find u
i
that is better

i j
L L
12-8
Multiple Comparisons Against One Candidate
Assume prior information suggests one of K points is
optimal, say u
m

Reduces number of comparisons from K(K1)/2 differences
o
ij
= to only K1 differences o
mj

Under null hypothesis, L(u
m
) > L(u
j
) for all j
Aim to reject null hypothesis
Implies that L(u
m
) < L(u
j
) for at least some j
Tests based on critical values < 0 for observed
differences o
mj

To show that L(u
m
) < L(u
j
) for all j requires additional
analysis

i j
L L
o
mj
12-9
Example of Many-to-One Test with
Known Variances (Example 12.3 in ISSO)
Suppose K = 4, m = 2 Need to compute 3 critical
values , , and for acceptance regions
Valid to take
Under Bonferroni/Chebyshev:
Under Bonferroni/normal noise:
Under Slepian/normal noise:
Note tighter (smaller) acceptance regions when assuming
normal noise

21
o
23
o
24
o
24 23 21
o = o = o o
96 . 3 = o
10 . 1 = o
09 . 1 = o
} {
2 2 j j
o > o
Widths of 95% Acceptance Intervals (< 0)
for Tukey-Kramer and Many-to-One Tests
(n
1
=n
2
==n
K
=10)
12-11
Ranking and Selection:
Indifference Zone Methods
Consider usual problem of determining best of K possible
options, represented u
1
, u
2
,, u
K

Have noisy loss measurements y
k
(u
i
)
Suppose analyst is willing to accept any u
i
such that L(u
i
)
is in indifference zone [L(u
-
), L(u
-
) + o)
Analyst can specify o such that
P(correct selection of u = u
-
) > 1 o
whenever L(u
i
) L(u
-
) > o for all u
i
= u
-

Can use independent sampling or common random
numbers (see Section 14.5 of ISSO)