Simulation 1

Output Data Analysis II

Simulation 2
Outline
Motivation and Terminology
Recall: sampling distribution and mean
Confidence intervals for mean and quintiles
Estimators and Correlation
Terminating simulations
Non-terminating Simulations:Steady state
estimation
Transient Removal
Stopping Rules
Independent Replications, Batch means and
Regeneration methods

Simulation 3
Non-terminating Simulations: Steady State
Using multiple runs, Y
i
s distributions can be estimated
how do they look like?
As the Y
i
s are not identically distributed, they will not all look the
same




Steady State:
In a queuing system, waiting times in a queue are short at the
beginning, but after some time, the system stabilizes and waiting
times become similar the system has reached a steady state
Distributions at the beginning (for small i) will be different, but
distributions will converge to a common distribution in the steady
state as i becomes large

1,1 1, 1,
2,1 2, 2,
,1 ,
... ...
... ...
... ... ... ...
... ... ...
i n
i n
R R n
Y Y Y
Y Y Y
Y Y
Simulation 4
Distributions of Y
i
in steady-state
Interval 1 Interval 2 Interval 3 Interval 4 Interval 5 Interval 6
Simulation Time
Warm-up Ends
Distribution for Output Response
Transient State Steady - State
Averaged
Output
Response
Measure

Y
Initial Conditions in the Queue
Simulation 5
s is the starting population in the queue

Simulation 6
Effect of initial conditions
The start-up problem: In general,
choosing initial conditions representative
of the steady-state is not possible.
Simulation usually starts from the empty
system
Simulation 7
Output analysis for steady-state simulation
A single run of a simulation model
Output: Y
1
, Y
2
, , : high degree of serial
autocorrelation typical for queuing systems

=

=
n
i
i
n
Y
n
1
1
lim
Goal: estimate a steady-state parameter
Simulation 8
Output Data Autocorrelation : Example
Fast food restaurant (McDonalds drive) with a
single drive-thru window
If one car must wait a long time because they joined the
end of a long queue, then it is likely that cars arriving
just before or just after this car will also experience
longer waits
Conversely, if a car arrives and the driver immediately
places an order, then it is likely that the next car
arriving will experience at most a short delay.
Cars arriving in close proximity to one another are
more likely to exhibit similar waiting times
Positively autocorrelated data
Simulation 9
Output analysis for steady-state simulation:
Problems
Estimating steady -state parameters when
part of the samples are taken while the
model has not yet reached the steady state
are not representative
How to determine the start of the steady
state?
Simulation cannot run forever
How to choose n to terminate simulation?
Achieve desired precision/confidence
interval in presence of output data
autocorrelation
Simulation 10
Outline
Motivation and Terminology
Recall: sampling distribution and mean
Confidence intervals for mean and quintiles
Estimators and Correlation
Terminating simulations
Non-terminating Simulations:Steady state
estimation
Transient Removal
Stopping Rules
Independent Replications, Batch means and
Regeneration methods

Simulation 11
Transient Removal
Difficult to define exactly what constitutes end of
transient state
Use heuristics:
Long runs
Proper initialization
Truncation
Initial data deletion
Moving average of replications
Batch means
Simulation 12
Long Runs
Use very long runs
Effects of transient state will be amortized
But wastes resources
And tough to choose how long is enough
Recommendation dont use long runs alone
Simulation 13
Proper Initialization
Start simulation in a state close to equilibrium
Put some customers into the queue of a queuing
system, e.g CPU scheduler may start with some jobs in
the queue
Determine starting conditions by previous
simulations or simple analysis
Difficult to identify state information that is close
to steady state yet not limiting the possible
different outcomes of a model
May result in decreased run length, but still may not
provide confidence that are in stable condition
Simulation 14
Truncation
Assume variability during
steady state is less than
during transient state
Variability measured in
terms of range
(min, max)
If a trajectory of range
stabilizes, then assume
stable state is achieved
Method:
Given n observations {x
1
,
x
2
, , x
n
}
ignore first k observations
Calculate (min,max) of
remaining n-k
Repeat for k = 1n
Stop when k+1-th
observation is neither min
nor max

(Example next)
Simulation 15
Truncation Example
Sequence: 1, 2, 3, 4, 5, 6,
7, 8, 9, 10, 11, 10, 9, 10,
11, 10, 9
Ignore first (k=1), range is
(2, 11) and 2
nd
observation
(k+1) is the min
Ignore second (k=2),
range is (3,11) and 3
rd

observation (k+1) is min
Finally, k=9 and range is
(9,11) and 10
th

observation is neither min
nor max
So, discard first 9
observations

Transient
Interval
Simulation 16
Initial Data Deletion (1 of 3)
Observation: in steady state, the average of
samples is not very sensitive to the deletion of
some observations
On the other hand, average over observation
changes when data from initial transient is
included or not
Problem: To protect against small random
fluctuations, each individual observation needs to
be smoothed

Simulation 17
Initial Data Deletion (2 of 3)
Given n replications m size each with x
ij
j-th observation of
i-th replication; i=1,2,,n ; j=1,2,,m, Note j varies
along time axis and i varies across replications
Get mean trajectory:

Get overall mean:


Set k=1. Assume transient state of k samples , delete first
k and repeat for remaining m-k


Compute relative change
Repeat with k from 1 to n-1. Plot. Relative change graph
will stabilize at knee. Choose k there and delete 1 through
k

=
ij j
x
n
x
1

=
j
x
m
x
1

+ =

=
m
k j
j k
x
k m
x
1
) (
1
x x x
k
/ ) (
Simulation 18
Initial Data Deletion (3 of 3)
x
ij
j

x
j
j

x
k

k

(
x
k

x
)

/

x

k

transient
interval
knee
n replications m size each Mean trajectory over n
k-samples transient state
Relative change
Simulation 19
Moving Average (Welchs Procedure)
1. Make n replications of the simulation (n>5)
each of length m observations (m large)
Let Y
ji
be the i-th observation from the j-th
replication (j=1,2,...,n; i=1,2,...,m)
2. Define the averaged output process
n Y Var Var
Y E E
Now
. ,...,m , /n, i Y
i
i
n
j
ji
/ ) ( ) Y (
) ( ) Y (
,
2 1 Y
i
i
1
i
=
=
= =

=
Simulation 20
Moving Average
3. To further smooth out the statistical
fluctuations calculate the moving average

4
1
1 2
1
1 2
) ( Y
1
1
i
m/ atisf ies w window w s where the
,...,w if i
i
Y
,...,m-w w if i
w
Y
w
i
) (i s
s i
w
w s
s i
s

=
+
+ =
+
=

=
+
=
+
Simulation 21
Moving Average
4. Plot the resulting process for i=1,2,...,m-w and
choose the length of the warm-up period to be
the value of i beyond which the sequence
appears to have converged
Simulation 22
Moving Average: Example
M/M/1 Queue (=0.5, =1, n=10, w=0)
Simulation 23
Moving Average: Example
M/M/1 Queue (=0.5, =1, n=10, w=10)
Simulation 24
Moving Average: Example
M/M/1 Queue (=0.5, =1, n=10, w=10)
Simulation 25
Moving Average: Example
M/M/1 Queue (=0.5, =1, n=10, w=20)
Simulation 26
Moving Average: Recommendations
Use as many samples m per replication as is
feasible
Choose window size w <= m/4
Make sure to use the smallest window size for
which the curve is sufficiently smooth
Larger windows over-aggregate the data, no real
information present any longer
Start with only n=5 or n=10 replications, with m
being large (the larger the better)
If good estimate for the transient period was not
found (curve stays to edgy even for k=n/4), add
another 5 or 10 replications and repeat
Simulation 27
Batch Means
Run for long time, N observations
Divide up into batches
m batches size n each so m = N/n
Compute batch mean
Compute variance of batch means as
function of batch size




Start with batch size n=1, then increase n
Plot variance versus size n
When variance starts decreasing, have
transient
R
e
s
p
o
n
s
e
s

Observation number
n 2n 3n 4n 5n
V
a
r
i
a
n
c
e

o
f

b
a
t
c
h

m
e
a
n
s

transient
interval
Batch size n
(Ignore)
i
x
mean overall X
X x
m
x Var
m
i
i

); (
) 1 (
1
) (
1

=

=
Simulation 28
Why does batch means method work?
Assume: actual length of transient period is T
While batch sizes are small, several batches will
come from the initial transient These batches have
different means than batches from the steady state
Many different batch means, resulting in a large
variance of the set of batch means
As batch size increases, fewer and fewer batches
come from the initial transient
These fewer batches will have a smaller and smaller
impact on the variance
Eventually, when batch size exceeds T, only one batch
from initial transient


Simulation 29
Outline
Motivation and Terminology
Recall: sampling distribution and mean
Confidence intervals for mean and quintiles
Estimators and Correlation
Terminating simulations
Non-terminating Simulations:Steady state
estimation
Transient Removal
Stopping Rules
Independent Replications, Batch means and
Regeneration methods

Simulation 30
When to stop?
Suppose the initial transient has been determined
Suppose an estimate for the steady-state mean of a
random process is to be computed
What rule to apply to stop a simulation?
Too short: results are highly variable (depending on
initial conditions like RNG seed)
Too long: waste of resources

Simulation 31
When to stop
Accuracy criteria: run simulation until confidence
interval for the mean is sufficiently narrow
Straightforward from sample mean and its
variance
However, formulas only apply for samples
consisting of independent observations
Observations in simulations are usually correlated,
hence not independent
Applying formulas for independent observations
to non-independent ones results in
underestimation of the sample mean variance

Simulation 32
When to stop
Problem Statement: Find procedures that properly
estimate the variance of an interdependent set of values
Similar to the transient removal problem
Two basic approaches possible
Fixed sample size:
Run a simulation of a fixed size
Construct a confidence interval from the resulting data
Sequential procedures
Increment the length of simulation run until a given quality criterion is
met
Quality criterion: (relative) length of confidence interval, absolute or
relative error
Often based on fixed-sample-size approaches

Simulation 33
Fixed sample size procedures
Multiple replications
Independent Replications
Single replication
Batch means
Regeneration

Simulation 34
Outline
Motivation and Terminology
Recall: sampling distribution and mean
Confidence intervals for mean and quintiles
Estimators and Correlation
Terminating simulations
Non-terminating Simulations:Steady state
estimation
Transient Removal
Stopping Rules
Independent Replications, Batch means and
Regeneration methods

Simulation 35
Independent Replications
Procedure to compute a point estimate and confidence
interval for steady-state mean
Similar to terminating case, but only take into account
observations after initial transient
Procedure
Perform n independent simulation runs of length m each m
much larger than initial transient k
For each replication, compute average of all observations
after initial transient
These averages are IID random variables, hence, usual point
estimates and confidence intervals can be computed
Using multiple independent replications eliminates the
problem of dependencies within a single simulation

Simulation 36
Independent Replications: Formally
n
X
X
n
i
i
n

=
=
1

n j
k m
X
X
m
k i
j i
j
... 2 , 1 ,
1
,
=

=

+ =

( )
1

1
2
2

=
n
X X
S
n
j
n j
n
n
S
t X
n
n n
2
, 1 o

Point estimation of mean

Average for observations
Estimation of sample
variance
Confidence interval, where
t
n-1,a
is the 1-a/2-quantile of the
Student distribution with n-1 degrees
of freedom

Simulation 37
Independent Replications: Pros and Cons
Advantages
Reasonably good statistical performance
Easy to implement and understand
Widely applicable
Possible to use it for simultaneous estimations
Disadvantages
Separate runs required to compute initial transient (pilot
runs) and the actual data (production runs )
Handling different runs can be cumbersome
No possibility to predict the length of the confidence
interval
Using four times as many observations will roughly cut
interval length in half


Simulation 38
Batch Means as Stopping Rule (1 of 2)
Collect long run of N samples + k samples of
transient phase
Divide into m batches of n observations each
n large enough so little correlation between
For each batch, compute the mean
Compute the variance of this set of means, using the
mean of these means
Use mean of means and variance of means to
compute confidence interval
Use standard formulas
Procedure is like independent replications
More efficient that independent replications -
transient phase only once
Simulation 39
Batch Means as Stopping Rule (2 of 2)
How to choose batch size n? Want covariance of
successive means small compared to variance
Cov(x
i
, x
i+1
) = 1/(m-2)E[(x
i
-x)(x
i+1
-x)]
Start n=1, then double n
Example:
Size Cov Var
1 -0.187 1.799
2 0.026 0.811
4 0.110 0.420

64 0.00010 0.06066
Becomes less than 1%, so can use n=64 as batch size
Simulation 40
Regeneration (1/2)
Basic observation: There are systems that at some
points of time enter again and again their initial
state. Such systems are called regenerative
Example: In a queuing system, queue occasionally
becomes empty
After such point, the waiting times of all following
customers or the queue length do not depend on
the previous customers. The system has started
from scratch, it has regenerated
Simulation 41
Regeneration (2/2)
Such points are called regeneration points,
period between two points regeneration
cycle
Regeneration points are random , so the
duration of the regeneration cycles
Systems exhibiting long-term memory are
usually non-regenerative
Simulation 42
Regeneration points and cycles
Simulation 43
Regeneration: Output analysis (1/2)
Compute mean for each regeneration cycle
As means for different cycles are by
definition independent, standard procedures
for confidence intervals should be
applicable
Simulation 44
Regeneration: Output Analysis (2/2)
Regeneration cycles are usually of various random
length
Averaging over means of cycles does not produce
overall mean
Ratios of length of cycles must be taken into account
Computation of confidence intervals is more
complicated than with other procedures
In both replication/deletion and batch means, individual
datasets used to computed means were all of the same
size
Idea: Use absolute sums, not means

Simulation 45
Regeneration: Formally (1/2)
N j X Y
j
n
i
j i j
... 2 , 1 ,
1
,
= =

=
N j n X X
j
n
i
j i j
j
... 2 , 1 ,
1
,
=

=
=

Compute the mean of each

cycle j
Compute sum for all
cycles
Let X
i,j
be observation i from regeneration cycle j, j=1,..,N
Cycle j is n
j
observations long
Calculate mean cycle
length
N n n
N
j
j
=
=
1
Simulation 46
Regeneration: Formally (2/2)
( )
n
S
S N s S
Y
X
N
j
j Y
= =

=
; 1
1
2 2
( ) ) (or
1 1 1

= = =
= =
N
j
j
N
j
j
N
j
j
n Y X N n Y X
N n
S
t X
Y
n o , 1

Calculate the difference between

the observed sum and the
expected sum in the cycle
Calculate estimation of the
variance of these differences
Confidence interval, where
t
n-1,a
is the 1-a/2-quantile of the
Student distribution with n-1 degrees
of freedom

Compute overall mean
j j j
X n Y s

=
Simulation 47
Regeneration: Pros and Cons

Advantages
Removal of initial transients is not necessary
usually part of each regeneration cycle
Easy to understand and implement
Disadvantages
Cycles are often rather difficult to determine
In complex systems, regeneration points can be
very rare (many conditions have to be met
simultaneously)
Length is unknown in advance
Complicated checks can be necessary to test for
a regeneration point

Simulation 48
Comparison of Methods
1. Independent Replications
Makes multiple runs to estimate m; in each replication
observations during the warm-up period are lost and each
run should be long enough to obtain an accurate estimate
for m.
2. Batch means
Makes a single run to estimate m; key point is the choice
of the batch size to obtain uncorrelated batch means.
3. Regenerative Method
Makes a single run to estimate m. No warm-up period
needed if one starts at regeneration point and no problem
with correlated output; but the mean length of a
regenerative cycle may be very large