SIMULATION MODELING AND ANALYSIS WITH ARENA

T. Altiok and B. Melamed

Chapter 8
Model Goodness: Verification and Validation
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 1

Verification and Validation

Verification assesses the correctness of the formal representation of the intended model (in our case, a computer simulation program), by inspecting computer code and test runs, and performing consistency checks on their statistics Validation assesses how realistic the modeling assumptions are, by comparing model performance metrics (predictions), obtained from model test runs, to their counterparts in the system under study
obviously, validation is possible only if the simulation program exists

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

Model Verification via Test Runs

Verification activities via inspection of test runs include
input parameters and output statistics using a debugger using animation

sanity checks

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

Model Verification via Performance

WIP inventory Processor Job Arrivals Job Departures

A simple workstation

A Generic workstation can be described as a queueing system the flow of incoming jobs, either singly or in batches, forms the arrival stream the time a job is delayed for processing at the workstation server is its service time. if an incoming job cannot start processing right away (because servers are busy),
then it is held in a buffer (finite or infinite), and in due time will be removed from the buffer and assigned a server (real-life workstations it is, of course, finite) a job that finds the buffer full on arrival is usually rerouted to another workstation or simply assumed to be lost the order in which jobs queue up in the buffer is called a queueing discipline. a workstation model may have additional wrinkles, such as server failure and repair, routinely modeled via random uptimes and random downtimes
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 4

Queueing Conventions and Notation

The arrival process is denoted by {A } , where A is the i-th interarrival i i interval, separating customers i-1 and i
for stationary arrival processes, the rate is denoted by l = 1 / E [A ]
i

The service process is denoted by { X } , where X i

is the i-th service time

i
i

for stationary service processes, the rate is denoted by m = 1/ E [X ]

A standard notation has been developed to specify a queueing system succinctly, employing slash-separated symbols, representing arrival process, service process, number of servers and queue capacity in this order (an omitted capacity is understood to be infinite)
the symbol GI stands for a general iid distribution (special cases are M for an exponential distribution and D for a deterministic distribution) the symbol G stands for a general distribution non- iid distribution M/M/1 specifies a queue with iid exponential distributions for inter-arrival and service times (Poisson process), a single server and infinite buffer capacity M/D/k/C specifies a queue with iid exponential inter-arrival times, deterministic service times, k servers and a system of capacity of C, of which the first k positions are occupied by servers
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 5

Service Disciplines
Common service disciplines:
FIFO (First-In-First-Out), also known as FCFS (First-Come-FirstServed) is the most common discipline, where jobs are served in order of arrival LIFO (Last-in-First-Out), also known as LCFS (Last-Come-FirstServed), serves the most recent arrival first SIRO (Service-In-Random-Order) selects the next job in the buffer randomly, with equal probabilities RR (Round Robin) has associated with it a (fixed) service time, often referred to as the time quantum, and jobs are served cyclically, one quantum at a time, until attaining their requisite service time PS (Priority-Service) assumes that jobs have priorities associated with them, and selects a job with the highest service priority jobs, such as FIFO, LIFO, SIRO, etc.
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 6

Queueing Performance Measures

Common queueing performance measures:
average number of jobs in the queue (buffer only) average number of jobs in the system (buffer and servers) average job waiting time (buffer only delay) average job sojourn time (buffer and service delays) server utilization (fraction of time a server is busy) throughput (output rate of the system)

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

Average Number of Jobs

Denote the following quantities as follows:

N s is the average number of jobs in the system, including queue (buffer) and servers
N q is the average number of jobs in the queue (buffer) only

For a queueing system in steady state of capacity (buffer size) C (finite or infinite),
N S = n Pn
n=1 C

where P n denotes the steady-state probability of n jobs in the system

For a single-server queueing system,

N q = N s - (1 - P 0 ) where 1 - P 0 is the steady-state average number of jobs in service
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 8

Queueing System Stability

In general, the server utilization, r , of a single-server queueing system with any queue capacity is given by r = 1 - P 0
For infinite capacity systems, the utilization can be expressed as the ratio of the input rate and the service rate, that is, r = l / m

When r 1, then system is said to be unstable, and the server is said to be exhausted
In this case, the server is not able to keep up with the incoming workload, and consequently the number of jobs in the system grows without bound in the long run (system explosion)

When r < 1, then system is said to be stable

stability is needed for long-run performance measures to exist in the context of simulation modeling we are typically interested in stable systems a finite-capacity system is always stable, because the number of jobs in it is bounded by its capacity.
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 9

Regenerative Queueing Systems

A regenerative queueing system is one where temporal histories can be decomposed into periods (random intervals), such that the associated stochastic processes are independent over distinct periods
the onset of a busy cycle is called a regeneration point, since the system regenerates itself there probabilistically each independent period is called a regeneration cycle (or busy cycle), and extends over a pair of consecutive busy period (when the server is busy) and idle period (when the server is idle), as shown below
Number In the System Idle Empty-and-Idle Busy T ime

Empty-and-Idle

Regeneration cycles consisting of successive idle and busy periods

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 10

Regeneration Cycles and utilization

Consider a regeneration cycle in a GI/GI/1 queue, and define
T B is the length of the busy period T
C

is the length of the regeneration cycle

Since the number of incoming and outgoing jobs in a regeneration cycle must coincide, one has
l E [T ] = m E [T ] or equivalently E [T ]/ E [T ] = l / m = r
C B B C

It follows that Pr{server is busy } = r = l / m For Poisson arrivals (iid exponential interarrival intervals)
Since residual interarrival intervals are exponential too, 1/ l Pr{server is idle } = 1 - r = E [T ] C E [X ] E [A ] 1/ m 1/ l i i = = Thus, E [T B ] = and E [T C ] = 1- r 1- r 1- r 1- r
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 11

Queuing System Throughput

The throughput of a queueing system is its long-run output rate (jobs served per unit time), given by
o = m(1 - P 0 )

which is the service rate while the server is busy For an infinite capacity queue, this becomes
o = l

by flow conservation

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

12

Littles Formula
Littles Formula is a key general conservation equation, linking the mean number of jobs in the system, N , with the arrival rate, l , and the mean waiting time,W , given by
N = l W

note carefully that l is not the offered arrival rate, but the effective arrival rate, excluding any lost jobs! when the capacity is infinite (so no losses are incurred), then the offered arrival rate and effective arrival rate are the same

The formula holds for any queueing subsystem

for the entire queueing system (buffer plus servers), N for the buffer subsystem only, N
b
S

= l W

= lW

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

13

Steady-State Flow Conservation

Define the following quantities:

p is the long-run fraction of jobs that find k jobs in the system on k arrival. Such quantities are referred to as customer-average probabilities arrival point probabilities. Note carefully that the number of jobs in the system is computed only at the (random) times of arrival and excludes the arriving job. For this reason this kind of system state is referred to as the state embedded at arrival times, or as the state seen by arriving jobs.

P is the long-run fraction of time that the system has precisely k k jobs in it. Such quantities are referred to as time-average probabilities or arbitrary time probabilities.

Generally p k Pk
For single-server queues with finite capacity, K ,
l (1 - p K ) = m(1 - P 0 )
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 14

The PASTA Property

While generally p k Pk , under certain conditions the equality
p = P , k = 0, 1, K ,

does hold

The most common case where the equality above holds is when the arrival stream is a Poisson process
This case is known as PASTA (Poisson Arrivals See Time Averages)

The general case when this equality holds is known as

ASTA (Arrivals See Time Averages) It is possible to have ASTA which is not PASTA

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

15

Model Verification: Single Workstation

As an example, consider an Arena model of a single workstation below

Arena model of a single workstation model

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 16

Single Workstation Queue Statistics

Arena Queues summary statistics report for the single workstation model
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 17

Single Workstation Resource Statistics

Arena Resources Summary statistics report for the single workstation model

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

18

Single Workstation User Statistics

Arena User Specified statistics report for the single workstation model
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 19

Single-Station Verification
For a replication r, define

D (r ) is the number of departures (jobs processed) during replication r

T (r ) is the length (duration) of replication r

the throughput estimator over that period in that replication is

o 1 (r ) =

D (r ) T (r )

The verification procedure to be outlined uses various sections of the Arena summary reports to estimate the throughput in multiple alternative computations, to show that they are all in reasonable agreement
note that is just circumstantial evidence of verification any major discrepancy among the computations would suggest an error in the simulation program (bug) and will preclude verification
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 20

Verification via Throughput

Throughput estimation from the Queues report and the Counter section in the User Specified report
an examination of the previous report shows that replication 1 ran for 100,000 hours, and the Counter section reveals that D(1) = 20,736 jobs were processed and 1,503 jobs were rejected. the first estimate of the throughput is
o 1(1) = 20, 736 / 100, 000 = 0.2074 jobs/hour

Throughput estimation from the Resources section and the throughput equation
an examination of the Resources report shows that the utilization was estimated as u = 0.8358 , and while the workstation was busy, it processed jobs at the rate of 0.25 jobs per hour by the throughput equation o = m(1 - P 0 ) , a second estimate of the throughput is o 2 (1) = mu = 0.25 0.8358 = 0.2090 jobs/hour
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

21

Verification via Throughput (Cont.)

Throughput estimation from the Tally section in the User Specified report
using the mean inter-departure time from the Tally section, a third estimate of the throughput is
o 3 (1) = 1 E int er depar t ur e t im e = 1 / 4.8226 = 0.2074 jobs/hour

Throughput estimation from the Counter section in the User Specified report and the effective arrival rate
the Counter section shows that the number of jobs that succeeded in entering the workstation was 20,736 out of a total of 20736+1503 jobs that arrived at the workstation, so that we estimate 1- p = 20736 / ( 20736 + 1503) = 0.9324 K since the throughput is the effective arrival rate, a fourth estimate is just o 4 (1) = l (1 - p K ) = (1/ 4.5) 0.9324 = 0.2072 jobs/hour
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 22

Verification via Littles Formula

Littles formula provides another common verification check, using the summary statistics in the previous report as follows:
for the buffer subsystem, Littles formula is N = l W
b eff b

from the Queues report of Figure 8.4, we have N

from the Queues report of Figure 8.4, we also have W

= 1. 3265
b

= 6. 3969

from the Counter Section of the User Specified report, the = 0.2074 effective arrival rate (equal to throughput ) is l
eff

computing the right-hand side of Littles formula yields

since the values of N

eff

= 0.2072 6.3969 = 1.3254

b

= 1. 3265 and l

we conclude that the simulation agrees with Littles formula

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 23

eff

= 1.3254 are close,

Model Verification: A Tandem Network

As an example, consider the Arena model of a network of two workstations (assembly and painting) in tandem, shown below
Assembly Painting

suppose that neither workstation has buffer capacity limitations assume further that the arrival stream consists of two types of jobs: type 1 and type 2, each with its own assembly and painting operations job inter-arrival times are exponentially distributed with means 4 hours for type 1 and 10 hours for type 2. for type 1 jobs, assembly times (in hours) are distributed according to the Tria(1,2,3) distribution, while painting times are deterministic lasting 3 hours for type 2 jobs, assembly times (again in hours) are distributed according to the Tria(1,3,8) distribution, while painting times are deterministic lasting 2 hours this model is simulated for 100,000 hours to obtain resource utilizations and mean flow times by job type, and average WIP (work-in-process) levels in each buffer
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 24

Tandem Network Arena Model

Arena model of the tandem flow line

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 25

Tandem Network Arena Model (Cont.)

Spreadsheet view of the Resource module with StateSet module names

Dialog box for the StateSet module defining the states of each resource
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 26

Tandem Network Arena Model (Cont.)

Dialog box and spreadsheet view for the Record module tallying flow times
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 27

Tandem Network Arena Model (Cont.)

Dialog box for the Statistic module with Frequency statistics

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 28

Tandem Network Resources Statistics

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

29

Tandem Network Frequencies Statistics

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

30

Tandem Network Queues Statistics

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

31

Tandem Network User Specified Statistics

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

32

Verification via Utilization

Theoretical utilizations at the Assembly workstation
the total workstation utilization is u = 0.9 the partial utilizations of type 1 jobs is u (1) = 2 / 4 = 0.5 the partial utilizations of type 2 jobs is u ( 2) = 4 / 10 = 0.4 the sum of utilizations checks, since
u = u (1) + u ( 2) = 0.5 + 0.4 = 0.9

Simulation utilizations at the Assembly workstation

the simulation results from the report estimate the utilization of the = 0.889 assembly workstation as u .

= 0.889 , are close, The two figures, u = 0.9 and u but would be even closer if we were to run the model longer

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

33

Verification via Littles Formula

A verification via Littles formula is carried out by the following procedure:
the total arrival rate is l = 0.25 + 0.1 = 0.35 The simulation estimates of average flow time of type 1 and 2 jobs

(1) = 31. 349 and W ( 2) = 30. 557 are, respectively,W

S S

The estimated combined average flow time over all job types (weighted by the relative arrival rates) is

The average number of jobs in the system is estimated from the simulation run as the sum of average buffer sizes plus workstation utilizations over all job types, namely,

= ( 0. 25 / 0. 35) 31. 349 + ( 0. 1 / 0. 35) 30. 556 = 31. 122

(k ) + u = [N (k )] = b
k

4.1456 + 4. 8111 + 0. 88924 + 0. 94098 = 10. 787 = 0. 35 31.122 = 10.893 = 10.787 and l W Thus, we find that N
agree closely, in agreement with Littles formula
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 34

Model validation
Validation activities are critical to the construction of credible models The standard approach to model validation is to collect data (parameter values, performance metrics, etc.) from the system under study and compare them to their model counterparts
the data collection effort of Input Analysis can provide the requisite data from the system under study
data collected is classified into input values and corresponding output values

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

35

Model validation (Cont.)

Consider a generic system
Raw input data is obtained by observing the system over a time period

A model is constructed from the observed data and run to produce simulation data (that serve as input data sets) from which the corresponding output data (performance metrics) are then estimated
The correspondence of the generic inputs, I , and generic outputs, O , k k is shown below
Period 1 2 Input Data Set I1 I2 Output Metric Set O1 I2

M
N

M
IN Generic input and output data
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

M
ON

36

Model validation (Cont.)

Each set of output metric values constitutes a sample of random values (say, daily mean delay times) two sets (samples) of output daily mean delay data are collected for days i = 1K , N
1. a sample {D , K , D } of observed daily mean delays, 1 N
collected from the real-life system under study 2.
,K , D } of estimated daily mean delays, a sample { D 1 N

collected from runs of the simulation model of the system under study

Thus,
for the real-life system under study, O i = D i , i = 1, K , N
, i = 1, K , N for the simulation runs, O = D i i
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 37

Model validation (Cont.)

To test that the observed and simulated delays have the same means, , i = 1, K , N , define the statistics G i = D i - D i assumed approximately normally distributed with mean m G 2 and variance s G H0 : m =0 G Perform hypothesis testing of the form H : m 0 G 1 G- m G with the test statistic t N - 1 = , where SG / N t N - 1 has a t distribution with N - 1 degrees of freedom G and S are the sample mean and sample standard deviation,
G

respectively, of {G , K , G } 1 N The confidence interval of the test at significance level a is

Pr{G t a / 2,N 1 SG / N m G t 1a / 2,N 1 SG / N } 1 a G

The null hypothesis cannot be rejected if the confidence interval above contains 0, supporting the model is valid
Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 38