Performance

Engineering
QUEUEING
Prof. Jerry Breecher

Queueing Models

WHAT WE ARE DOING HERE

We are about to embark on:
Queueing Lingo

- all the definitions youll ever need!

Queueing

- especially the Single Queue in DETAIL

Analytical Models

- how to solve a number of complex problems

using the equations we already know and love.
Works especially well for open models.

Operational Laws

- drawing conclusions about limits - what to do

when we cant solve the math exactly.

Mean Value Analysis - Using the equations in an iterative fashion to

solve closed models.

Queueing Models

Queueing Models
This section is about being able to describe the behavior of queues.
Queues are certainly a prevalent object in computer systems, and our goal
here is to write the equations that describe them. The language well use
here is mathematical, but nothing really more complicated than algebra.

Queueing Models

Queueing Lingo
Goals:
To understand the random/statistical nature of computer data. We will
emphasize the non-deterministic.
To understand distributions for simulation purposes.
To impress your friends.
For an observation, two things matter:
The value measured.
When measured.
The occurrence of an event can give us the "when".

Queueing Models

Queueing Lingo
A STOCHASTIC PROCESS is a mechanism that produces a collection of
measurements which all occur, randomly, in the same range of values.
It
applies to the VALUE measured for an observation. The dictionary says,
"random, statistical".
Stochastic processes are well behaved phenomena which don't do things which are
unpredictable or unplanned for.
Examples:
Throwing 2 dice always gives numbers in the range 2 - 12.
Actions of people are unpredictable ( unless the range of values is made very large.)
Someone can always respond in a way you haven't predicted.

Queueing Models

Queueing Lingo
THE POISSON PROCESS applies to WHEN an observation is made. It looks
random; the arrival points are uniformly distributed across a time interval.
Poisson processes can be defined by:
Event counting

The distribution of the number of events occurring in

a particular time is a Poisson distribution.

Time between events

The distribution of times between event occurrences is

exponential.

Example: Show how a random "look" leads to an exponential distribution. See the
next page for a picture of these distributions.

Queueing Models

F(t) = exp(-t)

Queueing Lingo
Exponential Curve

1
0.9
0.8
0.7

F (t)

0.6
0.5

F(t)

0.4
0.3
0.2
0.1
0
0

0.5

1.5

2.5

This is a simple exponential curve. What properties can you identify from it?
Queueing Models

F(k) = ( 5k / k! ) exp( -5 )

Queueing Lingo
Poisson Probability Density Function

0.2
0.18
0.16
0.14

F (k )

0.12
0.1

F(k)

0.08
0.06
0.04
0.02
0
0

10

12

Example of the Poisson Probability Density Function.

Queueing Models

Lab - You Get To See It Happen

Before Your Very Eyes!
This is a group lab designed to demonstrate that the number of random
events in a particular interval is a Poisson distribution, and that the
distribution of spaces between events is exponential.
Using your laptop or Smartphone, go to random.org. Your goal is to get 20
random numbers in the range 0 49. Each time, write down the
number presented beside the appropriate Segment in column I.
Count how many of the segments in Column I have 0 samples in them. Put
this number in the first row of Column II. Repeat for 1, 2, 3, ... samples.
For each of the samples in Column I, determine the distance or separation
between that sample and the next higher sample. If the distance is 0
( the two samples are the same ) put a tick in the 0-interval row of
column 3, if the separation is 3, tick the interval = 3 row, etc.
Queueing Models

Lab - You Get To See It Happen Before

Your Very Eyes!
Column I
Numbers falling
in each segment.
Segment

04
59

How
Many
Samples

Column II
Number of segments
containing N samples
N

Number of
Segments

Column III
Number of instances of
intervals between samples.
Interval

Number of
Instances

10 - 14

15 - 19

20 - 24

25 - 29

30 - 34

35 - 39

40 - 44

45 - 49

Queueing Models

10

Lab - You Get To See It Happen Before

Your Very Eyes!
//GeneratePoisson.
//Thisprogramgeneratesrandomnumbers,putsthemintobuckets,
//andcalculatesthedistancebetweennumbers.
//
//Inputs:
//N-numberofsamplestogenerate
#include<time.h>
#defineMAX_DATA_VALUE100
#defineMAX_BUCKETS100
#defineMAX_DATA1000
//Compareroutineusedinsorting
intcompare(constvoid*a,constvoid*b){
return(*(int*)a-*(int*)b);
}

printf("\nRawRandomNumbers\n");
for(Index=0;Index<NumberOfSamples;Index++)
printf("%d",Data[Index]);
printf("\n");
qsort(Data,NumberOfSamples,sizeof(int),compare);
printf("\nSortedRandomNumbers\n");
for(Index=0;Index<NumberOfSamples;Index++)
printf("%d",Data[Index]);
printf("\n");
printf("\nRangeNumber\n");
printf("inRange\n");
for(Index=0;Index<NumberOfBuckets;Index++)
printf("%2d-%2d,%d\n",(MAX_DATA_VALUE/NumberOfBuckets*
Index),
(MAX_DATA_VALUE/NumberOfBuckets*(Index+1))-1,
Bucket[Index]);

//Calculatedistributionofitemsineachbucket
printf("\n");
for(Index=0;Index<LargestBucketFill;Index++){
Temp=0;
for(i=0;i<NumberOfBuckets;i++)
if(Bucket[i]==Index)Temp++;
printf("Numberofbucketswith%ditems,%d\n",Index,Temp);
}
//Determinethedistancebetweentherandomitems.
if(argc<2){
for(Index=0;Index<MAX_BUCKETS;Index++)
printf("Usage:GeneratePoisson<number_of_samples>\n"); Bucket[Index]=0;
exit(0);

}
LargestBucketFill=0;
NumberOfSamples=atoi(argv[1]);
for(Index=0;Index<NumberOfSamples-1;Index++){
srand((unsigned)time(NULL));
Temp=Data[Index+1]-Data[Index];
for(Index=0;Index<NumberOfBuckets;Index++)
Bucket[Temp]++;
Bucket[Index]=0;
if(Temp>LargestBucketFill)
for(Index=0;Index<NumberOfSamples;Index++){
LargestBucketFill=Temp;
Data[Index]=rand()%MAX_DATA_VALUE;
}
Temp=(NumberOfBuckets*Data[Index])/MAX_DATA_VALUE; printf("\nDistanceNumber\n");
Bucket[Temp]++;
printf("Betweenwiththis\n");
if(Bucket[Temp]>LargestBucketFill)
printf("Samplesdistance\n");
LargestBucketFill=Bucket[Temp];
for(Index=0;Index<=LargestBucketFill;Index++)
}
printf("%2d,%d\n",Index,Bucket[Index]);

}
intmain(intargc,char*argv[]){
intData[MAX_DATA];
intBucket[MAX_BUCKETS];
intNumberOfSamples;
intNumberOfBuckets=10;
intLargestBucketFill=0;
intIndex,Temp,i;

Queueing Models
Code is here so it doesnt get lost!!

11

Lab - You Get To See It Happen Before

Your Very Eyes!
Experimental Results
The results of running this code can be seen here
This is the result of running 4 experiments of 20 samples each.
The results arent pretty!! Perhaps by running the tests many times, everything would look nice and
smooth. You can see the results you would expect, but theres lots of jitter.

Queueing Models

12

Queueing Lingo
Examples:
Suppose that a piece of software has an expected lifetime of 5 years, and
that the average bug rate for this type of product is one bug/year.

What is the bug expectation rate per year at the start of the five years,
assuming this code is "average"?

After two years, four bugs have been found. The code is still
considered "average". How many bugs/year can be expected for the
remaining three years?

Queueing Models

13

Queueing Lingo
MEMORYLESS means that the probability of an event doesn't depend on its past.
The above case highlights an example where the past does matter.
Examples:
Which depend on the past, and which don't?
Throwing dice?
A disk seek distance?
The address of an instruction execution?
The measurement of the length of a table?

Prepare to Have Your Mind

Bent In An Unexpected
Way!!

Example:
Consider a bus stop where the time between
bus arrivals is exponentially distributed with a
rate L. Thus the bus arrivals form a Poisson
process. If you walk up to the bus stop, how
long do you have to wait until the next bus
arrives?
Queueing Models

14

Queueing Lingo
Example:
Consider a bus stop where the time between bus arrivals is exponentially
distributed with a rate L. Thus the bus arrivals form a Poisson process. If you
walk up to the bus stop, how long do you have to wait until the next bus
arrives?
1. Possible solution: Since buses arrive at a rate L, the average time between
arrivals is 1/L. But since we walk up at random, we would wait for only half an
interval on the average. So we would wait 1/(2L) for the next bus.
2. Possible solution: Since the time between buses is exponentially distributed, it
is memoryless. So the residual lifetime for any time that I arrive should be
distributed exponentially the same way as the original distribution. Since the
average time between buses is 1/L, the average time ( residual ) to wait
should also be 1/L.
Queueing Models

15

Queueing Lingo
This program generates 1 week's worth of bus arrivals.
We assume that the average bus arrival rate is every 12 hours,
The arrival rate is 1/12 per hour. We'll take 1 week's worth
of arrivals - we will assume that there are 14 arrivals in a week
- that's how we set the average arrival rate.
Oh - and we assume the buses arrive on the hour (it's simpler that way).
//GenerateRandomBusArrivals
#include<time.h>
#defineMAX_DATA1000
//Hoursinaweek
#defineMAX_DATA_VALUE168

Sorted Random Numbers

2 5 73 81 100 100 102 109 134 136 145 148 152 154

So given these actual bus

arrivals, how long on average
must you wait for a bus?

//Compareroutineusedinsorting
intcompare(constvoid*a,constvoid*b){
return(*(int*)a-*(int*)b);
}
intmain(intargc,char*argv[]){
intData[MAX_DATA];
intNumberOfSamples=14;
intIndex;
printf("Thisprogramgenerates1week'sworthofbusarrivals.\n");
printf("Weassumethattheaveragebusarrivalrateisevery12hours,\n");
printf("Thearrivalrateis1/12perhour.We'lltake1week'sworth\n");
printf("ofarrivals-wewillassumethatthereare14arrivalsinaweek\n");
printf("-that'showwesettheaveragearrivalrate.\n");
printf("Oh-andweassumethebusesarriveonthehour(it'ssimplerthatway).\n");
srand((unsigned)time(NULL));
for(Index=0;Index<NumberOfSamples;Index++){
Data[Index]=rand()%MAX_DATA_VALUE;
}
qsort(Data,NumberOfSamples,sizeof(int),compare);
printf("\nSortedRandomNumbers\n");
for(Index=0;Index<NumberOfSamples;Index++)
printf("%d",Data[Index]);
printf("\n");
}

Queueing Models

16

Queueing Lingo
Types of Stochastic Processes

Arrival rates are often Poisson.

Service times are often exponential.
( In other words, they're both random. )
Queueing Models

17

PROPERTIES OF QUEUES

Customer
Arrivals

The queue A
place where
customers are
stored before
being serviced.

Customer
Departures
The device doing
the actual service
of the customers.

Queueing Models

18

PROPERTIES OF QUEUES
How do we describe a queue? These are the important aspects:
Arrival process:
Service Time Distribution:
Number of Servers:
System Capacity:
Population Size:
Service Discipline:
The shorthand for queue description is thus

A / S / m / B / K / SD.

The inter-arrival and service times are typically of the following types:
M Exponential
D Deterministic
G General

Memoryless, the distribution weve just been talking about.

the times are constant and there is no variance.
distribution is not specified and the results are valid for all distributions
Queueing Models

19

THE SINGLE QUEUE

If we have a single queue obeying certain properties, we can get all kinds of
nice metrics. But, it must have those required properties!!
REQUIRED PROPERTIES:

Arrivals are random with a rate of X per time. ( Poisson when we say
this, we mean the inter-arrival time is exponentially distributed. ) [ Note that
in steady state, throughput = arrival rate.] Many texts use

for this.

Service times are random with a value of D. (Exponential ) [ Note this is

the Demand we've seen before.] Many texts use
service is = 1/D.

for this. The rate of

There's the possibility of an infinite number of customers.

There's a single server. [ So the derivation were about to do doesn't work

for a multiprocessor CPU.]
Queueing Models
20

THE SINGLE QUEUE

These are general requirements and hold for many practical applications.
Other analysis can be done for cases outside these requirements, but we
don't do it here.
We will apply a method called local balance to a system like that pictured on
the next page:
The queue is of type M / M / 1.

Queueing Models

21

THE SINGLE QUEUE

X=

X=

State with 0
in Queue

State with 1
in Queue

= 1/D

State with 2
in Queue

= 1/D

For simplification, in this particular case, the utilization U is related to

throughput and demand by

U=XD

(Remember N = XS)

Note:
N

pi = U ,

p0 = ( 1 U )
Queueing Models

22

THE SINGLE QUEUE

By Definition:
serviced.

A queue is defined to contain customers that are both waiting and being

In an equilibrium state, from the picture below, the following equations can be formed:

pi = p i-1
p p i
p p 0U p 0
The probability of having i customers in the queue is
pi = ( 1 U ) U i
[ Note that p0 = ( 1 - U ) so p i > 0 = U. But this is just the utilization we defined before.]

X=
State with 0
in Queue

X=
State with 1
in Queue

= 1/D

Queueing Models

State with 2
in Queue

= 1/D

23

THE SINGLE QUEUE

The average number of customers in the queue (waiting and being serviced) is

N U

(1 U )

From Little's Law ( N = X T ) in steady state, we can derive the average time spent at
the queueing center ( both in the queue and being serviced ). Note what
happens to this response time as the utilization increases!

T Rk D

(1 U )

Queueing Models

24

THE SINGLE QUEUE

Example:
Pat is designing a communications server that receives requests from "higher
level" routines. The requests are collected by a Request Handler that does
nothing but put them into buffers. These requests are removed from the buffers
by the Request Processor on a first come first serve basis.
requests -> Request Handler -> Buffers[n] -> Request Processor ->
The requests arrive randomly at a rate of 5/second. The Request Processor can
service 10 items per second from the buffer.
Since allocating buffers is an expensive business, Pat wants to preallocate an
adequate number of buffers so none need be allocated 99% of the time. Clearly
there's a tradeoff here between memory usage and time-to-allocate.
How many buffers should be preallocated?
Queueing Models

25

THE SINGLE QUEUE

This is the setup for the case of M / M / 2. The probability of transition from a
lower population to a higher one is the same as before ( arrivals are the same.)
But the probability of one of the TWO servers finishing is twice as great when
both of them are filled.

X=
State with
0 in Queue

X=
State with
1 in Queue

= 1/D

X=
State with
2 in Queue

Queueing Models

State with
3 in Queue

26

THE SINGLE QUEUE

The following equations hold for the single queue.

Utilization:

U =X D
the

pn = ( 1 U ) Un

Mean number of customers in the

system:

N = U/(1U)

Mean time spent in the system:

T = D/(1U)

Probability of finding n or more jobs in

the system:

Un

Mean number of jobs served in one

busy period:

1/(1U)

Mean busy time duration:

D/(1U)

Probability of n
system:

customers in

Queueing Models

27

ANALYTICAL MODEL
Goals:
You should be able to create, use, and understand a simple analytical model.
You should have a general idea of when such models are applicable and should
understand some of the buzzwords.

PERSPECTIVE:
An Analytical Model uses mathematical operations to capture the relationships
between observable quantities. The computations don't necessarily mimic real
actions as they do for simulations.
Examples:
The equations we've been using such as Little's Law.
Local Balance Equations - these enumerate the states the system can be in and
then determine the transitions between states. (This is what we just did with single
queues.)
Mean Value Analysis - an iterative approach using the equations we've already
learned.
Queueing Models

28

ANALYTICAL MODEL
Analytical models can be applied to:
Single Queues
Queueing Networks
Queueing Networks are networks of queues; two or more queues tied together.
They can be:

Open: - Typical of transaction processing. Jobs enter and leave the system being
studied.

Closed: - typical of batch or terminal systems. Jobs always remain somewhere

within the system.

Single Class - the customers are indistinguishable from each other; they have
similar service demands and routing characteristics.

Multiple Class - several categories of customers can be identified. A batch class,

for example, might be heavily CPU bound while a terminal class is I/O bound. To
use a multiple class model requires determining the characteristics for EACH of the
classes involved.
Queueing Models
29

ANALYTICAL MODEL
SINGLE CLASS OPEN QUEUEING NETWORK MODEL SOLUTIONS:
This is EASY!! It's simply an extension of the equations we used for the single
queue.
Utilization:

Uk =

X Dk

Throughput:

Xk =

X Vk

Max. Throughput:

Xmax =

Residence Time:

Rk =

(Throughput = arrivals)

Dmax

Dk
Dk

{Dk is the service time)

(delay centers)

(1 U k ) (queueing centers)

Queueing Models

30

ANALYTICAL MODEL
SINGLE CLASS OPEN QUEUEING NETWORK MODEL SOLUTIONS:
Queue Length:

Uk

Qk =

Uk

(delay centers)

(1 U k )

System Response
Time:

T=R=

Average Number
In System:

N=Q=

Qk

(queueing centers)

Remember,
N = Number of requests in the "system"
X = Throughput
R = Residence time per request.
S = Service Time
Little's law is 90% of all you'll ever need.
Queueing Models

We'll be using the FIGURE on

the next page and applying
these Laws to the system
shown there, at a number of
different levels.

31

ANALYTICAL MODEL
4

Terminals

3
CPU

2
1

DISK C

DISK A
DISK B
Queueing Models

32

ANALYTICAL MODEL
Box 1 in the Figure:
A single resource,
not including the queue.
Here the population, N, is either 1 or 0 ( in use or
not ).

DISK C

Utilization is equal to the average number of

requests present, or the average N.
Throughput X is the number of requests
serviced per time.
Residence time R is, for this case, the service Example:
Suppose a disk, serves 100
time.
requests/second, with the average
request needing 0.008 seconds of disk
[NOTE: Little's Law reduces to Utilization Law.]
service.

N=X*R
U=X*S
R = S/(1 U )

What is the utilization of the disk?

Queueing Models

33

ANALYTICAL MODEL

Box 2 in the Figure: A single resource,

including the queue.

2
Now the population includes both those
requests in the queue and in service.
Throughput remains the rate that the
resource satisfies requests.
Residence time is the sum
queueing and service times.

DISK C

of

Example:
N=X*R
U=X*S
R = S/(1 U )

You will need to use the result from the previous slide.
What is the time a request spends at the disk subsystem ( spindle + queue)?
What is the average number of requests in Box 2?
For one of these requests, what is the average queueing time, and what is the
average service time?

Queueing Models

34

Box 3 in the Figure:

Central
Subsystem, not including terminals.
The population now is the number of
users with activity in the system - those
not "thinking".
Throughput is the rate that requests
flow between system and users.
Residence time is now what we call
response time.

ANALYTICAL
MODEL
Example:
The average system throughput (entering
& leaving Box3) is 0.5/sec. There are an
average of 7.5 "ready" (waiting) users in
the Box.
What is the average response time?

3
CPU

2
DISK C

DISK A
DISK B
Queueing Models

35

ANALYTICAL MODEL
Box 4 in the Figure:
including terminals.
4

Terminals

Entire system,

The population is the total number of

users, both waiting and thinking.
Throughput is the rate that requests
flow between system and users.
(same as box 3).
Residence time is the sum
response time and think time.

Example:
There are 10 users with average
think time of 5 seconds, and the
system has average response time
of 15 seconds.
What is the throughput?
Queueing Models

36

of

ANALYTICAL MODEL
For system wide applications of Little's Law, since time represents both thinking
and waiting for a response,
NSS = X R
NES = X ( R + Z )

for Subsystem or lower.

for the Entire system.

RESPONSE TIME LAW: R = N/X - Z

Example:
A System has 64 interactive users, with an average think time of 30 seconds.
Two interactions complete on average each second.
What is the average response time for this system?
How many interactions are in the system at any time?
What sized System ( how many CPU's ) would be needed for this application?
Queueing Models

37

ANALYTICAL MODEL
Example Problems You Should Now Be Able To Do:
The average delay experienced by a packet when traversing a computer network is 100 msec. The
average number of packets that cross the network is 128 packets/sec.
What is the average number of packets in transit in the network?

Example Problems You Should Now Be Able To Do:

Measurements taken during one hour from a Web server indicate that the utilization of the CPU
and the two disks are: UCPU = 0.25, Udisk1 = 0.35, and Udisk2 = 0.30. The Web server log shows
that 21,600 requests were processed during the measurement interval.
What are the service demands (the time used by each request) at the CPU and both disks?
What is the maximum throughput,
and what was the response time of the Web server during the measurement interval?

Example Problems You Should Now Be Able To Do:

A computer system is measured for 30 minutes. During this time, 5,400 transactions are completed
and 18,900 I/O operations are executed on a certain disk that is 40% utilized.
What is the average number of I/O operations per transaction on this disk?
What is the average service time per transaction on this disk?

Solutions on a later page

Queueing Models

38

ANALYTICAL MODEL
Example Problems You Should Now Be Able To Do:
A file server is monitored for 60 minutes, during which time 7,200 requests are completed. The disk
utilization is measured to be 30%. The average service time at this disk is 30 msec per IO.
What is the average number of accesses to this disk per file request?

Example Problems You Should Now Be Able To Do:

A computer system has one CPU and two disks: disk 1 and disk 2. The system is monitored for
one hour and the utilization of the CPU and of disk 1 are measured to be 32% and 60%,
respectively. Each transaction makes 5 I/O requests to disk 1 and 8 to disk 2. The average
service time at disk 1 is 30 msec and at disk 2 is 25 msec.
Find the system throughput.
Find the utilization of disk 2.
Find the average service demands at the CPU, disk 1, and disk 2.

Example Problems You Should Now Be Able To Do:

An interactive system has 50 terminals and the user's think time is equal to 5 seconds. The
utilization of one of the system's disk was measured to be 60%. The average service time at the
disk is equal to 30 msec. Each user interaction requires, on average, 4 I/Os on this disk.
What is the average response time of the interactive system?
Queueing Models

Solutions on a later page

39

ANALYTICAL MODEL
Problem Solution:
The average delay experienced by a packet when traversing a computer network is 100 msec. The
average number of packets that cross the network is 128 packets/sec.
What is the average number of packets in transit in the network?
Straight usage of Littles Law: N = XS = 128 packets/sec * 0.1 sec = 12.8 packets

Problem Solution:
Measurements taken during one hour from a Web server indicate that the utilization of the CPU
and the two disks are: UCPU = 0.25, Udisk1 = 0.35, and Udisk2 = 0.30. The Web server log shows
that 21,600 requests were processed during the measurement interval.
What are the service demands (the time used by each request) at the CPU and both disks?
What is the maximum throughput,
and what was the response time of the Web server during the measurement interval?
There are 21,600 requests/hour = 60 requests/sec. During each second, the CPU is used 250
milliseconds so each of the 60 requests is using 4.16 milliseconds. Similarly the disks use
5.83 milliseconds and 5 milliseconds of service per transaction.
Maximum throughput is determined by the device having the highest utilization, Udisk1 = 0.35. When
that disk is maxd out, there will be 1 / 0.35 more traffic or 2.86 more. Thus the maximum
throughput will be 2.86 * 60 transactions/second = 171 transactions/second.
You can determine the total response time by calculating the response time at each queueing
center. This is 4.16 msec/(1 0.35) + 5.83 msec / ( 1 0.35) + 5.0 msec / ( 1- 0.3)
= 6.4 + 8.97 + 7.1 = 22.5 msec
Queueing Models

40

ANALYTICAL MODEL
Problem Solution:
A computer system is measured for 30 minutes. During this time, 5,400 transactions are completed
and 18,900 I/O operations are executed on a certain disk that is 40% utilized.
What is the average number of I/O operations per transaction on this disk?
What is the average service time per transaction on this disk?
Put everything into the same time units (seconds usually work best)
5,400 transactions / 30 minutes = 3 transactions/second
18,900 IO / 30 minutes = 10.5 IOs / second
So the number of IOs/transaction = 10.5 / 3 = 3.5 IOs / transaction
In each second, this disk is busy 400 milliseoncds of time. During a second, 3 transactions
complete. So the disk service time per transaction is 133 milliseconds.

Problem Solution:
A file server is monitored for 60 minutes, during which time 7,200 requests are completed. The disk
utilization is measured to be 30%. The average service time at this disk is 30 msec per IO.
What is the average number of accesses to this disk per file request?
The idea is that a file server request may result in multiple IOs to the disk its not a 1 to 1 match
necessarily. 7,200 requests / 3,600 seconds = 2 requests/second.
The disk is 30% busy so it runs for 300 milliseconds each second. The service time is 30
milliseconds, so 10 IOs are completed each second. 10 IOs for 2 requests means there are on
average 5 IOs / request.
Queueing Models

41

ANALYTICAL MODEL
Problem Solution:
A computer system has one CPU and two disks: disk 1 and disk 2. The system is monitored
for one hour and the utilization of the CPU and of disk 1 are measured to be 32% and 60%,
respectively. Each transaction makes 5 I/O requests to disk 1 and 8 to disk 2. The average
service time at disk 1 is 30 msec and at disk 2 is 25 msec.
Find the system throughput.
Find the utilization of disk 2.
Find the average service demands at the CPU, disk 1, and disk 2.
It turns out here that disk 1 is where we have the most information: For this disk, its busy 600
milliseconds out of each second; it takes 30 milliseconds for each IO; so there are 20 IOs/second.
Since each transaction makes 5 IO requests to disk 1, that means there are 4 transactions/second.
Once we have this answer, we know there are 32 IOs to disk 2 per second. Each of those IOs takes
25 milliseconds, giving a total time usage of 800 milliseconds. That means this disk is 80% utilized.

Problem Solution:
An interactive system has 50 terminals and the user's think time is equal to 5 seconds. The
utilization of one of the system's disk was measured to be 60%. The average service time at the
disk is equal to 30 msec. Each user interaction requires, on average, 4 I/Os on this disk.
What is the average response time of the interactive system?
The disk is doing 20 IOs / second. Each transaction is 4 IOs, so the throughput is 5 trans/sec.
To get this throughput requires each of the 50 users is executing a transaction every 10 seconds.
Thus the time in the system must be 5 seconds (because the user is already thinking for 5 secs.)
Queueing Models
42

ANALYTICAL MODEL
FORCED FLOW LAW:
The Forced Flow Law states that the flow in all parts of a system must be
consistent.
Suppose we count both system completions and also completions at each
resource. The visit count ( visit ratio ) is defined as:
Resource Completion:

Ck

System Completion:

Visit Count:

Vk = C k / C

THE FORCED FLOW LAW:

Queueing Models

X k = Vk X
43

ANALYTICAL MODEL
Example:
Suppose a system has:
30 terminals ( N = 30 ).
18 seconds average think time ( Z = 18 ).
20 visits to a specific disk/interaction (Vdisk = 20 ).
30% utilization of that disk (Udisk = 0.30 ).
25 millisecs is the average service required per visit to the disk (Sdisk =
0.025 sec.).
We want to know:
Disk throughput
System throughput
Response time

=
=
=

RESPONSE TIME LAW: R = N/X - Z

Queueing Models

THE FORCED FLOW LAW:

X k = Vk X

44

ANALYTICAL MODEL
Example:
Consider the problem of a spy from Burger King trying to figure out how many
people are at a McDonald's. The spy can't sit inside and watch all day, so must
somehow calculate the number from information obtained from outside
observations. Thirty customers/hour arrive on the average ( over a long period of
time ) and the average customer exits after 12 minutes.
Assuming that all this time is spent standing in line, what is the mean queue
length in the restaurant?
If the Standard Deviation of the 30 customers is 3, what is the uncertainty of the
queue length?
What happens if both the arrival rate and the service time have uncertainties?

RESPONSE TIME LAW: R = N/X - Z

Queueing Models

THE FORCED FLOW LAW:

X k = Vk X

45

ANALYTICAL MODEL
EXAMPLE OF THE SOLUTION OF AN OPEN MODEL:
Model Inputs:

Vcpu = 121
Vdisk1 = 70 Vdisk2 = 50
Scpu = 0.005
Sdisk1 = 0.030
Sdisk2 = 0.027
Dcpu = 0.605
Ddisk1 = 2.1
Ddisk2 = 1.35
= 0.3 jobs/sec

Model Structure:
Departures

Disk1

Arrivals
Disk2
CPU

Queueing Models

46

Solution Of An Open Model

EXAMPLE:

Model Inputs:
Vcpu = 121
Vdisk1 = 70
Vdisk2 = 50
Scpu = 0.005 Sdisk1 = 0.030 Sdisk2 = 0.027
Dcpu = 0.605 Ddisk1 = 2.1
Ddisk2 = 1.35
= 0.3 jobs/sec

Model Structure:
Departures

Disk1

Arrivals
Disk2
CPU

Model Outputs:
= 1 / Dmax = 1 / 2.1 = 0.476 jobs/sec
Xcpu (0.3) = Vcpu

= (0.3)(121)

= 36.3 visits/sec

Ucpu (0.3) = Dcpu

= (0.3)(0.605)

= 0.182

Rcpu (0.3) = Dcpu / ( 1 Ucpu(0.3) ) = 0.605 / 0.818 = 0.740 secs

Qcpu (0.3) = Ucpu(0.3) / ( 1 Ucpu(0.3) )

= 0.182 / 0.818 = 0.222 jobs

R(0.3)

= 0.740 + 5.676 + 2.269 = 8.685 secs

Q(0.3)

= Rcpu(0.3) + Rdisk1(0.3) + Rdisk2(0.3)

= R() = (0.3) (8.685)

= 2.606 jobs
Queueing Models

47

SUMMARY OF
PERFORMANCE METRICS
T
A
C
W

is the length of TIME we observed the system.

is the number of request ARRIVALS observed.
is the number of request DEPARTURES observed.
is the ACCUMULATED TIME for all requests within the system time
spent both waiting for and using resources.
B
is the length of time that the resource was observed to be BUSY.
Z
is the think time of a terminal user.
Vk the visit ratio, is the number of times device k is visited per
transaction.
Arrival Rate
Y = A/ T
UTILIZATION LAW
U=XS
Throughput (Departure Rate)
X=C/T
LITTLE'S LAW
N=XR
Utilization
U=B/T
Service Requirement
S=B/C
RESPONSE TIME LAW R = N/X Z
Requests in system
N=W/T
THE FORCED FLOW LAW: Xk = Vk X
Residence time
R=W/C
Queueing Models

48

OPERATIONAL LAWS
Goals:
To increase facility using performance metrics.
To be able to calculate limits or boundaries on performance metrics.
To be able to do "back of the envelope" calculations.

UTILIZATION LAW

U=XS

LITTLE'S LAW

N=XR

RESPONSE TIME LAW

R = N/X Z

THE FORCED FLOW LAW:

Queueing Models

Xk = Vk X

49

OPERATIONAL LAWS
BOUNDARY VALUES:
The goal here is to make estimations of the outside limits of a performance
parameter. We do this by selective "blind" application of our simple
laws to complex systems; our results give upper and lower bounds, not
an exact answer.
Example:
An editor program wants to read 100 disk pages into memory. The resources
required for each disk read are:
5 milliseconds of CPU Processor time.
4 milliseconds of Controller Processor time.
2 milliseconds of SCSI Handshaking time.
10 milliseconds of DISK seek/rotation/transfer time.

What is the bottlenecking device? Warning!! This is a trick question!

What is the worst throughput for this system, assuming the editor single threads the
reads - waiting until each read completes before starting the next?
What is the best throughput for this system? How long will it take to read in the 100
disk blocks?
Queueing Models

50

OPERATIONAL LAWS
BOUNDARY VALUES:
Now that we know the limits for one user, let's expand this for N users.
Example:
Multiple users, each with their own copy of the editor, are reading files from the
same disk (parameters are the same as the last example.)

What is the bottleneck device in this case?

Considering only the bottleneck device, what is the best throughput

achievable?

Now taking into account all devices, what is the BEST throughput N users can
achieve ( assuming the requests NEVER get in each other's way?)

Now taking into account all devices, what is the WORST throughput N users
can achieve ( assuming the requests ALWAYS get in each other's way?)
Queueing Models

51

OPERATIONAL LAWS
BOUNDARY VALUES:
We define DEMAND as the amount of a resource used for a transaction; this is just
the number of visits to that resource during one transaction, and the service
time used at each of those visits:
Dk

= Vk * S k

Dtotal

=D

Dmax

= Dbottleneck

= Dk

SUMMARY:
Best throughput overall:

X = 1 / DBottleneck

Best throughput for N users:

X = N/D

Worst throughput for N users:

X = N/(N*D) = 1/D

Queueing Models

52

OPERATIONAL LAWS
Exercise:
Consider an interactive system
with a CPU and two disks. The
following measurement data was
obtained
by
measuring
the
system:

----------- Observation Data ---------------Observation interval

30 minutes
Active terminals
30
Think time
12 seconds
Completed transactions 1,600
Fast disk accesses
32,000
Slow disk accesses
12,000
CPU busy
1,080 seconds
Fast disk busy
400 seconds
Slow disk busy
600 seconds

Determine the visit counts Vk,

service times per visit Sk, and
service demands Dk at each
center.
Data At Centers
CPU
Fast Disk
Slow Disk
D = Dk

Vk

--

Sk

Dk

-Queueing Models

53

OPERATIONAL LAWS
Exercise:
Give optimistic and pessimistic
asymptotic
bounds
on
throughput and response time
for 10 and 40 active terminals.

----------- Observation Data ---------------Observation interval

30 minutes
Active terminals
30
Think time
12 seconds
Completed transactions 1,600
Fast disk accesses
32,000
Slow disk accesses
12,000
CPU busy
1,080 seconds
Fast disk busy
400 seconds
Slow disk busy
600 seconds

Bounds on Throughput and Response Time

Metric
-- 10 -Optimistic X
Pessimistic X
Optimistic R
Pessimistic R

Terminal Count
-- 40 --

1 / D max
N / {D+Z}
N/{N*D+Z}
D
N * D max - Z
N*D
Queueing Models

54

OPERATIONAL LAWS
Exercise:
Consider the following changes
to the system:
Move all files to the fast disk.
Replace the slow disk by a
second fast disk.
Increase the CPU speed by
50% (with the original disks.)
Increase the CPU speed by
50% and balance the disk load
across 2 fast disks.
Rank the
changes.

efficacy

of

----------- Observation Data ---------------Observation interval

30 minutes
Active terminals
30
Think time
12 seconds
Completed transactions 1,600
Fast disk accesses
32,000
Slow disk accesses
12,000
CPU busy
1,080 seconds
Fast disk busy
400 seconds
Slow disk busy
600 seconds

What changes should be made so

response time will not exceed 10
seconds?

these
Queueing Models

55

MEAN VALUE ANALYSIS

This is an extension of the Operational Analysis we did before. This is the method that should
be used on closed systems.
Assumptions:
Applies to various service disciplines and service time distributions. (We will only do
fixed capacity centers here.)
The arrival rate is not dependent on the load.
Arrivals are a Poisson process.
We will use a technique called Mean Value Analysis that is based on the repetitive application
of three equations:
1. Little's Law applied to the whole network:
2. Little's Law applied to each service center:
3. The service center residence time equations:
Menasce, et. al. do this in Section
12.3 of the text.

A great book on this, now out of print is at

http://www.cs.washington.edu/homes/lazowska/qsp/
Queueing Models
56

MEAN VALUE ANALYSIS

1. Little's Law applied to the whole network:
N
X (N )
K
Z Rk ( N )
k 1

where X(N) is the system throughput, Z is the think time, and Rk (N) the residence time at center k when
there are N customers in the network.

2. Little's Law applied to each service center:

Qk ( N ) X ( N ) Rk ( N )
3. The service center residence time equations:

Rk ( N ) Dk

Rk ( N ) Dk 1 Ak ( N )

Delay Centers
Queueing Centers

where Ak (N) is the average number of customers seen at center k when a new customer arrives.
The number seen on arrival (in equilibrium) is equal to the average queue length when there is one less
customer in the system! So the equations bootstrap!!!

Ak ( N ) Qk ( N 1)
Queueing Models

57

MEAN VALUE ANALYSIS

////////////////////////////////////////////////////////////////////////////////
//Thisprogramdoesverysimplemeanvalueanalysis.
////////////////////////////////////////////////////////////////////////////////
#defineMAX_NUMBER_OF_CENTERS30
intnumber_of_customers;
intnumber_of_centers;/*#ofcentersrequestedbytheuser.*/
intnumber_of_customers;/*Thepopulationineachclass*/
floatinput;
doublethink_time;/*Thetimespentattheterminal*/
doubledemand[MAX_NUMBER_OF_CENTERS];/*Servicedemandofacenter.*/
doubleq_length[MAX_NUMBER_OF_CENTERS];
doubleresidence_time[MAX_NUMBER_OF_CENTERS+1];//Residencetimeatacenter.
doublethroughput;
doublesystem_response_time;
//Prototypes
voidprint_the_numbers(intn);

This program is at: http://web.cs.wpi.edu/~jb/perf/Lectures/mva.c

Queueing Models

58

MEAN VALUE ANALYSIS

main(){
charinput_string[80];
intcenter,n;
////////////////////////////////////////////////////////////////////////////
//Gettheinputinformationfromtheuser.
////////////////////////////////////////////////////////////////////////////
printf("Numberofusers:");gets(input_string);
number_of_customers=atoi(input_string);
printf("Numberofcenters:");gets(input_string);
number_of_centers=atoi(input_string);
printf("ThinkTime:");gets(input_string);
sscanf(input_string,"%f",&input);
think_time=(double)input;
for(center=0;center<number_of_centers;center++)
{
printf("Demandforcenter%d:",center);gets(input_string);
sscanf(input_string,"%f",&input);
demand[center]=(double)input;
}
////////////////////////////////////////////////////////////////////////////
//Nowthatthenetworkisdescribed,weperformtheevaluation.
//Beginbyinitializingtothetrivialsolutionfor0customers.
////////////////////////////////////////////////////////////////////////////
for(center=0;center<number_of_centers;center++)
{
q_length[center]=0;
}

Queueing Models

59

MEAN VALUE ANALYSIS

////////////////////////////////////////////////////////////////////////////
//Thealgorithmnowiteratesthrougheachpopulationvaluen.
////////////////////////////////////////////////////////////////////////////
for(n=0;n<=number_of_customers;n++)
{
system_response_time=0;
for(center=0;center<number_of_centers;center++)
{
residence_time[center]=demand[center]*(1.0+q_length[center]);
system_response_time+=residence_time[center];
}
////////////////////////////////////////////////////////////////////////////
//NextuseLittle'sLawtocomputethesystemthroughput.
////////////////////////////////////////////////////////////////////////////
throughput=n/(think_time+system_response_time);
////////////////////////////////////////////////////////////////////////////
//FinallyuseLittle'sLawtocomputecenterqueuelengths.
////////////////////////////////////////////////////////////////////////////
for(center=0;center<number_of_centers;center++)
{
q_length[center]=residence_time[center]*throughput;
}
print_the_numbers(n);
}/*ENDOFLOOPTHROUGHPOPULATION*/
exit(0);
}/*ENDOFMAIN*/

Queueing Models

60

MEAN VALUE ANALYSIS

////////////////////////////////////////////////////////////////////////////////
//Printtheresultsofeachloop
////////////////////////////////////////////////////////////////////////////////
voidprint_the_numbers(intn){
inti;
printf("\n");
printf("Iteration:%dSystemResponseTime:%fThroughput:%f\n",
n,system_response_time,throughput);
for(i=0;i<number_of_centers;i++)
printf("Center:%dResidenceTime:%fQueueLength:%f\n",
i,residence_time[i],q_length[i]);
}//ENDOFPRINT_THE_NUMBERS

Queueing Models

61

MEAN VALUE ANALYSIS

EXAMPLE OF THE SOLUTION OF A CLOSED MODEL:
Model Inputs:
Model Structure:

Vcpu = 121
Vdisk1 = 70 Vdisk2 = 50
Scpu = 0.005
Sdisk1 = 0.030
Sdisk2 = 0.027
Dcpu = 0.605
Ddisk1 = 2.1
Ddisk2 = 1.35
= 0.3 jobs/sec
Disk1

Terminals

Disk2
CPU

The INPUT numbers are the same as were used for the open model. We modify the picture so there are
no arrivals or departures, but instead there are terminals the jobs stay in the picture (which is what
makes it closed.)
There are 3 terminal users with average think Queueing
time of 15Models
seconds;

N = 3,

Z = 15.

62

Solution Of An Open Model

EXAMPLE:

X (N )

Model Inputs:
Vcpu = 121
Vdisk1 = 70
Vdisk2 = 50
Scpu = 0.005 Sdisk1 = 0.030 Sdisk2 = 0.027
Dcpu = 0.605 Ddisk1 = 2.1
Ddisk2 = 1.35
= 0.3 jobs/sec

N
K

Z Rk ( N )
k 1

Qk ( N ) X ( N ) Rk ( N )

Rk ( N ) Dk
Rk ( N ) Dk 1 Ak ( N )

To start with, we initialize the queue lengths to 0:

Qcpu

= Qdisk1

= Qdisk2

=0

Iteration Number 1:
Rcpu = 0.605 sec
Rdisk1 = 2.1 sec
Rdisk2 = 1.35 sec.
X = 1 / ( 15 + 4.055 )
= 0.0525
Qcpu = 0.0318
Qdisk1 = 0.1102
Qdisk2 = 0.0708
Iteration Number 2:
Rcpu = 0.624 sec
X = 0.1030
Qcpu = 0.0643

Rdisk1 = 2.331 sec

Rdisk2 = 1.446 sec.

Qdisk1 = 0.2403

Qdisk2 = 0.1490

Queueing Models

Ak ( N ) Qk ( N 1)

63

Solution Of An Open Model

EXAMPLE:

X (N )

Model Inputs:
Vcpu = 121
Vdisk1 = 70
Vdisk2 = 50
Scpu = 0.005 Sdisk1 = 0.030 Sdisk2 = 0.027
Dcpu = 0.605 Ddisk1 = 2.1
Ddisk2 = 1.35
= 0.3 jobs/sec

N
K

Z Rk ( N )
k 1

Qk ( N ) X ( N ) Rk ( N )

Rk ( N ) Dk
Rk ( N ) Dk 1 Ak ( N )

Iteration Number 3:
Rcpu = 0.644 sec
X = 0.1515
Qcpu = 0.0976

Rdisk1 = 2.605 sec

Rdisk2 = 1.551 sec.

Qdisk1 = 0.3947

Qdisk2 = 0.2350

Ak ( N ) Qk ( N 1)

From these numbers we can calculate:

X(3)
R(3)
Q(3)

=(N/X) - Z
= N - X(3) Z

= 0.1515
= 4.74
= 0.72

Theres a very nice example on Jain, pp 578-9.

Queueing Models

64

CONCLUSION
This section has discussed:
Queueing Lingo

- all the definitions youll ever need!

Queueing

- especially the Single Queue in DETAIL

Analytical Models

- how to solve a number of complex problems

using the equations we already know and love.
Works especially well for open models.

Operational Laws

- drawing conclusions about limits - what to do

when we cant solve the math exactly.

Mean Value Analysis - Using the equations in an iterative fashion to

solve closed models.

Queueing Models

65