February 24-26, 2009

Sample-path Coverage: Stochastic Model

Application for Random Testing
by
Victor Besyakov
Tundra Semiconductor Corporation

1
Agenda
• Introduction
– DUT example
– motivation
– main idea
• Random Generator: Sample-path Probabilistic Model
• Sample-path Probabilistic Metrics
• Sample-path Coverage Framework
• Results and Conclusions

“The generation of random numbers

is too important to be left to chance.”
R. Coveyou

Victor Besyakov, Tundra Semiconductor Corporation 2 of 9

Introduction (DUT example):

• Transaction Attributes : type (Read, Write, Int), size of data payload, port
destination
• Incoming transaction submits a request for service to the switching fabric End-
Point.
• IF free queue element is available THEN it places into the first available input
queue entry THEN transaction will be sent forward through the switching fabric
to the appropriate end-point port.
• IF End-point queue is full, THEN the transaction is retried
• Transaction is removed from the queue, when request/response pair is
completed.

Victor Besyakov, Tundra Semiconductor Corporation 3 of 9

Introduction: Motivation
• Traditional Constrained
Random-based Verification flow
– Step 1: Run random test
– Step 2: Check the
functional/code coverage.
When a desirable coverage
has been achieved,
verification is declared
completed, (otherwise
Step 3)
– Step 3: Adjust the random
stimulus generators by
using constraints and go to
Step1

Victor Besyakov, Tundra Semiconductor Corporation 4 of 9

Introduction: Motivation
DAC 1999 Panel: Functional
Verification - Real Users, Real
Problems, Real Opportunities
“Part of problem with random
testing is that the
distribution of random
events tends to follow a
traditional bell-shaped
curve; adding more and
more random tests does
little to guarantee that
“new” states of the design
are covered. In fact,
random tests typically
exercise the same design
space over and over again,
with only slight variations”.

Victor Besyakov, Tundra Semiconductor Corporation 5 of 9

Introduction: Motivation
• Functional Coverage and Directed
Tests : “What You Thought of is
What You Verify Only”.
• Code coverage : “What Have Been
Missed ?“
• Random is designed to catch
something you did not think of.
But questions related to the test
duration still remain open:
1. How long should a single random
test run?
2. Which type random test is better -
long or short?
3. If a random test is run longer,
what is the probability of explore
the new design states

Victor Besyakov, Tundra Semiconductor Corporation 6 of 9

Introduction: Main Idea

• Collect simulation statistics for multiple random tests

(Sample-path Probabilistic Model )
• Assess similarity between corresponding test runs
(Probabilistic Similarity Metrics )

Victor Besyakov, Tundra Semiconductor Corporation 7 of 9

Agenda
• Introduction
• Random Generator: Sample-path Probabilistic Model
– Sample-path Model
– Markov Chain Model
– Stationary mode, Optimal test length
• Sample-path Probabilistic Metrics
• Sample-path Coverage Framework
• Results and Conclusions

“A random number generator is much like sex:

when it’s good it’s wonderful,
and when it’s bad it’s still pretty good.”
G. Marsaglia

Victor Besyakov, Tundra Semiconductor Corporation 8 of 9

Random Generator: Sample-path
Model
• Random stimulus is created by
a random generator and
constrained according to the
DUT functionality.
• A Probabilistic Model is a set of
mathematical equations which
describe the behavior in terms
of random variables and their
associated probability
distributions.
• Markov Stochastic process :
The future values are only
dependent on present values.
Past values are not considered.

Victor Besyakov, Tundra Semiconductor Corporation 9 of 9

Random Generator: Markov Chain
Model

Random Generator

Victor Besyakov, Tundra Semiconductor Corporation 10 of 9

Random Generator: Markov Chain
Model

• Underlying process model of a discrete-event stochastic system is

generalized as a discrete, irreducible, ergodic Markov chain (MC)
• Stationary distribution (equilibrium distribution, or invariant measure)

Victor Besyakov, Tundra Semiconductor Corporation 11 of 9

Random Generator: Markov Chain
Model Stationary mode, Optimal test
length:

• After equilibrium has been achieved the likelihood to have a non-

repeated, unique sequence of sample-path declines.
• Possibility of discovering new design space points by continuing the
same simulation run declines as well.

Victor Besyakov, Tundra Semiconductor Corporation 12 of 9

Agenda
• Introduction
• Random Generator: Sample-path Probabilistic Model
• Sample-path Probabilistic Metrics
– Introduction
– Kullback-Leibler distance
– Empirical Results MCMC
• Sample-path Coverage Framework
• Results and Conclusions

“Call it entropy… nobody knows what entropy really is, so in a

debate you will always have the advantage.” Von Neumann

Victor Besyakov, Tundra Semiconductor Corporation 13 of 9

Sample-path Probabilistic Metrics.
Introduction

• In MATH: Metric or Distance function is a function which defines a distance

between elements of a set.
Example: Euclidean metric
• Metric or Distance is required to satisfy the following conditions:
1. d(a,b) ≥ 0 (non-negativity)
2. d(a,b) = 0 iff a=b (reflexivity)
3. d(a,b) = d(b,a) ( symmetry)
4. d(a,b) ≤ d(b,c) + d(a,c) (triangle inequality)
Victor Besyakov, Tundra Semiconductor Corporation 14 of 9
Sample-path Probabilistic Metrics:
Kullback-Leibler distance
• In Probability: Metric or Distance
function is a function which defines • The Kullback-Leibler cannot be
a distance between two probability called a real metric:
functions P1(x) and P2(x) – it does not satisfy the triangle
inequality
• Kullback-Leibler distance
– it is not symmetric

• Triangle Inequality (Can be • Symmetrization

ignored) – j-divergence (simple average)
– Decreasing relative entropy KL(P1 ,P2 )+KL(P2 ,P1 )
does not necessarily imply J(P1 ,P2 ) =
2
that two distributions are
getting closer together. – resistor average (harmonic
mean)
– Kullback-Leibler distance will
not be used as a proximity KLR(P1 ,P2 )=(KL(P1 ,P2 )-1 +KL(P2 ,P1 )-1 )-1
measure
Victor Besyakov, Tundra Semiconductor Corporation 15 of 9
Sample-path Probabilistic Metrics:
Empirical Results MCMC
• j-divergence and resistor average
give the best results when applied
to the SPC
• Empirical bound for Kullback–
Leibler distance similarity measures
= 0.005

Victor Besyakov, Tundra Semiconductor Corporation 16 of 9

Agenda
• Introduction
• Random Generator: Sample-path Probabilistic Model
• Sample-path Probabilistic Metrics
• Sample-path Coverage Framework
• Results and Conclusions

“All methodologies are based on fear” Kent Beck

Victor Besyakov, Tundra Semiconductor Corporation 17 of 9

Sample-Path Coverage
Framework
• Step1: Initialization
– Run the first simulation as long as possible.
– Collect statistics from the identified random generators, compute
transition probability matrix, and store it.
• Step 2: Next simulations run
– Collect the statistics on the fly
– Compose partial simulation transition matrices (defined time
interval, certain numbers of transactions)
• Step 3: Make a comparison (All stored matrices vs. current
transition matrix by using Kullback-Leibler measures)
– Case 1: If metrics are rising above 0.005 threshold level, then run
this simulation as long as it possible and at the end of simulation,
save current transition matrix for the next comparison round
– Case 2: If the metrics are falling below the threshold level (due to
the statistical equality of the two simulations), then stop this
simulation and dismiss the current transition matrix.
• Step 4: Repeat steps 2 to 4

Victor Besyakov, Tundra Semiconductor Corporation 18 of 9

Agenda
• Introduction
• Random Generator: Sample-path Probabilistic Model
• Sample-path Probabilistic Metrics
• Sample-path Coverage Framework
• Results and Conclusions
– Applications of SPC method
– Arrival-departure event counting process
– Conclusions

“Reading this paper was similar to performing some type MCMC.

After first iteration I only understood 15% of the paper, and
after a few iterations I got to 40%, but then another careful
reading and I was back down to 30%. Finally, my
understanding of this paper converged to a stable high-80%
(I believe)”. Tomasz Malisiewicz

Victor Besyakov, Tundra Semiconductor Corporation 19 of 9

Results and Conclusions: Applications
of SPC method.
• Applications of SPC method include, but are not limited to,
the assessment of Testbench stimulus random generator
quality.
• All FSM like Testbench or DUT components (FIFO, queues,
etc) can be modeled by the Markov process and
consequently could be treated as Sample-Path Random
Generators.

Victor Besyakov, Tundra Semiconductor Corporation 20 of 9

Results and Conclusions: Arrival-
departure event counting process.

• The incoming transaction submits a request for service to the switching fabric
end-point
• If the input end-point queue is full, the transaction will be asked to retry until the
free queue element is available.
• The transaction is accepted by the DUT, it places the transaction into the first
available input queue entry.
• Then transaction will be sent forward through the switching fabric to the
appropriate end-point port.
• After the request-response pair is completed, the transaction is removed from
the queue, freeing the space for the next pending request.

Victor Besyakov, Tundra Semiconductor Corporation 21 of 9

Results and Conclusions: Arrival-
departure event counting process.

• The arrival-departure process is well approximated by the stationary ergodic

semi-Markov process for queue size equal to m with embedded transition
matrix
• Compare to the TB random generator SPC
– Numbers of states for the MC transition matrices are less compared to TB
random generator - more likely to achieve a stationary mode during one
simulation run.
– Digital design has a tendency to be insensitive to the small sample-path
perturbation and what can look like a different sample-path behavior
(metric > 0.005) at the TB random generator output, can be the same in
terms of buffer release process (metric < 0.005).
Victor Besyakov, Tundra Semiconductor Corporation 22 of 9
Results and Conclusions (Q&A)
1. Which type random test is better - long or short?
An optimal test length is determined by specifying a Sample-path
Probabilistic Model
2. If a random test is run longer, what is the probability of explore the
new design states?
After equilibrium has been achieved, for the non-uniform stationary
distribution, the likelihood to have a non-repeated, unique
sequence of sample-path elements starts to decline.
3. How long should a single random test run?
The metrics < 0.005 (statistical equality) - stop simulation.
The metrics > 0.005 (statistical inequality) - run simulation as long
as it possible.
• By using SPC methods, it is possible to create a verification strategy,
and determine:
– How long we need to run single test.
– How to save computer farm time and leave out the test that exercises
the same design area over and over again. Actual experiment results
show that SPC saves up to 60% of machine cycles compared to the
traditional approach.

Victor Besyakov, Tundra Semiconductor Corporation 23 of 9