Uncertainty From Validation Studies

Uncertainty from validation studies
S Ellison
The preparation of this paper was supported under contract with the Department of Trade and Industry as part of the National
Measurement System Valid Analytical Measurement (VAM) Programme
/EURACHEM 2002/mu_val/ 1
What is Measurement Uncertainty?

A parameter, associated with the result of a
measurement, that characterises the dispersion of
the values that could reasonably be attributed to
the measurand
(ISO Guide)
The number after the

ISO Guide approach

z
Specify the measurand
including complete equation
Quantify significant uncertainties in all parameters

A: from statistics of repeated experiment
B: by any other means (theory, certificates,
judgement...)
Express as standard deviation
Combine according to stated principles
Implementing ISO in Chemistry

Building models
z
Every determinand is unique
Every element, every molecule, every formulation
Every matrix is unique
Different interactions with substrate
Interactions with environment and substrate rarely

understood
Models are difficult to build!
Extraction processes
Solvent
Incurred
Added reference
A simpler model
z
The best available estimate of precision
An effect varied representatively during a precision

experiment requires no further study
The best available estimate of bias and its

uncertainty
Other significant effects evaluated
By experiment, or from standing data
In-house validation
Bias
uncertainty
Precision
(long term)
MU
Good reference
needed
Analytical recovery a
problem
Physical uncertainties
usually negligible
Chemical effects need
study
Other
effects
Collaborative Study
Sampling
Effects
Reproducibility
sd (sR)
MU
Requires correct
study and sound
QC/QA
Matrix
Effects
Experiments for measurement

uncertainty estimation
z
Random variaton
Recovery studies
Other effects
Simple sensitivity analysis
Numerical calculations
Response surfaces
Random variation
z
Standard deviation of replicate measurements;
Results of ANOVA
within- or between-group standard deviation;
Significance tests;
in a t-test the denominator can be interpreted as a

standard uncertainty.
Standard deviations
s=
xi
n 1
ANOVA variance components

Sample
Effect
Replicates
SS
Source of variation
Between groups
Within groups
Total
within
= MS
between
df
MS
MSB
MSW
MS MS
=
n
B
An approach to Recovery uncertainties

ycorr = yobs Rm Rs Rrep
Rm corrects for mean reference recovery
Rs is a nominal correction for sample effects
Rrep corrects for spike recovery differences
Uncertainties in Recovery estimates

z
Reference value uncertainty
Measurement on reference value
LARGE unless matrix-specific RM
Spike behaviour
LARGE in most heterogeneous systems.
see Analyst 124 981-990 (1999)

Rm
Repeatability dominated: Small
Relevance of reference to sample
Traceability Uncertainties - Small
Rs
Rrep
Evaluating u(Rm)
z
With representative CRM or reference method:

Analysis of CRM or with reference procedures:
u(Rm) from stats and certified uncertainty
Without representative CRM

Spiking studies: u(Rm) as above, but needs u(Rrep)
Changes in extraction system, worst case CRM:
u(Rm) from extreme values
Recovery behaviour monitoring/modelling:
u(Rm) from model
Characterising Extraction Behaviour

800
GC signal (arbitrary)
700
600
500
400
300
Dibenz(a,h)anthracene
200
"Hot Ball" model (approx)

1st order fit
100
Dibenz(a,h)anthracene extraction
0
0
10
20
30
40
50
Extraction cycles
60
70
80
90
Improving technology
35.00
mext
mext (pred, 5 terms)
30.00
mext (pred, to 1e-6)
mass%DEHP
25.00
20.00
.
15.00
10.00
ASE of DEHP from PVC

Pellets (Sum of % DEHP)
5.00
0.00
0
20
40
60
80
Time/min
100
120
140
Evaluating u(Rs)
(sample effect)
z
Replicate analysis of range of CRMs

or Comparison of results with standard procedures
Spiking studies on range of materials
u(Rm) from statistical observation and certified

uncertainty
u(Rm) as above, but needs u(Rrep)
NB: Some intricate equations*
*V. J. Barwick, S. L. R. Ellison

Analyst, 1999, 124, 981-990.
Evaluating u(Rrep)
(representativeness of spiking)
z
Analysis of spiked CRM(s)
Behaviour of similar materials
Monitoring & modelling extraction rates (native vs

spiked)
Other effects: A simple uncertainty

experiment
( y2 y1 )
Measurement
Gradient
=
result y
( x2 x1 )
y2
u(y)
y1
x1
x2
u(x)
Input parameter xi
Sensitivity analysis: Crude Fibre

analysis
Dietary Fibre: Summary of contributions to the
measurement uncertainty
Parameter
Uncertainty*
Precision
u(P)
0 0265
Acid concentration u(concacid)
0 0003
Alkali concentration u(concalkali)
0 00048
Acid digestion time u(timeacid)
0 01
Alkali digestion time u(timealkali)
0 0072
Drying temperature
u(dry)
0 115 %(w/w)
and time
*all the uncertainties are quoted as relative
standard deviations except for u(dry)
Response surface modelling

Triglycerdide saponification
%YIELD
20
1
% Yield
Time (min)
15
115
105
10
75
82
87
91
97
103
106
above
95
85
75
15
60
10
Time (min)
5
TIME
30
35
65
55
50
45
Temp (C)
40
TEMP
1
1
0
20
30
40
50
60
Temperature (C)
70
Improved model
y = y0+ aTexp(-kT/T)
+aTt t.exp(-kT/T)
(Empirical kinetic form)
70
80
90
95
100
103
above
Full kinetic study

C18:2 - acid-catalysed methylation
Nominal
time/temp
15
30
45
60
75
90
97.5
above
Modelling and experimental studies

z
Sparse (eg verification) data may give

inappropriate fitting
Chemical knowledge helps - but models may be

more complex
Uncertainty estimates broadly insensitive to

models if they fit the data well.
The most useful information may be evidence of

negligible contributions!
Numerical calculation
Calculated
result y
y(x+u(x))
y(x)
u(y)
u(y)
y(x+u(x)) - y(x)
x x+u(x)
Uncertainty Components
Determination of Cholesterol
S a m p le r e c o v e r y
M e th o d re c o v e r y
P r e c is io n
S a m p le w e ig h t
W e ig h t o f b e t u lin a d d e d
TO TAL
0
0 .0 0 5
0 .0 1
0 .0 1 5
0 .0 2
0 .0 2 5
0 .0 3
0 .0 3 5
C o n t r ib u t io n t o U n c e r t a in t y ( R S D )
0 .0 4
0 .0 4 5
Uncertainty Components
Determination of Acesulfam-K
Method recovery
Sample recovery
Precision
Calibration
Stock concentration
Sample volume
Final volume
TOTAL
0
0.01
0.02
0.03
0.04
0.05
0.06
Contribution to Uncertainty (RSD)
0.07
0.08
Precision and Bias

contributions to uncertainty
Crude Fibre (2.5%)
Fe in feeds
As in PVC
Other
Other
Other
Recovery
Precision
Precision
Precision
Recovery
Recovery
Ammonium
in water
Hydroquinone in cosmetics
Other
Other
Cholesterol in
fats
Precision
Recovery
Precision
Recovery
Precision
Other
Precision
Recovery
Bias
uncertainty
Other
Conclusions
z
Validation studies and measurement uncertainty

studies are views of the same problem
prediction and observation = Science
Where MU is required, analysts need methods of

using validation data
EURACHEM/CITAC guide incorporates general
guidance
Still some specific issues with detailed experimental
studies

Uncertainty From Validation Studies

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Uncertainty From Validation Studies

Uploaded by

Copyright:

Available Formats

Uncertainty from validation studies

What is Measurement Uncertainty?

The number after the

ISO Guide approach

Specify the measurand

including complete equation

Quantify significant uncertainties in all parameters

Express as standard deviation

Combine according to stated principles

Implementing ISO in Chemistry

Every determinand is unique

Every element, every molecule, every formulation

Every matrix is unique

Different interactions with substrate

Interactions with environment and substrate rarely

Models are difficult to build!

The best available estimate of precision

An effect varied representatively during a precision

The best available estimate of bias and its

Other significant effects evaluated

By experiment, or from standing data

Experiments for measurement

Standard deviation of replicate measurements;

within- or between-group standard deviation;

in a t-test the denominator can be interpreted as a

ANOVA variance components

An approach to Recovery uncertainties

Uncertainties in Recovery estimates

Reference value uncertainty

Measurement on reference value

LARGE unless matrix-specific RM

LARGE in most heterogeneous systems.

see Analyst 124 981-990 (1999)

Repeatability dominated: Small

Relevance of reference to sample

Traceability Uncertainties - Small

With representative CRM or reference method:

Without representative CRM

Changes in extraction system, worst case CRM:

u(Rm) from extreme values

Recovery behaviour monitoring/modelling:

u(Rm) from model

Characterising Extraction Behaviour

"Hot Ball" model (approx)

mext (pred, to 1e-6)

ASE of DEHP from PVC

Replicate analysis of range of CRMs

Spiking studies on range of materials

u(Rm) from statistical observation and certified

NB: Some intricate equations*

*V. J. Barwick, S. L. R. Ellison

Analysis of spiked CRM(s)

Behaviour of similar materials

Monitoring & modelling extraction rates (native vs

Other effects: A simple uncertainty

Sensitivity analysis: Crude Fibre

Response surface modelling

Full kinetic study

Modelling and experimental studies

Sparse (eg verification) data may give

Chemical knowledge helps - but models may be

Uncertainty estimates broadly insensitive to