DRAFT PREDECISIONAL

Version 9 March 2012

Guidelines for Validation of Qualitative Chemistry Methods

DRAFT PREDECISIONAL

Version 9 March 2012

Table of Contents

1.0 2.0 3.0 4.0. 5.0 6.0

Scope Terms and Definitions Selectivity Study Cross-Reactivity Study Matrix Study Collaborative Study

DRAFT PREDECISIONAL

Version 9 March 2012

Scope The purpose of this document is to provide a guideline for the validation of binary qualitative methods intended to detect biological and chemical analytes.

Terms and Definitions Where appropriate, definitions have been taken from international standards and the source is noted. Sources of definitions include the following: ISO/IEC Guide 99:2007, International vocabulary of metrology Basic and general concepts and associated terms (VIM) ISO 3534-2:2006, Statistics Vocabulary and symbols Part 2: Applied statistics ISO 14971:2007, Medical devices Application of risk management to medical devices ISO 17511:2003, In vitro diagnostic medical devices Measurement of quantities in biological samples Metrological traceability of values assigned to calibrators and control materials ISO 5725-1: 1994, Accuracy (trueness and precision) of measurement methods and results Part 1: General principles and definitions USP 31:2008, US Pharmacopeia General Information/<1223> Validation of Alternative Microbiological Methods Candidate Method The method submitted for validation. Lower Limit of Applicability (LLA) Intended concentration that subject matter experts determine that a method must detect the analyte(s) with a specified POD. Both the LLA and POD may be specified by subject matter experts. Laboratory Probability of Detection (LPOD) The average probability of detection across all laboratories. Matrix Totality of components of a material system except the analyte (ISO 17511). Method A procedure that includes sample processing, assay, and data interpretation. Probability of Detection (POD) The proportion of positive analytical outcomes for a qualitative method for a given matrix at a given analyte level or concentration. POD is concentration dependent. Qualitative Binary Method A method of analysis with two possible outcomes. Reproducibility

DRAFT PREDECISIONAL

Version 9 March 2012

Precision under reproducibility conditions (ISO 5725-1). Reproducibility Conditions Conditions where test results are obtained with the same method on identical test material in different laboratories with different operators using different equipment. Sample A small portion or quantity, taken from a population or lot that is ideally a representative selection of the whole. Samples are taken from lots for purposes of scientific examination and analysis and are intended to provide characteristic information about the population, generally by applying statistical calculations. Source: ISO 3534-1:1993 Laboratory sample Sample as prepared for sending to the laboratory and intended for inspection or testing. Source: [ISO 7002:1986] Test portion A fraction of a sample intended for analysis. There are cases (liquid products, analysis of symptoms, etc.) where the laboratory sample is also the test sample. Source: ISO 21572. 3.0 Selectivity Study The Selectivity study is designed to demonstrate that a candidate method can detect the different varieties of the claimed analyte, especially when the claim is for a class of compounds such as beta-lactam antibiotics. 3.1 Collect samples representing the variation and forms present in the analyte population and organize into a selectivity test panel. For example, if the analyte is beta-lactam antibiotics, then a primary test panel of a variety of beta-lactam antibiotics samples must be collected. Document the source and origin of each primary test panel sample. All documentation of analyte identity must be on file and available for review. Prepare at least one replicate of each selectivity test panel sample at the Lower Level of Applicability (LLA) concentration. Test each selectivity test panel sample using the candidate method. 3.4 If an individual selectivity sample tests negative, it may be retested with a number of replicates to be determined by subject matter experts. The level of replication will determine the lower confidence interval for the POD estimate. For example, using 96 replicates with no failures allowed will demonstrate a 95% lower confidence limit on the probability of detection (POD) of 0.95 or higher for that panel member.

3.2

3.3

DRAFT PREDECISIONAL

Version 9 March 2012

4.0

Cross-Reactivity Study A cross-reactivity study demonstrates that a candidate method does not cross-react with similar compounds, and therefore the candidate method is sufficiently specific. 4.1 Collect samples chosen to adequately cover potentially cross reactive substances into a cross-reactivity test panel. Prepare at least one replicate of each cross-reactivity test sample at least 10 times (10X) the LLA of the analyte of interest. Test each cross-reactivity sample using the candidate method. If an individual cross-reactivity sample tests positive, it may be retested with a number of replicates to be determined by subject matter experts. The level of replication will determine the lower confidence interval for the POD estimate. For example, using 96 replicates with no failures allowed will demonstrate a 95% lower confidence limit on the probability of detection (POD) of 0.05 or higher for that panel member.

4.2

4.3 4.4

5.0

Matrix Study The matrix study is a single-laboratory study designed to demonstrate that a candidate method can detect the analyte in the claimed matrixes. Test samples of the claimed matrix with the analyte at various levels are tested. Generally, the LLA is included as one of the test levels. Analysis of samples is usually performed by the method developers laboratory, or by an independent laboratory under the direction of the method developer. 5.1 Matrix Categories An organization may recognize claims for only the range of matrix categories or specific matrix types included in a method developer study and/or collaborative study. The number of different matrices to be tested depends on the claims and intended use of the method. 5.2 Study Design The method should be tested at various levels to develop a rudimentary POD response curve. A matrix blank should be studied, as well as a sample with a high enough concentration to assure very high POD. In addition, a sample should be prepared to validate the response at or near the LLA of the analyte of interest (note: all further references to LLA are for the analyte of interest). In general, a minimum of 5 levels should be run at this stage, but more could be included at the discretion of subject matter experts. The number of replicates per level should be determined by the ERP or Stakeholder Panel. Replicates at the LLA may be higher than at other levels. For example, the LLA may have 96 replicates while other levels only 5. A more balanced approach would spread replicates across all levels, at 20 replicates at each of 5 levels. Some discretion is allowed with consultation by the Statistical Advisor. For example, if many levels are desired, the number of replicates per level could be reduced to

DRAFT PREDECISIONAL

Version 9 March 2012

maintain the overall cost of the experiment. The decisions on number of replicates should be made with an understanding of the desired level of confidence in the final results. Obtain enough master sample of each matrix to prepare the required number of test portions for each level. (It is recommended to allot an additional 10% of material to compensate for loss). Test portions are to be taken from each sub-sample. If the method is intended to detect more than one analyte simultaneously from the same test portion, the validation study should be designed so that analyses are spiked into a common sample and the validation tests are performed in a simultaneous manner. Incurred or Spiking: A master sample may be incurred (the analyte is present in the sample) or spiked. If a master sample with naturally occurring analyte is used, then additional material that is known to be free of the analyte can be used to dilute the incurred sample to the desired concentration. If spiked material is used, the base material should be sub-divided into test portions and then spiked. Raw and/or processed materials: Both processed (cooked) and raw samples should be represented if the assay claims to detect the analyte in cooked foods. If spiked samples are used, the analyte must be spiked into the material previous to processing, and processing must be equivalent to standard processing. In either case, prepare the required number of test portions of the matrix with the analyte at the specified concentration. Blind-code the prepared samples with a random number. Similarly, prepare 96 test portions of each matrix with one of the specificity test panel compound/substances at 10 times (10X) the LLA. Code the prepared samples. Randomly mix the blinded coded samples. Use the candidate method to evaluate the blinded samples. The analyst performing the analyses should not have knowledge of the study design or the blind-codes of the test portions. The analyst should be informed that the design of the study does include a certain number of blank samples and that both positive and negative outcomes should be expected. Plot the response of the method as POD response vs. concentration of analyte. 5.3 Statistical Analysis Refer to Annex 1 for guidance on statistical analysis of data.

DRAFT PREDECISIONAL

Version 9 March 2012

6.0

Collaborative Study

A Collaborative Study characterizes the performance parameters (e.g., POD, repeatability, reproducibility) of the candidate method across laboratories. Methods shall be validated under simulated conditions of intended use. For example, a method intended for use by trained factory operators at a grain inspection site must be validated under conditions that simulate the grain inspection site and should include representative end users as collaborators.

6.1

Study Design A collaborative study with a minimum of at least 8 valid data sets is recommended. Deviations from this recommendation should be documented and justified. It is recommended to collaboratively study at least 12 replicates at each of five concentrations of the analyte(s) of interest representing: zero concentration, low concentrations, mid-range concentrations, the LLA concentration, and at least one concentration above the LLA. The 12 (or more) replicates may be spread over the collaborators, i.e., 4 collaborators might analyze 3 replicates each at one concentration, while another 4 collaborators might analyze 3 replicates each at a second level. Other analyte concentrations, determined by the subject matter experts, may be studied. All laboratory samples must be blind-coded and shipped to each collaborator. Replicated test portions must be performed blindly at the laboratory sample stage and not by collaborators as replicate test portions from the same laboratory sample. Collaborators shall perform all replicate analyses independently. If the method is intended to detect more than one analyte simultaneously from the same test portion, the validation study should be designed so that analytes are spiked into a common sample and the validation tests are performed in a simultaneous manner. The number of test portions per sample per collaborator is 10

6.2

Data Analysis 6.2.1 Raw Data Tables For each matrix and concentration level, report each result from each test portion separately. 6.2.2 Statistical Analysis Refer to Annex 1 for guidance on statistical analysis of data.

DRAFT PREDECISIONAL

Version 9 March 2012

6.2.5

Collaborator Comments Comments on the candidate method should be collected from all collaborators and reported in the Collaborative Study report.

References

1. Wehling, LaBudde, Brunell, & Nelson, PROBABILITY OF DETECTION (POD) AS A STATISTICAL MODEL FOR THE VALIDATION OF QUALITATIVE METHODS, Journal of AOAC INTERNATIONAL VOL. 94, NO. 1, 2011. 2. Macarthur & von Holst, A PROTOCOL FOR THE VALIDATION OF QUALITIATIVE METHODS OF DETECTION, Monitoring and Quality Assurance (MoniQA), pending.

DRAFT PREDECISIONAL DO NOT DISTRIBUTE

Annex 1

Validation for qualitative methods of detection Qualitative methods are those that give two responses which can usually be interpreted as analyte detected or analyte not detected. Their performance can be validated by collaborative trials, single laboratory validation studies, or by using observed long-term performance in much the same way, practically, as analytical methods that give quantitative responses, with similar consideration given to ensuring that the data used to validate methods covers a representative scope and range with adequate replication between laboratories, analysts, or days. The chief practical difference is just that more analytical replicates are needed within each analytical condition (lab, day, concentration etc). However, the statistical treatment of results needed to get estimates of method performance, principally the probability of detection for a given concentration of analyte and how much this probability might vary, is a little different to that used for quantitative methods. For example, the relations between the average probability of detection across laboratories; the reproducibility standard deviation of the probability of detection and the interval within which we can expect laboratories probabilities of detection to lie are not the same as the analogous relations for measurement results produced by quantitative methods in collaborative trials. Three recent publications give guidance on the statistical treatment of results for the validation of qualitative methods of analysis which are based on examining the probability of detection: Probability of Detection (POD) as a statistical model for the validation of qualitative methods [1], How to Validate Qualitative methods of Detection [2], and Probability of Identification (POI): a Statistical Model for the Validation of Qualitative Botanical Identification Methods [3]. These publications give methods for analyzing the results of method validation using two important components of method performance: the average probability of detection [1, 3] and the interval within which probabilities of detection on each instance of use (e.g. across different laboratories or, different days) may lie [2]. A method can be validated using one or both of these approaches depending on the extent to which information about the average probability of detection or the interval within which probabilities of detection may lie is needed to make a decision about method performance. For example, where a user needs a method to test their own product and they undertake an inhouse validation, then the average probability of detection [1, 3] observed during the validation study may be the most important measure of analytical performance for them because the costs and benefits of higher and lower probabilities of detection on each instance of use belongs to them. If they decide to offer the analysis as a commercial service then their customers may need assurance that the analytical method meets a target for a probability of detection on each instance of use. This can be achieved by an analysis of the validation study to estimate the interval within which the probability of detection may lie on different days, using the statistical techniques described in [2]. In this example both approaches are needed to satisfy different stakeholders. Similar considerations apply to validation by collaborative trial or by using observed long-term performance. The methods described in [1], [2] and [3] are designed to be accessible to all users. Where sufficient statistical expertise is available, a more accurate assessment of method performance may sometimes be achieved by fitting a model for the probability of detection across analyte concentrations. 1 2 Wehling, P. LaBudde, R.A. Brunelle, S.L. Nelson, M.T. 2011, Probability of Detection (POD) as a statistical model for the validation of qualitative methods, J AOAC Int 94(1):335-47 Macarthur, R. von Holst, C. How to validate qualitative methods of detection, Analytical Methods, accepted for publication.

DRAFT PREDECISIONAL DO NOT DISTRIBUTE

Annex 1

LaBudde, R.A. and Harnly, J. M. 2012. Probability of Identification (POI): a Statistical Model for the Validation of Qualitative Botanical Identification Methods. J AOAC 95(1): 1-1-13.