Validation Study Protocol and Guidelines IUPAC Protocol for the Design, Conduct and Interpretation of Method- Performance Studies. Pure & Appl. Chem., 67(2), 331-343 (1995) http://www.iupac.org/publications/pac/67/2/0331/ AOAC International AOAC Official Methods of Analysis (2002). Interlaboratory Collaborative Study, Appendix D: Guidelines for Collaborative Study Procedures To Validate Characteristics of a Method of Analysis http://www.aoac.org/vmeth/guidelines.htm ISO 5725-2:1994 Accuracy (trueness and precision) of measurement methods and results -- Part 2: Basic method for the determination of repeatability and reproducibility of a standard measurement method MAFF 28
Comparison of Protocol for the study * test statistics and application procedure are different between the protocols . MAFF 29
Single-laboratory Validated Method General Criteria in Codex Procedure Manual single-laboratory validated methods must fulfill the following criteria: (i) the method is validated according to an internationally recognized protocol (e.g. IUPAC Guidelines) (ii) the use of the method is embedded in a quality system in compliance with the ISO/IEC 17025 MAFF 30
The method should be complemented with information on accuracy demonstrated for instance with: – regular participation in proficiency schemes, where available; – calibration using certified reference materials, where applicable; – recovery studies performed at the expected concentration of the analytes; – verification of result with other validated method where available. (Codex Procedural Manual) MAFF 31
Guidelines on Single-Laboratory Validation Harmonized Guidelines for Single-Laboratory Validation of Methods of Analysis IUPAC/ISO/AOAC, 2002 Performance characteristics • Applicability • Selectivity • Calibration and linearity • Trueness, Precision • Recovery • Limit of Detection, Limit of Quantification • Sensitivity • Ruggedness etc. MAFF 32
Recommended methods in Codex General Methods of Analysis for Contaminants ( CODEX STAN 228-2001) Recommended Methods of Analysis and Sampling (CODEX STAN 234-1999) MAFF 33
Criteria approach Identify method performance characteristics based on existing method validation data. and establish criteria for evaluating acceptable method of analysis ⇒ laboratory can choose a method meeting criteria (flexibility) MAFF 34
Guidelines for Criteria approach <Codex Procedural Manual> Working Instructions for the Implememtation of the Criteria Approach in Codex Guidelines for Establishing Numeric Values for Method Criteria and/or Assessing Methods for Compliance Thereof MAFF 35
Examples for numeric values for the criteria * The sR should be calculated from the Horwitz / Thompson equation MAFF 36
Terms related to variability of analytical results MAFF 37
Terms related to variability of test results Measurement Accuracy Uncertainty Trueness Precision Reproducibility Intermediate Repeatability precision MAFF 38
Distribution of results for repeated analysis C A µ: True value m: mean X-axis: concentration Y-axis: frequency m µ m µ B D m m µ µ MAFF 39
accuracy, trueness, precision µ (true value); m (mean); σ (standard deviation); c ( m + s ) B C A s σ m m m ○ × × accuracy trueness ○ × ○ precision ○ ○ × MAFF 40
Accuracy The closeness of agreement between a test result or measurement result and a reference value. • When applied to a test method, the term accuracy refers to a combination of trueness and precision. Guidelines on Analytical Terminology (CAC/GL 72-2009) A reference value is usually provided with reference to a) a certified reference material; b) a reference measurement procedure; c) a comparison of measurement standards. MAFF 41
Trueness The closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value. • Measurement trueness is inversely related to systematic measurement error, but is not related to random measurement error. • Measurement accuracy should not be used for ‘measurement trueness’ and vice versa . Guidelines on Analytical Terminology (CAC/GL 72-2009) MAFF 42
Precision The closeness of agreement between independent test/measurement results obtained under stipulated conditions. • depends only on the distribution of random errors and does not relate to the true value or to the specified value. • usually expressed in terms of imprecision and computed as a standard deviation of the test results. Less precision is reflected by a larger standard deviation. • Quantitative measures of precision depend critically on the stipulated conditions. Repeatability and reproducibility conditions are particular sets of extreme conditions. Guidelines on Analytical Terminology (CAC/GL 72-2009) MAFF 43
Repeatability Precision under repeatability conditions. Repeatability conditions: Observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in the same test or measuring facility by the same operator using the same equipment within short intervals of time. Guidelines on Analytical Terminology (CAC/GL 72-2009) MAFF 44
Reproducibility Precision under reproducibility conditions. Reproducibility conditions: Observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in different test or measurement facilities with different operators using different equipment. Guidelines on Analytical Terminology (CAC/GL 72-2009) MAFF 45
Relative standard deviation Relative standard deviation (%) = standard deviation (SD) / mean × 100 same as CV (coefficient of variation) 2 2 + + μ μ ( x ) ( x ) 1 n SD n 1 RSD ｒ : Repeatability relative standard deviation RSDR : Reproducibility relative standard deviation MAFF 46
Horwitz equation Horwitz equation (%) 1 0 5 log - . C 2 10 RSD R approximation of Horwitz equation RSDR （ % ） ＝ 2 C -0.1505 RSDR （ % ） : reproducibility relative standard deviation C ： concentration ratio if concentration ratio =100 %, then C = 1 1 %, C = 0.01 A ppm C = A × 10-6 MAFF 47
Graphical expression of Horwitz equation 200 128% 150 45% 100 16% 相対標準偏差 [%] 4% 50 0 -50 -100 RSDR -150 -200 1 ppt 1 ppb 1 ppm 1% MAFF 48
Horwitz/Thompson equation C < 1.2 × 10-7 22 RSDR （％）＝ 1.2 × 10- 7 ≤ C ≤ 0.138 2 C-0.1505 C > 0.138 C-0.5 RSDR （ % ） : Reproducibility relative standard deviation C ： concentration ratios Thompson, M., Analyst, 125, 385-386 (2000) MAFF 49
Graphical expression of Horwitz/Thompson equation 30 16% 20 10 22% 0 -10 -20 Relative standard deviatio -30 1 ppt 1 ppb 1 ppm 1% Concentration 0.1ppm Thompson, M., Analyst, 125, 385-386 (2000) MAFF 50
HorRat Value The ratio of the reproducibility relative standard deviation to that calculated from the Horwitz equation, HorRat(R) = RSDR/PRSDR HorRat(r) =RSDr/PRSDR RSDr, RSDR: observed relative standard deviation (sr/C, sR/C) PRSDR: predicted relative standard deviation calculated from Horwitz/Thompson No rmal range of HorRat Guidelines on Analytical Terminology (CAC/GL 72-2009) MAFF 51
Limit of Detection (LOD) and Limit of Quantification (LOQ) MAFF 52
What is Limit of Detection ? Limit of Detection (LOD) : The true net concentration or amount of the analyte in the material to be analyzed which will lead, with probability (1- β ), to the conclusion that the concentration or amount of the analyte in the analyzed material is larger than that in the blank material. Critical Value (Lc) : The value of the net concentration or amount the exceeding of which leads, for a given error probability α , to the decision that the concentration or amount of the analyte in the analyzed material is larger than that in the blank material . Guidelines on Analytical Terminology (CAC/GL 72-2009) MAFF 53
LOD and Critical Value (Lc) Results of LOD Sample of which concentration concentration sample > LOD will lead to the decision that the concentration of the material is larger than that in β: Type II error the blank material with β probability larger than 1- β (false negative) LOD If results are lower than Lc, they are N.D. If results > Lc, then they will lead to (could not decided to be the decision that it is not blank, detected) detected with probability larger than 1- α. α: Type I error (false positive) Lc Results of blank 0 MAFF 54
The limit of detection LOD is estimated by, LOD ≈ 2 × t1- αν σo [where α = β], — Where t1- α ν : Student's - t , based on ν degrees of freedom for a one- sided confidence interval of 1- α σo : the standard deviation of the true value (expectation). LOD = 3.29 σo, when the uncertainty in the mean (expected) value of the blank is negligible, α = β = 0.05 Usually estimated by using variance of sample (SD of True value is not known.) LOD depends on probability (α and β) Calculate t -value depending on number of data (LOD should not always equals to 3.29 × s (SD. observed)) MAFF 55
t -Distribution and Standard Normal Distribution t -distribution, df= １ t -distribution, df=3 t -distribution, df =10 Standard normal distribution α=0.05: Degree of Freedom = 1 ⇒ t 0.95,1 = 6.31 (2 × t -value=12.62) Degree of Freedom = 10 ⇒ t 0.95,9 = 1.81 (2 × t -value=3.62) Degree of Freedom = ∞ ⇒ t 0.95,∞= 1.64 (2 × t -value=3.29) MAFF 56
Limit of Quantification Limit of Detection (LOD) : A method performance characteristic generally expressed in terms of the signal or measurement (true) value that will produce estimates having a specified relative standard deviation (RSD), commonly 10% (or 6%). LOQ is estimated by: LOQ = kQ × σQ , — Where kQ = 1/RSDQ If σ is known and constant (SD of the estimated quantity is independent of concentration) , and σQ = σo : substitution of kQ = 10% ⇒ LOQ = (10 × σQ) = 10 σo （ 3.04 × LOD, α = β = 0.05 ） Guidelines on Analytical Terminology (CAC/GL 72-2009) MAFF 57
Example of Method Criteria at Different Concentrations (Codex Procedural Manual) MAFF 58
Topic 3 Internal Quality Control and Proficiency testing MAFF 59
3-1 Internal Quality Control (IQC) MAFF 60
Guidelines on internal Quality Control Harmonized Guidelines for Internal Quality Control in Analytical Chemistry Laboratories, Pure & Appl. Chem. 67 (1995) 649-666. Introduction Definition Quality assurance practices and internal quality control Internal quality control procedure IQC and within-run precision Control materials in IQC Recommendations Conclusions MAFF 61
Internal Quality Control Definition Internal Quality Control ： Set of procedures undertaken by laboratory staff for the continuous monitoring of operation and the results of measurements in order to decide whether results are reliable enough to be released. MAFF 62
Control materials in IQC Control materials are characterized substances that are inserted into the run alongside the test materials and subjected to exactly the same treatment. • appropriate concentration of the analyte • same matrix in terms of bulk composition, including minor constituents • similar physical form • stable and possible to divide the material into effectively identical portions for analysis MAFF 63
Use of Certified reference material (1) Certified reference material (CRM): Reference material accompanied by documentation issued by an authoritative body and providing one or more specified property values with associated uncertainties and traceability, using valid procedures Documentation is given in the form of a “certificate” (ISO guide 30:1992). Procedures for the production and certification of certified reference materials are given, e.g. in ISO Guide 34 and ISO Guide 35 . GUIDELINES ON ANALYTICAL TERMINOLOGY (CAC/GL 72-2009) MAFF 64
Metrological Traceability Metrological Traceability: Property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations, each contributing to the stated measurement uncertainty. Metrological traceability of a measurement result does not ensure that the measurement uncertainty is adequate for a given purpose or that there is an absence of mistakes. Guidelines on Analytical Terminology (CAC/GL 72-2009) MAFF 65
Metrological Traceability (2) Comparability of measurement result, for quantities of given kind. (metrologically traceable to the same reference) BIPM Definition of unit, SI base National Metrology Primary national Primary national Institute, Designated standard standard (other countries) Institute Calibration Reference standards organization Competent authority, Working standards Industry, Academia Measurement result End user Larger Measurement Uncertainty in lower stage MAFF 66
Use of Certified reference material (2) COMAR (International Database) http://www.comar.bam.de/en/ Purchase from reagents manufacturer or reagents sales company Comparison of the measurement results with the certified value MAFF 67
Comparison of the measurement results with the certified value Δm ： absolute difference between mean C Δ C measured valued and certified value m m CRM Cm: mean measured value CCRM : Certified Value UΔ ： expanded measurement uncertainty + 2 2 u u u um: measurement uncertainty Δ m CRM uCRM : uncertainty of the certified value U 2 u Δ Δ If Δm ≤ U Δ then there is no significant difference between the measurement result and certified value. http://www.erm-crm.org/ERM_products/application_notes/Pages/index.aspx MAFF 68
Constraint on the use of CRM For the majority of analysis there is no closely matching CRM available. Cost to stock every relevant CRMs Not applicable to unstable materials Availability of sufficient amounts for IQC to use over extended periods ⇒ Preparation of House reference material e.g. materials in proficiency testing, spiked control materials, etc. MAFF 69
Recovery checks If the use of reference material is not practical, limited check on bias is possible by a test of recovery. applicable unstable analyte/matrices, ad hoc analysis A test portion of the test sample spiked with a known amount of the analyte is analyzed alongside the original test material. MAFF 70
Blank determination an analysis of a sample without the analyte, or an analysis without a sample, i.e. going through all steps of the procedure with the reagents only. 1. In many analyses sample results are calculated by subtracting blank readings from sample readings. 2. Blank readings can be excellent monitors in quality control of reagents, analytical processes, and proficiency. 3. They can be used to estimate several types of method detection limits. MAFF 71
Insertion of control materials for IQC (1) Recommendation 1. Short (e.g. n<20) frequent runs of similar materials • at least once per run • analyze in duplicate at least half of the test materials selected at random • insert at least one blank determination 2. Longer (e.g. n>20) frequent runs of similar materials • at an approximate frequency of one per ten test materials at least once per run • analyze in duplicate at minimum of five test materials selected at random • insert one blank determination per ten test materials MAFF 72
Insertion of control materials for IQC (2) 3. Frequent runs containing similar materials but with a wide range of analyte concentrations • at an approximate frequency of one per ten test materials at least once per run. • at least two concentration levels, one close to the median level of typical test materials, and the other is approximately at the upper or lower of decile as appropriate. • duplicate a minimum of five test materials. • insert one procedural blank per ten test materials. 4. ad hoc analysis (statistical control is not applicable) • duplicate analysis on all of the test materials • recovery tests or use of formulated control material • blank determination MAFF 73
Use of Control Charts Control Chart (Shewhart Chart): A statistical tool used to monitor process stability and control. One of the Seven Basic Tools of Quality Control . 1. Understanding current and past process performance and its degree of consistency 2. Establishing a " state of statistical control " by identifying and removing causes of unnatural (or "special cause") variation so as to achieve a consistent and predictable level of process quality over time; MAFF 74
Common Control Charts X - R chart chart X - s chart X - R MAFF 75
How to calculate/plot X-Bar and R chart (1) MAFF 76
How to calculate/plot X-Bar and R chart (2) : Calculate the following statistics X : The mean for each subgroup + + + X X X 1 2 n X n n : The number of measurements within a subgroup (here, n=2) : The grand mean of all subgroup averages X + + + X X X 1 2 k X k k : The number of subgroups (here, k=9) MAFF 77
How to calculate/plot X-Bar and R chart (3) R : The average of the ranges for all subgroups + + + R R R 1 2 k R k R i : The individual range for each subgroup k : The number of subgroups (here, k=9) MAFF 78
How to calculate/plot X-Bar and R chart (4) : Calculation of Upper and Lower control limit Upper control limit (UCL): + For X chart: UCL X A R 2 For R chart: UCL D R 4 Lower control limit (LCL): For X chart: LCL X A R 2 For R chart: LCL D R 3 MAFF 79
How to calculate/plot X-Bar and R chart (5) Constants for Calculating Limits for X-Bar and R charts http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc321.htm MAFF 80
How to calculate/plot X-Bar and R chart (6) Time line (Day/Run) MAFF 81
Example of X-Bar and s chart Time line (Day/Run) MAFF 82
3-2 Proficiency testing MAFF 83
External Program for Quality Control (Proficiency testing) Proficiency testing (PT) : evaluation of participant performance against pre-established criteria by means of interlaboratory comparisons (ISO/IEC 17043:2010) • Participating laboratories analyze same sample distributed from proficiency testing provider. • Reported results were compared to the assigned value and statistically evaluated. • Participants can confirm their results. MAFF 84
How to participate in PT program? Proficiency testing provider Internationally renowned: Fera(FAPAS, FEPAS), AOACI, AOCS, etc. Domestic (In case of Japan): JSAC, FDSC, JAB, etc. Participation Through internet, agency, etc. Check schedule of individual proficiency testing round. MAFF 85
Example of Proficiency Testing Scheme example ： FAPAS • Round: Nutritional components, Food ingredients • contaminants, Pesticides, Veterinary drug residues, Food additives, etc. • Participating laboratories analyze distributed sample using their method. Report the results and the method information through website . • Results were Statistically evaluated and z-score were returned to participating laboratories. | z | ≦ 2, 2 ＜ | z | ＜ 3, | z | ≧ 3 MAFF 86
Result of PT Usually shown in z-score defined as follows: ｚ = (x - X) / σ p x: the participant’s reported result X ： the assigned value e.g., robust mean after excluding results that are clearly spurious and outliers by statistical evaluation. σ p ： the target value for standard deviation e.g., using RSDR from collaborative study, calculation from Horwitz equotion etc. MAFF 87
Graphical expression of z-Scores (example) z-Scores for Patulin (80.7 g/L) in Apple Purée Test Material From: FAPAS PROTOCOL FOR THE ORGANISATION AND ANALYSIS OF DATA SIXTH EDITION, 2002 MAFF 88
How to interpret a result of PT? z- Scores “| z | ≤ 2 ” is “satisfactory” ? • A result of |z| >2 is not specially rare. • Is that a usual operation for the laboratory? Importance of Participating PT Checking bias of the results Review of method procedure and management system of laboratory by periodical participation Information on methods used by other laboratories MAFF 89
Topic 4 Measurement Uncertainty MAFF 90
Measurement Uncertainty in Codex (1) Definition Measurement uncertainty : Non-negative parameter characterizing the dispersion of the values being attributed to a measurand, based on the information used. Expanded measurement uncertainty : product of a combined standard measurement uncertainty and a factor larger than the number one Guidelines on Analytical Terminology (CAC/GL 72-2009) MAFF 91
What is Measurement Uncertainty? Estimated range in which the true value would be found Showing reliability of the results Estimated by experimental data and statistical evaluation MAFF 92
Analytical results and its variability • Analytical results of repeated analyses of the same sample can be found in a normal distribution. • In case of chemical quantitative analysis, reproducibility, between laboratory variation, depends on concentration of analyte, independent of variety of food or analyte. MAFF 93
Normal Distribution Mean : µ SD : σ 68% 95% 99% − 4σ − 3σ − 2σ − σ +σ +2σ +3σ +4σ µ 68% of the observations have values within the range of µ ± 1σ 95% of the observations have values within the range of µ ± 2σ MAFF 94
Measurement Uncertainty and Bias ( Value we want to know ) True Value Y-axis: Density (unknown) Population distribution (unknown) X-axis: Concentration Mean Measurement Uncertainty : Estimated range in which the true value would be found . Sample dist stribution on Bias: difference between the result and the true value MAFF 95
Measurement Uncertainty in Codex (2) Measurement Uncertainty have to be estimated in Codex. One of the requirements of the ISO/IEC 17025:2005 The measurement uncertainty of a result shall be estimated and then made available if requested. Guidelines CAC/GL 27-1997 require laboratories involved in the import/export of foods to comply with general criteria in ISO/IEC 17025. Guidelines on Measurement Uncertainty (CAC/GL 54-2004) MAFF 96
Measurement Uncertainty in Codex (3) Reporting the results in a form of “ a ± U ” — where a: The best estimate of the true value of the concentration of the measurand (the analytical result) u: The Standard uncertainty U: The expanded uncertainty (usually equal to 2u). The range “a ± 2u” represents a 95% level of confidence in which the true value would be found. Guidelines on Measurement Uncertainty (CAC/GL 54-2004) MAFF 97
Estimating Measurement Uncertainty 1. Bottom-up approach Identification of uncertainty sources Estimation of the size of the uncertainty component associated with each potential source. Calculation of combined uncertainty 2. Top-down approach Use data from collaborative trials, proficiency studies, validation studies or intra-laboratory quality control samples, Guidelines on Measurement Uncertainty (CAC/GL 54-2004) MAFF 98
Typical Value of Expanded Uncertainty CAC/GL 54-2004 GUIDELINES ON MEASUREMENT UNCERTAINTY MAFF 99
MU in Compliance Decisions ＋ U Result Maximum － U Level ii ） iii ） iv ） Situation i ） Result ± U Result ± U Result > ML Result < ML but but above ML < ML ML within U ML within U Guidelines on Measurement Uncertainty ( CAC/GL 54-2004 ) Guidelines on Estimation of Uncertainty of Results ( CAC/GL 59-2006 ) MAFF 10 0
Recommend
More recommend
