Assessing repeatability and reproducibility of dose-response experiments Marc Weimer and Annette Kopp-Schneider German Cancer Research Center, Heidelberg
Dose response data Laboratory Lab.A, Compound Cmp.1 200 150 response 100 Lab.A:Cmp.1:Exp.1 Lab.A:Cmp.1:Exp.2 Lab.A:Cmp.1:Exp.3 50 0 1 100 dose ◮ Relationship between dose of a substance and biological response ◮ Quantitative toxicology: ED 50 values as predictors for toxicity
ED 50 estimation Laboratory Lab.A, Compound Cmp.1 200 150 response 100 Lab.A:Cmp.1:Exp.1 Lab.A:Cmp.1:Exp.2 Lab.A:Cmp.1:Exp.3 50 0 1 100 dose d − c ◮ Fit model f ( x ) = c + 1 + exp( b (log( x ) − e )) ◮ Parameter e ≡ log(ED 50 )
ED 50 estimates Laboratory Lab.A, Compound Cmp.1 200 150 response 100 Lab.A:Cmp.1:Exp.1 Lab.A:Cmp.1:Exp.2 Lab.A:Cmp.1:Exp.3 50 0 1 100 dose Estimate Lower Upper Lab.A:Cmp.1:Exp.1 37.97 34.29 42.05 Lab.A:Cmp.1:Exp.2 32.68 29.01 36.81 Lab.A:Cmp.1:Exp.3 40.14 36.93 43.62
Different laboratories Laboratory Lab.A, Compound Cmp.1 Laboratory Lab.B, Compound Cmp.1 200 200 150 150 response response 100 100 Lab.A:Cmp.1:Exp.1 Lab.B:Cmp.1:Exp.1 Lab.A:Cmp.1:Exp.2 Lab.B:Cmp.1:Exp.2 Lab.A:Cmp.1:Exp.3 50 Lab.B:Cmp.1:Exp.3 50 0 1 100 0 1 100 dose dose Estimate Lower Upper Estimate Lower Upper Lab.A:Cmp.1:Exp.1 37.97 34.29 42.05 Lab.B:Cmp.1:Exp.1 39.77 36.68 43.12 Lab.A:Cmp.1:Exp.2 32.68 29.01 36.81 Lab.B:Cmp.1:Exp.2 39.23 35.74 43.05 Lab.A:Cmp.1:Exp.3 40.14 36.93 43.62 Lab.B:Cmp.1:Exp.3 44.43 41.44 47.63
Different labs and different compounds Compound Cmp.1 Compound Cmp.2 200 Lab.A:Cmp.2:Exp.1 200 Lab.A:Cmp.2:Exp.2 Lab.A:Cmp.2:Exp.3 Lab.B:Cmp.2:Exp.1 150 Lab.B:Cmp.2:Exp.2 150 response response Lab.B:Cmp.2:Exp.3 Lab.A:Cmp.1:Exp.1 100 100 Lab.A:Cmp.1:Exp.2 Lab.A:Cmp.1:Exp.3 Lab.B:Cmp.1:Exp.1 Lab.B:Cmp.1:Exp.2 50 Lab.B:Cmp.1:Exp.3 50 0 1 100 0 1 100 dose dose Compound Cmp.3 Compound Cmp.4 200 200 150 150 response response Lab.A:Cmp.3:Exp.1 Lab.A:Cmp.4:Exp.1 100 Lab.A:Cmp.3:Exp.2 100 Lab.A:Cmp.4:Exp.2 Lab.A:Cmp.3:Exp.3 Lab.A:Cmp.4:Exp.3 Lab.B:Cmp.3:Exp.1 Lab.B:Cmp.4:Exp.1 Lab.B:Cmp.3:Exp.2 Lab.B:Cmp.4:Exp.2 Lab.B:Cmp.3:Exp.3 Lab.B:Cmp.4:Exp.3 50 50 0 1 100 0 1 100 dose dose
Different labs and different compounds Compound Cmp.1 Compound Cmp.2 200 Lab.A:Cmp.2:Exp.1 200 Lab.A:Cmp.2:Exp.2 Lab.A:Cmp.2:Exp.3 Lab.B:Cmp.2:Exp.1 150 Lab.B:Cmp.2:Exp.2 150 response response Lab.B:Cmp.2:Exp.3 Lab.A:Cmp.1:Exp.1 100 100 Lab.A:Cmp.1:Exp.2 Lab.A:Cmp.1:Exp.3 Lab.B:Cmp.1:Exp.1 Lab.B:Cmp.1:Exp.2 50 Lab.B:Cmp.1:Exp.3 50 0 1 100 0 1 100 dose dose “What is the intra- and interlaboratory variability of the assay?” Compound Cmp.3 Compound Cmp.4 200 200 150 150 response response Lab.A:Cmp.3:Exp.1 Lab.A:Cmp.4:Exp.1 100 Lab.A:Cmp.3:Exp.2 100 Lab.A:Cmp.4:Exp.2 Lab.A:Cmp.3:Exp.3 Lab.A:Cmp.4:Exp.3 Lab.B:Cmp.3:Exp.1 Lab.B:Cmp.4:Exp.1 Lab.B:Cmp.3:Exp.2 Lab.B:Cmp.4:Exp.2 Lab.B:Cmp.3:Exp.3 Lab.B:Cmp.4:Exp.3 50 50 0 1 100 0 1 100 dose dose
Different labs and different compounds Compound Cmp.1 Compound Cmp.2 200 Lab.A:Cmp.2:Exp.1 200 Lab.A:Cmp.2:Exp.2 Lab.A:Cmp.2:Exp.3 Lab.B:Cmp.2:Exp.1 150 Lab.B:Cmp.2:Exp.2 150 response response Lab.B:Cmp.2:Exp.3 Lab.A:Cmp.1:Exp.1 100 100 Lab.A:Cmp.1:Exp.2 Lab.A:Cmp.1:Exp.3 Lab.B:Cmp.1:Exp.1 Lab.B:Cmp.1:Exp.2 50 Lab.B:Cmp.1:Exp.3 50 “What is the intra- and interlaboratory variability of the assay?” 0 1 100 0 1 100 dose dose basically means Compound Cmp.3 Compound Cmp.4 “What is the intra- and interlaboratory variability of ED 50 values ?” 200 200 150 150 response response Lab.A:Cmp.3:Exp.1 Lab.A:Cmp.4:Exp.1 100 Lab.A:Cmp.3:Exp.2 100 Lab.A:Cmp.4:Exp.2 Lab.A:Cmp.3:Exp.3 Lab.A:Cmp.4:Exp.3 Lab.B:Cmp.3:Exp.1 Lab.B:Cmp.4:Exp.1 Lab.B:Cmp.3:Exp.2 Lab.B:Cmp.4:Exp.2 Lab.B:Cmp.3:Exp.3 Lab.B:Cmp.4:Exp.3 50 50 0 1 100 0 1 100 dose dose
ED 50 as parameter ◮ F-testing: Should we model with different ED 50 parameters? ◮ Compound i , laboratory j , experimental run k ◮ e ijk the ED 50 parameter for a given experimental run ◮ Model M b : ED 50 parameter for each compound, e ijk = e ij ′ k ′ ◮ Model M w : ED 50 parameter for each cmp:lab, e ijk = e ijk ′ ◮ Model M f : Different ED 50 parameters for each run ? ◮ Reject M b vs M w ⇒ Poor ED 50 (inter-lab) reproducibility ? ◮ Reject M w vs M f ⇒ Poor ED 50 (intra-lab) repeatability ◮ Are we really interested in ED 50 parameters?
ED 50 as observation Compound Cmp.1 Compound Cmp.2 200 Lab.A:Cmp.2:Exp.1 200 Lab.A:Cmp.2:Exp.2 Lab.A:Cmp.2:Exp.3 Lab.B:Cmp.2:Exp.1 150 Lab.B:Cmp.2:Exp.2 150 response response Lab.B:Cmp.2:Exp.3 Lab.A:Cmp.1:Exp.1 100 100 “What is the intra- and interlaboratory variability of the assay?” Lab.A:Cmp.1:Exp.2 Lab.A:Cmp.1:Exp.3 Lab.B:Cmp.1:Exp.1 Lab.B:Cmp.1:Exp.2 translates into 50 Lab.B:Cmp.1:Exp.3 50 “Will ED 50 estimates for the same compound be similar in future 0 1 100 0 1 100 experiments within and between labs?” dose dose Compound Cmp.3 Compound Cmp.4 translates into 200 200 “How well will ED 50 observations for the same compound agree 150 in future experiments within and between labs?” 150 response response Lab.A:Cmp.3:Exp.1 Lab.A:Cmp.4:Exp.1 100 Lab.A:Cmp.3:Exp.2 100 Lab.A:Cmp.4:Exp.2 Lab.A:Cmp.3:Exp.3 Lab.A:Cmp.4:Exp.3 Lab.B:Cmp.3:Exp.1 Lab.B:Cmp.4:Exp.1 Lab.B:Cmp.3:Exp.2 Lab.B:Cmp.4:Exp.2 Lab.B:Cmp.3:Exp.3 Lab.B:Cmp.4:Exp.3 50 50 0 1 100 0 1 100 dose dose
Agreement statistics: Standard problem specification ◮ Two observers Y 1 and Y 2 ◮ Each subject i measured once by each observer: Observations y i 1 , y i 2 ◮ Observers measure the same quantity ◮ No gold standard ◮ Agreement if Y 1 is“close to” Y 2 Y 2 Standard problem ED 50 estimation 45 ◦ -line Observer Laboratory Subject Compound Observation ED 50 estimate Y 1
Limits of Agreement (LOA): Basic idea 5 5.02 ∆ := Y 1 − Y 2 3.56 µ ∆ , σ 2 � � ∆ ∼ N ∆ 0 Y 1 − Y 2 −1.07 LOA u , l := µ ∆ ± 1 . 96 σ ∆ −5 −5.7 lower LOA −7.16 lower bound of CI of lower LOA 12 14 16 18 20 1 2 ( Y 1 + Y 2 ) “Reference interval” : LOA expected to contain the difference of observations for 95% of pairs of future observations Bland and Altman 1986, 1999
ED 50 estimates as observations Compound 1 Compound 2 Compound 3 Compound 4 Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Cmp.1 Cmp.1 Cmp.1 Cmp.2 Cmp.2 Cmp.2 Cmp.3 Cmp.3 Cmp.3 Cmp.4 Cmp.4 Cmp.4 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 5 Laboratory B 4 3 2 1 log(ED50) Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Cmp.1 Cmp.1 Cmp.1 Cmp.2 Cmp.2 Cmp.2 Cmp.3 Cmp.3 Cmp.3 Cmp.4 Cmp.4 Cmp.4 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 5 Laboratory A 4 3 2 1 Experimental run
ED 50 estimates as observations Compound 1 Compound 2 Compound 3 Compound 4 Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Cmp.1 Cmp.1 Cmp.1 Cmp.2 Cmp.2 Cmp.2 Cmp.3 Cmp.3 Cmp.3 Cmp.4 Cmp.4 Cmp.4 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 5 Laboratory B 4 3 2 1 log(ED50) Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Cmp.1 Cmp.1 Cmp.1 Cmp.2 Cmp.2 Cmp.2 Cmp.3 Cmp.3 Cmp.3 Cmp.4 Cmp.4 Cmp.4 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 5 Laboratory A ◮ 2 random variables L A and L B 4 ◮ Realization of L X is a log(ED 50 ) observation in laboratory X for a 3 randomly chosen compound 2 ◮ Idea: Use LOA to assess log(ED 50 ) reproducibility between labs 1 ◮ Difference of logs with convenient interpretation in terms of ratios Experimental run ◮ Multiple ED 50 observations for each compound in each lab
ED 50 estimates as observations: Variance components Compound 1 Compound 2 Compound 3 Compound 4 Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Lab.B Cmp.1 Cmp.1 Cmp.1 Cmp.2 Cmp.2 Cmp.2 Cmp.3 Cmp.3 Cmp.3 Cmp.4 Cmp.4 Cmp.4 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 5 Laboratory B 4 3 2 1 log(ED50) Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Lab.A Cmp.1 Cmp.1 Cmp.1 Cmp.2 Cmp.2 Cmp.2 Cmp.3 Cmp.3 Cmp.3 Cmp.4 Cmp.4 Cmp.4 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 Exp.1 Exp.2 Exp.3 5 Laboratory A 4 3 2 Variability of true log(ED 50 ) value across compounds 1 Var( L A ) = σ 2 t + . . . Experimental run Var( L B ) = σ 2 t + . . .
Recommend
More recommend
