Ana Cristina Braga: acb@ dps.uminho.pt Lino Costa: lac@ - PowerPoint PPT Presentation

Ana Cristina Braga: acb@ dps.uminho.pt Lino Costa: lac@ dps.uminho.pt Pedro Nuno Oliveira: pno@ dps.uminho.pt

 Development of a new methodology which allows the comparison of ROC curves that cross each other;  Identification of the regions of the ROC space in which the tests have better performance;  Construction of nonparametric confidence intervals for measures proposed. 2

1,00 0,75 TPF (sensitivity) − A A = 1 2 Z ~ N (0,1) 0,50 + − 2 2 SE SE 2 rSE SE 1 2 1 2 high 0,25 little moderate ref. 0,00 0,00 0,25 0,50 0,75 1,00 FPF (1-specificity) 3

1. Sampling the ROC curves  S ampling lines st art ing from a reference point  Int ersect ion point s of t he sampling lines wit h t he ROC curves  Euclidean dist ance from t he int ersect ion point s t o t he reference point 2. Measures  Ext ension – proport ion of t he space where a curve is bet t er t han ot her  Locat ion – regions of t he space where a curve is bet t er t han ot her 4

3. Nonparametric statistical evaluation  S tatistical Evaluation of the Difference between Areas - Permutation test  Confidence Interval for the Difference of the areas - bootstrap resampling 5 8/18/2010

Reference point: (1,0) Number of sampling lines: 3 Line Slope Outcome L 1 22.5º Curve 2 L 2 45º Curve 1 L 3 67.5º Curve 1 Extension measure Curve 1: 66.7 % Curve 2: 33.3 % 6

0,0006 0,0005 0,0004 Difference between areas 0,0003 0,0002 0,0001 0 0º 10º 20º 30º 40º 50º 60º 70º 80º 90º -0,0001 -0,0002 -0,0003 Slope of the sampling line sensitivity sensitivity specificity specificity Extension measure Location measure [0 ° , 15.4 ° ] and [28 ° , 90 ° ] Curve 1: 86.4 % [15.8 ° , 27.6 ° ] 7 Curve 2: 13.6 %

 Based on the notion of permutation tests, the difference of the areas between the two empirical ROC curves are permuted;  Bootstrapped confidence intervals are calculated;  All computations performed using R package. 8

Conditions: ( ) f x  Generate distributions of abnormal ( ) and A f ( ) x normal ( ) for two modalities; N x  Greater values of variable correspond to the abnormal status; = X ~ N (50,25) X ~ N (60,25) n n , and ;  N A A N K = 100  S ampling lines: . 9

= n n AUC1 SE1 AUC2 SE2 AUC1-AUC2 A N Mean 0.918 0.0384 0.925 0.0358 -0.00626 25 Median 0.922 0.039 0.926 0.0363 -0.008 minimum 0.813 0.0073 0.826 0.0023 -0.1248 0.992 0.0657 0.998 0.0595 0.1664 maximum Mean 0.924 0.0256 0.920 0.0264 0.00394 50 Median 0.924 0.0259 0.921 0.0266 0.004 minimum 0.816 0.0087 0.806 0.0130 -0.1236 maximum 0.985 0.0428 0.971 0.0433 0.1236 0.922 0.0185 0.922 0.0183 -0.00048 Mean 100 Median 0.923 0.0185 0.923 0.0181 0.0001 0.867 0.0113 0.855 0.0107 -0.0834 minimum maximum 0.967 0.0253 0.965 0.0265 0.0649 10

Z Test n Rejection No Rejection B Test 25 7 4 Rejection 50 7 1 100 12 1 25 0 189 No 50 1 191 Rejection 100 4 183 ≥ 5 # Cross 0 1 2 3 4 n=25 31 61 60 29 18 1 Freq. n=50 12 48 41 36 25 38 n=100 10 31 30 41 32 56 11

1.0 0.8 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0

Areas Between ROC Curves 0.008 0.006 0.004 0.002 Area 0.000 -0.002 -0.004 0 20 40 60 80 Degrees 13 8/18/2010

1 P1 AUC1: 0.91978 P2 0,9 S E(AUC1)=0.007 P int P3 0,8 AUC2: 0.92580 P4 S E(AUC2)=0.005 TPF (sensitivity) 0,7 Diff: -0.00603 0,6 0,5 0,4 P5 0,3 0,2 Mod1 0,1 P6 Mod2 0 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 FPF (1-specificity) 14

-0.02247822 < diff (boot)< 0.01017388 Areas Between ROC Curves 0.002 0.001 Area 0.000 -0.001 -0.002 0 20 40 60 80 Extension measure Location measure Degrees ]25.8 ° , 48.8 ° ] Curve 1: 25.8 % ]48.8 ° , 82.8 ° ] Curve 2: 38.6 % 15 8/18/2010

 The proposed methodology allows partial and global comparisons of two ROC curves without a fixing FPF;  Graphical representation that elucidates the dominance regions in terms of sensitivity and specificity;  Nonparametric alternative based on bootstrap resampling for the comparison of two ROC curves when they cross each other . 16

 To study the randomness of the crossing points between ROC curves;  To extend the methodology to the comparison of more than two ROC curves. 17

J. A. Hanley and B. J. McNeil. A Met hod of Comparing t he Areas Under Receiver Operat ing  Charact erist ic Curves Derived from t he S ame Cases. Radiology, 148(3):839-843, 1983. IS S N 0033- 8419. E. R. DeLong, D. M. DeLong, and D. I. Clarkepearson. Comparing The Areas Under 2 or More Correlat ed  Receiver Operat ing Charact erist ic Curves - A Nonparamet ric Approach. Biomet rics, 44 (3):837-845, S ep 1988. IS S N 0006-341x. H. E. Rocket t e, N. A. Obuchowski, and D. Gur. Nonparamet ric-Est imation of Degenerate ROC Dat a S et s  Used For Comparison of Imaging-S yst ems. Invest igat ive Radiology, 25(7):835-837, Jul 1990. IS S N 0020- 9996.  D. K. McClish. Analyzing a Port ion of t he ROC Curve. Medical Decision Making, 9(3):190-195, Jul-S ep 1989. IS S N 0272-989x. S . Wieand, M. H. Gail, B. R. James, and K. L. James. A Family of Nonparamet ric S t at ist ics for  Comparing Diagnost ic Markers wit h Paired or Unpaired Dat a. Biomet rika, 76(3):585-592, S ep 1989. IS S N 0006-3444.  Y . L. Jiang, C. E. Met z, and R. M. Nishikawa. A receiver operating: Charact erist ic part ial area index for highly sensit ive diagnost ic t est s. Radiology, 201(3):745-750, DEC 1996. IS S N 0033-8419. 1995 RS NA S cient ific Assembly, CHICAGO, IL, NOV 26-DEC 01, 1995. D. D. Zhang, X. H. Zhou, D. H. Freeman, and J. L Freeman. A Non-Paramet ric Met hod for The  Comparison of Part ial Areas Under ROC Curves and It s Applicat ion t o Large Healt h Care Dat a S et s. S t at ist ics In Medicine, 21(5):701-715, Mar 2002. IS S N 0277-6715. L. E. Dodd and M. S . Pepe. Part ial AUC est imat ion and regression. Biomet rics, 59(3):614-623, S EP 2003.  IS S N 0006-341X. C. M. Fonseca and P . J. Fleming. On t he performance assessment and comparison of st ochast ic  mult iobj ect ive opt imizers. In Proceedings of Parallel Problem S olving from Nat ure IV , pages 584- 593. S pringer, 1996. 18 8/18/2010