Combination of protein biomarkers UseR! 2009 Rennes, July 8, 2009 Xavier Robin 1
Outline Outline Introduction clinical problem biomarkers Combining biomarkers ROC Curves Comparison Comparing panels with single biomarkers Conclusion Acknowledgements 2
Aneurysmal Subarachnoid Aneurysmal Subarachnoid hemorrhage (aSAH) hemorrhage (aSAH) SAH: rupture of a blood vessel just outside the brain Main cause (80%): aneurysm (dilation of a blood vessel) : aSAH 1/10 000 people each year “Young patients” (mean: 55) Many patients are chronically disabled Needs: prognosis tools to aid physician for the management of patient and family. 3
Biomarkers Biomarkers Biomarkers are “characteristics objectively measured” whose concentration are different in two groups of patients. Diagnosis, prognosis, therapeutic monitoring, … At the BPRG we are interested in several brain damage markers discovered by comparing ante - and post - mortem cerebrospinal fluid When several proteins are considered in a single classifier (potentially with clinical information) one calls this a panel New overfitting and reproducibility problems 4
Biomarkers Biomarkers Name Biological Role Marker for H-FABP Fatty acid-binding Lipid Binding Cardiac, brain protein damage Nucleoside regulation of Brain damage NDKA diphosphate apoptosis kinase A Ubiquitin fusion protein degradation Brain damage UFD1 degradation protein 1 Protein DJ-1 protein binding Brain damage, DJ1 Parkinson Protein S100-B protein binding Brain damage S100B Troponin I, cardiac protein binding Cardiac Troponin-I muscle (but also brain) 5
Cohorts and Cohorts and Goal of the study Goal of the study Cohort: 113 patients validation: 25 patients from the same hospital collected later Goal: Predict outcome after 6 months Focus attention on patients at risk of poor outcome Want a high specificity to avoid false positives (good outcome patients classified as poor outcome) and give them the best management. Use partial area under the ROC curve With biomarkers or a combination of them 6
Data Description Data Description Quantitative measure of protein (continuous) and clinical (discrete) data Box-cox transformation (Yeo and Johnson, 2000) 7
Biomarkers & Clinical parameters Biomarkers & Clinical parameters NDKA H-FABP S100B S100B is the 100 100 100 best protein 80 80 80 60 60 60 biomarker 40 40 40 20 20 20 WFNS is the 0 0 0 100 80 60 40 20 0 100 80 60 40 20 0 100 80 60 40 20 0 best clinical Troponin-I UFD1 DJ1 100 100 100 Sensitivity (%) marker 80 80 80 60 60 60 Their accura 40 40 40 20 20 20 cies are low 0 0 0 (3.4% of the 100 80 60 40 20 0 100 80 60 40 20 0 100 80 60 40 20 0 WFNS Fisher Age total area) 100 100 100 80 80 80 60 60 60 40 40 40 — 113-set 20 20 20 — 25-set 0 0 0 ◊ pAUC 100 80 60 40 20 0 100 80 60 40 20 0 100 80 60 40 20 0 Specificity (%) 8
Combining biomarkers Combining biomarkers RIL SVM 100 100 RIL : simple threshold- 80 80 based method 60 60 40 40 20 20 Packages used: 0 0 100 80 60 40 20 0 100 80 60 40 20 0 LM GLM kernlab (svm) 100 100 Sensitivity (%) 80 80 stats (lm & glm) 60 60 40 40 pls 20 20 0 0 100 80 60 40 20 0 100 80 60 40 20 0 kknn PLS KKNN 100 100 80 80 60 60 40 40 20 20 — 113-set 0 0 — 25-set 100 80 60 40 20 0 100 80 60 40 20 0 ◊ pAUC Specificity (%) 9
Combining biomarkers Combining biomarkers RIL: best on Mean of pAUC of Validation cross-validation k*n pAUCs means of n predictions SVM: best on RIL 5.6 4.0 3.6 SVM 3.1 3.1 5.3 validation cohort PLS 4.2 2.3 3.2 LM 5.1 3.7 2.8 GLM 4.6 3.1 2.8 KNN 2.9 1.0 2.6 Different methods to compute pAUC give different results Validation cohort is small (25 patients) 10
Comparing ROC Curves Comparing ROC Curves 100 Several methods are available: 80 Bootstrapping Sensitivity (%) 60 DeLong, 1988 40 We will compare: 20 S100B RIL The best individual 0 100 80 60 40 20 0 predictor (S100B) Specificity (%) The best combination method (RIL) and see how comparison methods perform 11
Comparing ROC curves: Bootstrap Comparing ROC curves: Bootstrap D = AUC1 − AUC2 Sigma computed by bootstrapping D ~ N(0, 1) (see Hanley & McNeil, Radiology, 1983) 12
Comparing ROC curves: Bootstrap Comparing ROC curves: Bootstrap 100 Advantage: 80 Flexible Sensitivity (%) 60 Applicable to pAUCs p = 0.082 40 Disadvantage: 20 S100B RIL Slow 0 100 80 60 40 20 0 Same observations Specificity (%) Resampled 60 Frequency 40 20 0 0.0 0.2 0.4 0.6 0.8 1.0 p-values 13
Comparing ROC curves: DeLong Comparing ROC curves: DeLong Based on U statistics: n m AUC = 1 mn ∑ ∑ X i ,Y j j = 1 j = 1 X ,Y = { Y X 1 Y = X ½ Y X 0 Variance computed according to Hoeffding's theory DeLong et al., Biometrics, 1988 14
Comparing ROC curves: DeLong Comparing ROC curves: DeLong 100 Advantages: 80 Fast and easy Sensitivity (%) 60 Based on robust statistics p = 0.081 40 Non parametric 20 S100B RIL 0 100 80 60 40 20 0 Specificity (%) Resampled Frequency 40 20 0 0.0 0.2 0.4 0.6 0.8 1.0 p-values 15
Comparing ROC curves Comparing ROC curves Bootstrap is flexible and displays good results DeLong’s method works equally well pAUC computations should be straightforward Combinations does not appear significantly better than individual biomarkers 16
Comparing panels with single Comparing panels with single biomarkers biomarkers We want to be sure that the chosen panel performs better than the biomarkers taken individually Panel performances are cross-validated; individual biomarkers are not How can we compare them fairly? Do we absolutely need a “validation” cohort? 17
Conclusion Conclusion The use of protein biomarkers is already widely spread We are not sure if using combination of several protein or clinical parameters can significantly increase accuracy we don't know the influence of no cross- validation for single molecules Acceptance by the medical community Model must be simple and clear, understandable to non-experts 18
Acknowledgements Acknowledgements Natacha Turck Markus Müller Alexandre Hainard Frédérique Lisacek Loïc Dayon Other Natalia Tiberti collaborations Catherine Fouda Louis Puybasset Nadia Walter Paola Sanchez Jean-Charles Sanchez ����������������� Laszlo Vutskits Marianne Gex-Fabry 19
Recommend
More recommend
