Measuring variable importance in random forests Variable Variable - PowerPoint PPT Presentation

Measuring variable importance in random forests Variable Variable importance in RF importance in RF 1 1 1 1 Start Start Start Start p < 0.001 p < 0.001 p < 0.001 p < 0.001 A Comparison of Different ≤ 8 > 8 ≤ 12 > 12 ≤ 1 > 1 2 3 2 7 2 3 ≤ ≤ 8 > 8 > Conditional n = 13 Age Age n = 49 n = 8 Number Conditional y = (0.308, 0.692) p < 0.001 p < 0.001 y = (1, 0) y = (0.375, 0.625) p < 0.001 2 3 n = 15 Start ≤ 87 > 87 ≤ 68 > 68 ≤ 4 > 4 y = (0.4, 0.6) p < 0.001 variable 4 5 3 6 4 7 variable n = 36 Start Number n = 12 Age n = 31 ≤ ≤ 14 > 14 > y = (1, 0) p < 0.001 p < 0.001 y = (0.25, 0.75) p < 0.001 y = (0.806, 0.194) Variable Importance Measures ≤ 13 > 13 ≤ 4 > 4 ≤ 125 > 125 importance in RF 4 5 importance in RF n = 34 n = 32 6 7 4 5 5 6 y = (0.882, 0.118) y = (1, 0) n = 16 n = 16 n = 11 n = 9 n = 31 n = 11 y = (0.75, 0.25) y = (1, 0) y = (1, 0) y = (0.556, 0.444) y = (1, 0) y = (0.818, 0.182) 1 Number 1 1 Other variable 1 p < 0.001 Start Start Other variable Start p < 0.001 p < 0.001 p < 0.001 ≤ 5 > 5 2 9 ≤ 12 > 12 ≤ 14 > 14 Age n = 11 importance 2 7 2 7 importance ≤ ≤ 12 > 12 > p < 0.001 y = (0.364, 0.636) Age Number Age n = 35 ≤ 81 > 81 p < 0.001 p < 0.001 p < 0.001 y = (1, 0) 2 3 3 4 n = 38 Number n = 33 Start ≤ 18 > 18 ≤ 3 > 3 ≤ 71 > 71 measures y = (0.711, 0.289) p < 0.001 y = (1, 0) p < 0.001 measures 3 4 8 9 3 4 ≤ 12 > 12 n = 10 Number n = 28 n = 21 n = 15 Start 5 6 y = (0.9, 0.1) p < 0.001 y = (1, 0) y = (0.952, 0.048) y = (0.933, 0.067) p < 0.001 ≤ 3 ≤ > 3 > n = 13 Start y = (0.385, 0.615) p < 0.001 ≤ 4 > 4 ≤ 12 > 12 4 5 ≤ 15 > 15 n = 25 n = 18 5 6 5 6 7 8 y = (1, 0) y = (0.889, 0.111) n = 12 n = 10 n = 16 n = 15 Carolin Strobl (LMU M¨ unchen) n = 12 n = 12 Summary y = (0.417, 0.583) y = (0.2, 0.8) y = (0.375, 0.625) y = (0.733, 0.267) Summary y = (0.833, 0.167) y = (1, 0) 1 1 Start 1 1 Number p < 0.001 Start Start p < 0.001 p < 0.001 p < 0.001 References ≤ 12 ≤ > > 12 ≤ 6 > 6 References 2 7 2 7 ≤ 12 > 12 ≤ 8 > 8 Age Start Number n = 10 p < 0.001 p < 0.001 p < 0.001 y = (0.5, 0.5) 2 5 2 5 ≤ 27 ≤ > 27 > ≤ ≤ 13 > 13 Age Start Start Age ≤ 3 > 3 p < 0.001 p < 0.001 p < 0.001 p < 0.001 3 4 8 9 3 6 n = 10 Number n = 11 n = 37 Start n = 37 y = (1, 0) p < 0.001 y = (0.818, 0.182) y = (1, 0) p < 0.001 y = (0.865, 0.135) ≤ 81 > 81 ≤ 13 > 13 ≤ 3 > 3 ≤ 136 > 136 ≤ 4 ≤ > > 4 ≤ 13 > 13 3 4 6 7 3 4 6 7 5 6 n = 20 n = 16 n = 11 n = 34 n = 12 n = 14 n = 47 n = 8 4 5 n = 14 n = 9 y = (0.85, 0.15) y = (0.188, 0.812) y = (0.818, 0.182) y = (1, 0) y = (0.667, 0.333) y = (0.143, 0.857) y = (1, 0) y = (0.75, 0.25) n = 10 n = 24 y = (0.357, 0.643) y = (0.111, 0.889) y = (0.8, 0.2) y = (1, 0) 1 1 1 1 Start Start Start Start p < 0.001 p < 0.001 p < 0.001 p < 0.001 ≤ 8 > 8 Wien, J¨ anner 2009 ≤ 8 ≤ > > 8 ≤ 12 > 12 ≤ 12 > 12 2 3 n = 18 Start y = (0.5, 0.5) p < 0.001 2 5 2 5 2 3 Start Start Age Start n = 28 Start ≤ 12 > 12 p < 0.001 p < 0.001 p < 0.001 p < 0.001 y = (0.607, 0.393) p < 0.001 4 5 n = 18 Number y = (0.833, 0.167) p < 0.001 ≤ 1 ≤ > > 1 ≤ 12 ≤ > > 12 ≤ 71 > 71 ≤ 14 > 14 ≤ 14 > 14 ≤ 3 > 3 3 4 6 7 3 4 6 7 4 5 n = 9 n = 13 n = 12 n = 47 n = 15 n = 17 n = 17 n = 32 n = 21 n = 32 6 7 y = (0.778, 0.222) y = (0.154, 0.846) y = (0.833, 0.167) y = (1, 0) y = (0.667, 0.333) y = (0.235, 0.765) y = (0.882, 0.118) y = (1, 0) y = (0.905, 0.095) y = (1, 0) n = 30 n = 15 y = (1, 0) y = (0.933, 0.067) Measuring variable importance in random forests Measuring variable importance in random forests Variable Variable importance in RF importance in RF Conditional Conditional ◮ Gini importance variable variable importance in RF importance in RF mean Gini gain produced by X j over all trees Other variable Other variable (can be severely biased due to estimation bias and importance importance measures measures mutiple testing; Strobl et al., 2007) Summary Summary References References

Measuring variable importance in random forests The permutation importance within each tree t Variable Variable importance in RF importance in RF Conditional Conditional ◮ Gini importance variable variable importance in RF importance in RF � � � � y ( t ) y ( t ) mean Gini gain produced by X j over all trees � � ( t ) I y i = ˆ ( t ) I y i = ˆ i i ,π j Other variable i ∈ B i ∈ B Other variable VI ( t ) ( x j ) = − (can be severely biased due to estimation bias and importance importance � ( t ) � � ( t ) � � B � B � � � � measures measures � � mutiple testing; Strobl et al., 2007) Summary Summary ◮ permutation importance References References y ( t ) = f ( t ) ( x i ) = predicted class before permuting ˆ i mean decrease in classification accuracy after permuting X j over all trees y ( t ) i ,π j = f ( t ) ( x i ,π j ) = predicted class after permuting X j ˆ (unbiased when subsampling is used; Strobl et al., 2007) � x i ,π j = ( x i , 1 , . . . , x i , j − 1 , x π j ( i ) , j , x i , j +1 , . . . , x i , p Note: VI ( t ) ( x j ) = 0 by definition, if X j is not in tree t What kind of independence corresponds to The permutation importance this kind of permutation? Variable Variable importance in RF importance in RF Conditional Conditional variable variable importance in RF importance in RF obs Y X j Z over all trees: Other variable Other variable 1 y 1 x π j (1) , j z 1 importance importance . . . . . . . . measures measures . . . . Summary Summary � ntree t =1 VI ( t ) ( x j ) i y i x π j ( i ) , j z i VI ( x j ) = . . . . References . . . . References ntree . . . . n y n x π j ( n ) , j z n H 0 : X j ⊥ Y , Z or X j ⊥ Y ∧ X j ⊥ Z H 0 P ( Y , X j , Z ) = P ( Y , Z ) · P ( X j )

What kind of independence corresponds to What kind of independence corresponds to this kind of permutation? this kind of permutation? Variable Variable importance in RF importance in RF Conditional Conditional variable variable importance in RF importance in RF Other variable Other variable the original permutation scheme reflects independence of X j the original permutation scheme reflects independence of X j importance importance measures measures from both Y and the remaining predictor variables Z from both Y and the remaining predictor variables Z Summary Summary References ⇒ a high variable importance can result from violation of References either one! Suggestion: Conditional permutation scheme Technically Variable Variable importance in RF importance in RF obs Y X j Z 1 y 1 x π j | Z = a (1) , j z 1 = a Conditional Conditional variable variable ◮ use any partition of the feature space for conditioning 3 y 3 x π j | Z = a (3) , j z 3 = a importance in RF importance in RF 27 y 27 x π j | Z = a (27) , j z 27 = a Other variable Other variable importance importance 6 y 6 x π j | Z = b (6) , j z 6 = b measures measures 14 y 14 x π j | Z = b (14) , j z 14 = b Summary Summary 33 y 33 x π j | Z = b (33) , j z 33 = b References References . . . . . . . . . . . . H 0 : X j ⊥ Y | Z H 0 P ( Y , X j | Z ) = P ( Y | Z ) · P ( X j | Z ) H 0 or P ( Y | X j , Z ) = P ( Y | Z )

Technically Toy example Variable Variable spurious correlation between shoe size and reading skills in importance in RF importance in RF school-children Conditional Conditional variable variable ◮ use any partition of the feature space for conditioning importance in RF importance in RF ◮ here: use binary partition already learned by tree > mycf <- cforest(score ~ ., data = readingSkills, Other variable Other variable importance importance for each tree + control = cforest_unbiased(mtry = 2)) measures measures > varimp(mycf) ◮ determine variables to condition on (via threshold ) Summary Summary nativeSpeaker age shoeSize ◮ extract their cutpoints References References 12.62926 74.89542 20.01108 ◮ generate partition using cutpoints as bisectors > varimp(mycf, conditional = TRUE) nativeSpeaker age shoeSize Strobl et al. (2008) 11.808192 46.995336 2.092454 from party 0.9-991 Peptide-binding data Simulation results Variable Variable importance in RF importance in RF 25 mtry = 1 Conditional Conditional 15 ● ● unconditional ● ● variable variable ● ● importance in RF importance in RF 5 ● 0 ● ● ● ● ● 0.005 Other variable Other variable importance importance 50 measures measures mtry = 3 0 30 Summary Summary ● ● ● ● 10 ● References References conditional ● ● 0 ● ● ● ● ● 0.005 80 mtry = 8 60 40 ● 0 ● * 20 ● ● ● h2y8 flex8 pol3 ● 0 ● ● ● ● ● ● 1 2 3 4 5 6 7 8 9 10 11 12 variable