Versions of Random Forests: Properties and Performances Choongsoon - PowerPoint PPT Presentation

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES Versions of Random Forests: Properties and Performances Choongsoon Bae Google Inc. U.C.Berkeley March 26, 2009 Joint work with Peter Bickel

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES Outline Definition Motivation Breiman’s Random Forests CART Purely Random Forest CART construction Bagging averaged 1-nearest Examples neighbor classifier Bagging Data Adaptive Weighted Random Definition Forests Comparison Performances Basic Idea and Issues Example I Random Forests Example II

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES The truth ✛ ✛ Y Nature X Goals : Prediction : Information

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES Large and High dimensional Data Set • Internet advertisements data : 3, 279 data, 1, 558 attributes. ( n = 3, 279, d = 1, 558). • Microsoft web data : 37, 711 data, 294 attributes. ( n = 37, 711, d = 294). • Corel Image data : 68, 040 images, 89 attributes. ( n = 68, 040, d = 89). • Spam E-mail Data: 4, 601 data, 57 attribute. ( n = 4, 601, d = 57).

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES Issues • Fast calculation. • Excellent accuracy. • Good insights into the inside of black box

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES Machine Learning Methods • Kernel smoothing. • Classification and Regression Tree (CART). • Support Vector Method (SVM). • Boosting. • Bagging(Bootstrap Aggregating). • Random Forests.

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES Outline Definition Motivation Breiman’s Random Forests CART Purely Random Forest CART construction Bagging averaged 1-nearest Examples neighbor classifier Bagging Data Adaptive Weighted Random Definition Forests Comparison Performances Basic Idea and Issues Example I Random Forests Example II

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART ✎ ☞ ✍ ✌ 400 makes, models and vehicle types ☛ ✟ ☛ ✟ Yes No ❄ ❄ ✡ ✠ ✡ ✠ Other makes and models Ford Taurus ☛ ✟ ☛ ✟ Yes No ❄ ❄ ✡ ✠ ✡ ✠ Other makes and models Honda Accord ☛ ✟ ☛ ✟ Yes No ❄ ❄ ✡ ✠ ✡ ✠ · · · · · · Ford F-150 Taken from Critical Features of HIgh Performance Decision Trees Salford Systems

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(Growing) ✬ ✩ Model X ( 1 ) , . . . , X ( d ) � � �� ∈ { 1, . . . , K } × R d ✫ Y i , ✪ i i i = 1, . . . , n ✛✘ ❄ ✚✙

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(Growing) ✬ ✩ n � = � � X ( 1 ) �� α 1 , ˆ � ˆ ∑ Y i � = α 1 1 ≤ γ 1 β 1 , ˆ γ 1 argmin 1 i ( α 1, β 1, γ 1 ) ∈ R 3 i = 1 Model � � X ( 1 ) �� . + 1 Y i � = β 1 1 > γ 1 . i X ( 1 ) , . . . , X ( d ) � � �� . ∈ { 1, . . . , K } × R d ✫ Y i , ✪ i i i = 1, . . . , n � = n � � X ( d ) �� α d , ˆ � ˆ β d , ˆ argmin ∑ 1 Y i � = α d 1 ≤ γ d γ d i ( α d , β d , γ d ) ∈ R 3 i = 1 � � X ( d ) �� + 1 Y i � = β d 1 > γ d i ✛✘ ❄ ✚✙

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(Growing) ✬ ✩ n � = � � X ( 1 ) �� α 1 , ˆ � ˆ ∑ Y i � = α 1 1 ≤ γ 1 β 1 , ˆ γ 1 argmin 1 i ( α 1, β 1, γ 1 ) ∈ R 3 i = 1 Model � � X ( 1 ) �� . + 1 Y i � = β 1 1 > γ 1 . i X ( 1 ) , . . . , X ( d ) � � �� . ∈ { 1, . . . , K } × R d ✫ Y i , ✪ i i i = 1, . . . , n � = n � � X ( d ) �� α d , ˆ � ˆ β d , ˆ argmin ∑ 1 Y i � = α d 1 ≤ γ d γ d i ( α d , β d , γ d ) ∈ R 3 i = 1 � � X ( d ) �� + 1 Y i � = β d 1 > γ d i ✛✘ ❄ ⇓ n � � X ( j ) �� � � X ( j ) �� ✚✙ ˆ Y i � = ˆ t = argmin ∑ Y i � = ˆ ≤ ˆ + 1 > ˆ 1 α j 1 γ j β j 1 γ j i i j = 1,..., d i = 1

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(Growing) ✬ ✩ n � = � � X ( 1 ) �� α 1 , ˆ � ˆ ∑ Y i � = α 1 1 ≤ γ 1 β 1 , ˆ γ 1 argmin 1 i ( α 1, β 1, γ 1 ) ∈ R 3 i = 1 Model � � X ( 1 ) �� . + 1 Y i � = β 1 1 > γ 1 . i X ( 1 ) , . . . , X ( d ) � � �� . ∈ { 1, . . . , K } × R d ✫ Y i , ✪ i i i = 1, . . . , n � = n � � X ( d ) �� α d , ˆ � ˆ β d , ˆ argmin ∑ 1 Y i � = α d 1 ≤ γ d γ d i ( α d , β d , γ d ) ∈ R 3 i = 1 � � X ( d ) �� + 1 Y i � = β d 1 > γ d i ✛✘ ❄ ⇓ n X ( ˆ t ) � � X ( j ) �� � � X ( j ) �� ✚✙ ˆ Y i � = ˆ t = argmin ∑ Y i � = ˆ ≤ ˆ + 1 > ˆ 1 α j 1 γ j β j 1 γ j i i j = 1,..., d i = 1 ✛✘ ✛✘ X ( ˆ t ) ≤ ˆ X ( ˆ t ) > ˆ γ ˆ γ ˆ t t ❄ ❄ ✚✙ ✚✙

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(Growing) ♠ X ( 3 )

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(Growing) ♠ X ( 3 ) ♠ ♠ ❄ ❄ X ( 4 ) X ( 1 )

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(Growing) ♠ X ( 3 ) ♠ ♠ ❄ ❄ X ( 4 ) X ( 1 ) ♠ ♠ ❄ ❄ X ( 3 ) X ( 6 ) ♠ ♠ ♠ ♠ ❄ ❄ ❄ ❄ X ( 2 ) X ( 4 ) X ( 2 ) X ( 1 )

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(Growing) ♠ X ( 3 ) ♠ ♠ ❄ ❄ X ( 4 ) X ( 1 ) ♠ ♠ ♠ ♠ ❄ ❄ ❄ ❄ X ( 1 ) X ( 4 ) X ( 3 ) X ( 6 ) ♠ ♠ ♠ ♠ ♠ ♠ ♠ ♠ ❄ ❄ ❄ ❄ ❄ ❄ ❄ ❄ X ( 5 ) X ( 7 ) X ( 2 ) X ( 4 ) X ( 2 ) X ( 4 ) X ( 2 ) X ( 1 )

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(pruning) ♠ X ( 3 ) ♠ ♠ ❄ ❄ X ( 4 ) X ( 1 ) ♠ ♠ ♠ ♠ ❄ ❄ ❄ ❄ X ( 1 ) X ( 4 ) X ( 3 ) X ( 6 ) � � ♠ ♠ ♠ ♠ ♠ ♠ ♠ ♠ ❄ ❄ ❄ ❄ ❄ ❄ ❄ ❄ X ( 5 ) X ( 7 ) X ( 2 ) X ( 4 ) X ( 2 ) X ( 4 ) X ( 2 ) X ( 1 )

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(pruning) ♠ X ( 3 ) ♠ ♠ ❄ ❄ X ( 4 ) X ( 1 ) ♠ ♠ ♠ ♠ ❄ ❄ ❄ ❄ X ( 1 ) X ( 4 ) X ( 3 ) X ( 6 ) � � ♠ ♠ ♠ ♠ ♠ ♠ ❄ ❄ ❄ ❄ ❄ ❄ X ( 5 ) X ( 7 ) X ( 2 ) X ( 4 ) X ( 2 ) X ( 1 )

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(pruning) ♠ X ( 3 ) ♠ ♠ ❄ ❄ X ( 4 ) X ( 1 ) ♠ ♠ ♠ ♠ ❄ ❄ ❄ ❄ X ( 1 ) X ( 4 ) X ( 3 ) X ( 6 ) ♠ ♠ ♠ ♠ ❄ ❄ ❄ ❄ X ( 2 ) X ( 4 ) X ( 2 ) X ( 1 )

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART(pruning) ♠ X ( 3 ) ♠ ♠ ❄ ❄ X ( 4 ) X ( 1 ) ♠ ♠ ♠ ♠ ❄ ❄ ❄ ❄ X ( 1 ) X ( 4 ) X ( 3 ) X ( 6 ) ♠ ♠ ♠ ♠ ❄ ❄ ❄ ❄ X ( 2 ) X ( 4 ) X ( 2 ) X ( 1 ) ❄ ❄ ❄ ❄ ❄ ❄ Majority Majority Majority Majority Majority Majority

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART - I • Advantages • Universally applicable to both classification and regression problems. • Deals with categorical variables efficiently. • Invariant to monotone transformation of input variables. • High resistance to irrelevant input variables. • Extremely robust to the effect of outliers. • Computing is fast. • Provide valuable insights for data structure (Interpretation).

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES CART - II • Drawbacks • Poor accuracy - SVM often have 30% lower error rates than CART. • Instability (high variance) - If we change the data a little, the tree picture can be change a lot.

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES Example I • Internet advertisements data (From UCI Machine Learning Repository) • A set of possible advertisements on internet pages. • Task : Predict whether an image is an advertisement. • Number of data: 3,279 (458 ads, 2821 non-ads) • 1,558 independent variables • Geometry of image, phrases occuring in the URL, image’s URL, the anchor text, word near the anchor text. Accuracy of CART(Matlab) : 0.9508 with 10-fold cross validation.

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES Example II • Spam E-mail data (From UCI Machine Learning Repository) • Task : Classify E-mail as spam or non-spam. • Number of data: 4,601 (2788 spam, 1813 non-spam) • 57 independent variables • Percentage of words in the e-mail that match a certain word. Accuracy of CART(Matlab) : 0.9194 with 10-fold cross validation.

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES Outline Definition Motivation Breiman’s Random Forests CART Purely Random Forest CART construction Bagging averaged 1-nearest Examples neighbor classifier Bagging Data Adaptive Weighted Random Definition Forests Comparison Performances Basic Idea and Issues Example I Random Forests Example II

M OTIVATION CART B AGGING R ANDOM F ORESTS P ERFORMANCES Bagging I • Ensemble of base learners. M  1 ∑ T m ( X ) (Regression)    M  ˆ m = 1 F ( X ) = M ∑  1 ( T m ( X ) = j ) argmax (Classification)    j m = 1 where T m : base learner. • Making base learners is different from Boosting. • Use bootstrap sample to make base learners.