Ensembled Multivariate Adaptive Regression Splines Ensembled - PowerPoint PPT Presentation

CO COMPSTAT2010 in Paris S 2010 Ensembled Multivariate Adaptive Regression Splines Ensembled Multivariate Adaptive Regression Splines with Nonnegative Garrote Estimator ith N ti G t E ti t Hiroki Motogaito g Osaka University Osaka University M Masashi Goto hi G t Biostatistical Research Association NPO Biostatistical Research Association, NPO. JAPAN JAPAN

Agenda Agenda g • Introduction and motivation • Introduction and motivation • Tree methods Tree methods  Multivariate Adaptive Regression  M lti  Multivariate Adaptive Regression i t Ad ti R i Splines(MARS) Splines(MARS) p ( )  Bagging MARS  Bagging MARS gg g • Our method proposed Our method proposed O th d d  Ensembled MARS with nonnegative garrote  Ensembled MARS with nonnegative garrote • Example and simulation • Example and simulation • Concluding remarks • Concluding remarks g 2

Agenda Agenda g • Introduction and motivation • Introduction and motivation • Tree methods Tree methods  Multivariate Adaptive Regression  M lti  Multivariate Adaptive Regression i t Ad ti R i Splines(MARS) Splines(MARS) p ( )  Bagging MARS  Bagging MARS gg g • Our method proposed Our method proposed O th d d  Ensembled MARS with nonnegative garrote  Ensembled MARS with nonnegative garrote • Example and simulation • Example and simulation • Concluding remarks • Concluding remarks g 3

Introduction and motivation Introduction and motivation Unstable Unstable Less interpretable Less interpretable ˆ f f ( ( x ) ) ˆ x ˆ x f f ( ( x ) ) f f ( ( x ) ) ˆ x ˆ St bili i Stabilizing f ( ) MARS Bagging gg g (Friedman,1991) (Friedman,1991) (Breiman,1996) (Breiman,1996) M ti Motivation ti a new version MARS that has both stability and interpretability i MARS th t h b th t bilit d i t t bilit 4

Agenda Agenda g • Introduction and motivation • Introduction and motivation • Tree methods Tree methods  Multivariate Adaptive Regression  M lti  Multivariate Adaptive Regression i t Ad ti R i Splines(MARS) Splines(MARS) p ( )  Bagging MARS  Bagging MARS gg g • Our method proposed Our method proposed O th d d  Ensembled MARS with nonnegative garrote  Ensembled MARS with nonnegative garrote • Example and simulation • Example and simulation • Concluding remarks • Concluding remarks g 5

M lti Multivariate Adaptive Regression Splines(Friedman,1991) i t Ad ti R i S li (F i d 1991) • Model form • Model form Regression model Regression model Basis function Basis function M M K K    m ˆ ˆ ˆ            f f m B B ( x ( ) ) q q B B ( ( x ) ) [ [ i i ( ( x t )] )] MARS 0 m   m m ( ( k k , , m m ) ) p p ( ( k k , , m m ) ) ( ( k k , , m m ) )  m 1  k 1 • Algorithms • Algorithms  Forward stepwise  Forward stepwise 0.5 0.45 0.4  Increase basis functions  Increase basis functions 0.35  Backward stepwise  Backward stepwise 0.3 0 3 数の値 基底関数 0 25 0.25  P  Prune off ff 基 0 2 0.2 0.15 0.15      Select the best tree  Select the best tree [ [ ( ( x 0 . 5 )] )] [ [ ( ( x 0 . 5 )] )] 0.1     p p p p 0.05 1 x x 0 0 0 0.2 0 2 0.4 0 4 0 6 0.6 0 8 0.8 1 1 0 2 0.2 0.4 0.5 0.6 0 4 x 0 5 0 6 0.8 1 0 8 p p q= 1 and knot t= 0.5 1 d k t t 0 5 6

Bagging (Breiman,1996) Bagging (Breiman 1996) gg g ( , ) • Model form(Bagging MARS) • Model form(Bagging MARS) Regression model Regression model Each tree Each tree 1 1 E f      ˆ ˆ ˆ  f f f ( ( x ) ) f f ( ( x ) ) : MARS model MARS d l Bagging gg g MARS e e E E e e 1 1 • Algorithms • Algorithms Sample p Bootstrap sample Bootstrap sample Bootstrap sample Bootstrap sample Bootstrap sample ＋ ＋・・・＋ ＋・・・＋ ˆ ˆ ˆ ˆ f f 1 x ( ( x ) ) f f 2 x ( ( ) ) f f ( ( x ) ) f f ( x ( ) ) e E ˆ f ( x ) averaging 7 7

Agenda Agenda g • Introduction and motivation • Introduction and motivation • Previous research Pre io s research  Multivariate Adaptive Regression  M lti  Multivariate Adaptive Regression i t Ad ti R i Splines(MARS) Splines(MARS) p ( )  Bagging MARS  Bagging MARS gg g • Our method proposed Our method proposed O th d d  Ensembled MARS with nonnegative garrote  Ensembled MARS with nonnegative garrote • Example and simulation • Example and simulation • Concluding remarks • Concluding remarks g 8

Proposed method Proposed method p Motivation a new version MARS that has both stability and interpretability a new version MARS that has both stability and interpretability Stable, but less interpretable Stable and interpretable 2 3 1 1 1 Selection 4 & Ranking Ranking 5 5 4 4 Typical tree Typical tree nonnegative nonnegative Bagging Bagging Proposed method Proposed method garrote t (B (Breiman,1995) i 1995) 9

Ensembled MARS Ensembled MARS with non-negative garrote （ 1/2 ） ith non negati e garrote （ 1/2 ） g g • Model form • Model form Regression model g Each tree    ˆ ˆ ˆ x E E ˆ   c c ˆ f f c c e f f ( ( x x ) ) f f ( ( x ) ) : MARS model : MARS model , ： non-negative garrote estimator ： non negative garrote estimator e e e e 1 • Algorithms • Algorithms  Generate Bagging trees  Generate Bagging trees. ˆ c c  Att  Attach h on each tree and estimate using nonnegative h t d ti t i ti e e e e garrote(Breiman,1995). g ( , ) ˆ   ― Select candidate trees(If , the tree is removed). Select candidate trees(If c c 0 0 the tree is removed) ˆ  e    e ˆ E  ˆ f f c e f f ( ( x ) )  Get . e e 1 1 ˆ • • Interpretable structure through typical tree(max Interpretable structure through typical tree(max ) c c ) e 10 10

Ensembled MARS Ensembled MARS with non-negative garrote （ 2/2 ） ith non negati e garrote （ 2/2 ） g g non-negative garrote (Breiman,1995) ti t (B i 1995) N P P          ˆ ˆ P P     ( ( p ) ) 2 2   ˆ { { c } } arg g min ( ( Y c x ) ) c 0 , , c s , subject to , , j , p p 1 1 n n p p p p n n p p p p P P { c }   p 1 n 1 p 1  1 p ˆ     s   1 1 P P where h is the least square estimator and . i th l t ti t d p Ensembled MARS with non-negative garrote g g N N E E E E       ˆ      E 2 { { c c ˆ } } arg arg min min ( ( Y Y c c f f ( ( x x ) ) ) ) c c 0 0 , c c 1 1 , subject to , bj t t e 1 n e e n e e E { { c e c } }    1 1 n 1 1 e 1 1 e 1 1 ˆ ˆ f f x ) ( ( ) where is MARS model. e e n n characteristics h t i ti • All All indicates Bagging. indicates Bagging   c e c 1 1 / / E E  s s s s 1 • Selection of optimal is unnecessary( ). p y( ) 11 11

Agenda Agenda g • Introduction and motivation • Introduction and motivation • Previous research Pre io s research  Multivariate Adaptive Regression  M lti  Multivariate Adaptive Regression i t Ad ti R i Splines(MARS) Splines(MARS) p ( )  Bagging MARS  Bagging MARS gg g • Our method proposed O Our method proposed th d d  Ensembled MARS with non-negative garrote  Ensembled MARS with non negative garrote • Example and simulation • Example and simulation • Concluding remarks • Concluding remarks g 12 12

Literature example Literature example p Prostate cancer data (Stamney et al 1989: Tibshirani 1996) Prostate cancer data (Stamney et al .,1989: Tibshirani,1996) y y • : Level of prostate-specific antigen L l f t t ifi ti  T T x ( ( x , , ,..., x ) ) • : Clinical measures 1 1 8 8 x : Log of tumor size x : Log of tumor size 1 x : Weight of prostate : Weight of prostate 2 2 x : Patient’s age P ti t’ 3 x : Log of benign prostatic hyperplasia amount x : Log of benign prostatic hyperplasia amount 4 4 x : Dummy variables of whether it is metastasizing to seminal vesicle y g 5 5 x : Log of capsular penetration x : Log of capsular penetration 6 x : Gleason score : Gleason score 7 7 x : Gleason score’s ratio of 4 or 5 x Gl ’ ti f 4 5 8  Sample size : 97 N • p 13 13

Literature example Literature example p 0 5 0.5 0.45 0 4 0.4 V CV 0 35 0.35 GC G 0.3 0 25 0.25 0 2 0.2 Ensembled Bagging gg g MARS NNG MARS-NNG MARS MARS 14 14

Literature example Literature example p • Number of trees • Number of trees Bagging Ensembled MARS-NNG 97 97 9 9 • Structure • Structure Bagging Ensembled MARS-NNG x x x x 2 1 2 2 x x x 4 1 Typical tree Typical tree candidates candidates 15 15