Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic - PowerPoint PPT Presentation

Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation Caner Hazırba ş Joint work with Julia Diebold and Daniel Cremers

Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation Caner Hazırba ş min E = min { λ · D + R } Runtime Accuracy Joint work with Julia Diebold and Daniel Cremers

Optimizing the Relevance-Redundancy Tradeoff Train Classification with 
 Validation Feature Ranking selected features Test Feature Set Feature Selection Segmentation min E = min { λ · D + R } x I I y h 3 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Feature Set Haar-Like Texton Color x I I y Location y Depth h 4 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Feature Analysis Train Classification with 
 Validation Feature Ranking selected features Test Feature Set Feature Selection Segmentation min E = min { λ · D + R } x I I y h 5 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Feature Ranking Objective : • maximize the relevance between the feature and its class • minimize the redundancy between the feature pairs max Φ ( Rel, Red ) , Φ = Relevance − Redundancy 1 X = MI ( f i ; class ) − MI ( f i ; f j ) m − 1 i 6 = j p ( x, y ) Z Z MI ( X ; Y ) = log p ( x ) p ( y ) dxdy Ranking : Y X • iteratively rank the features, select one feature at a time • maximize the objective function at each iteration: " # 1 X f m = arg max MI ( f i ; c ) − MI ( f i ; f j ) m − 1 f i ∈ F\F m − 1 f j ∈ F m − 1 6 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Incremental Feature Analysis 100 90 C T H C C H C T T H T H T T C H T T T T T T L L T H T T T H T H H H H H H H H H H H 84.1 80 75.9 L H T T T T T H H H H H H H H H H H H H H Classification Accuracy (%) T T H C T C T C T T T T T T H L T T D H C H T H T T T H C T H T C T H T T C L L H T T H T T H T T T T H H H H H H H H 70 C 66.3 C T L L H T H T H H 65.1 H H C T H H H H T T H 60 H T C H T T T T H H T T T C H C T C H H D H C T 50 H T C D D H T T T T T C L T H L T T C T H T T T T H H H H C H H H H C H H H H H H H D C 44.4 40 T C T C T C C T 30 D Selected Sowerby eTrims Corel NYUv1 NYUv2 20 5 10 15 20 25 30 35 40 45 Size of Feature Set 7 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Relevance of H aar-Like Features 100 HHHHHHHHHHH 90 H T T T T T T L L T H T T T H T C C H C T T H T H T T C T H C 84.1 80 L H T T T HHHHHHHH HH H HH H 75.9 Classification Accuracy (%) T T H C T C T C T T T T T T H L T T HHHHHH HH H T T C L L H T T H T T H T T T T T T 70 H T C T D H C H T H T H C T C T T 66.3 C T L L H T H T T HHHHHHHH 65.1 T T H C T C H T T T T HH T T T H 60 C H C T C HH D H C T 50 H T C T T C L T H L T T C T H T T C H H C HH HH HHHH HH HH H H T D C T T D D T T 44.4 40 T C T C T C C T 30 D Selected Sowerby eTrims Corel NYUv1 NYUv2 20 5 10 15 20 25 30 35 40 45 Size of Feature Set 8 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Relevance of T exton Features 100 H TTTTTT L L T H T 90 TT H T H H H H H H H H H H H TT C C C H C TT H T H T H C 84.1 80 L H TTT TT 75.9 H H H H H H H H Classification Accuracy (%) H H H H H H H C T C T C TTTTTT H L TT H TT C L L H TT H TT H TTTT TT H H H H H H H H 70 T H C T H T C T TT D H C H T H C 66.3 C T L L H T H TTT H T H H H H H H 65.1 H H C C T C H H T C H TTTT H H TTT 60 H C H D H C T 50 H T C TT C L T H L TT C T H TT TT H T TT D C C H H C H H H H H H H H H H H H H D D 44.4 40 TT C T C T C C 30 D Selected Sowerby eTrims Corel NYUv1 NYUv2 20 5 10 15 20 25 30 35 40 45 Size of Feature Set 9 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Relevance of C olor Features 100 90 H T T T T T T L L T H T H H H H H H H H H H H T T H T T T C CC H C T T H T H C T H 84.1 80 75.9 L H T T T H H H H H H H H Classification Accuracy (%) T T H H H H H H H C T C T C T T T T T T H L T T H T T C L L H T T H T T H T T T T T T H H H H H H H H 70 C T H C T H T C T CC T T D H C H T H 66.3 T L L H T H T H H 65.1 H H T H H H H T T H 60 H T C H T T T T H H T T T C H D H C H T CC T C H H T 50 D C C H H C T T C L T H L T T C T H T T T T H H H H H H H H H H H H H D D H T T T 44.4 40 T C T C T CC T 30 D Selected Sowerby eTrims Corel NYUv1 NYUv2 20 5 10 15 20 25 30 35 40 45 Size of Feature Set 10 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Relevance of L ocation Features 100 H T T T T T T LL T H T 90 H H H H H H H H H H H T T H T T T C C C H C T T H T H T H C 84.1 80 L H T T T 75.9 H H H H H H H H Classification Accuracy (%) T T H H H H H H H C T C T C T T T T T T H L T T H T T C LL H T T H T T H T T T T T T H H H H H H H H 70 C T H C T H T C T T T D H C H T H 66.3 C T LL H T H T H H 65.1 H H C T H H H H T T H 60 H T C H T T T T H H T T T C H C T C H H D H C T 50 H T C T T C L T H L T T C T H T T D C T T C H H C H H H H H H H H H H H H H D D H T T T 44.4 40 T C T C T C C T 30 D Selected Sowerby eTrims Corel NYUv1 NYUv2 20 5 10 15 20 25 30 35 40 45 Size of Feature Set 11 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Relevance of D epth Features 100 90 H T T T T T T L L T H T H H H H H H H H H H H T T H T T T C C C H C T T H T H T H C 84.1 80 75.9 L H T T T H H H H H H H H Classification Accuracy (%) T T H H H H H H H C T C T C T T T T T T H L T T H T T C L L H T T H T T H T T T T T T H H H H H H H H 70 C T H C T H T C T T T D H C H T H 66.3 C T L L H T H T H H 65.1 H H C T H H H H T T H 60 H T C H T T T T H H T T T C H C T C H H D H C T 50 H T C D C T T C L T H L T T C T H T T DD T T C H H C H H H H H H H H H H H H H H T T T 44.4 40 T C T C T C C T 30 D Selected Sowerby eTrims Corel NYUv1 NYUv2 20 5 10 15 20 25 30 35 40 45 Size of Feature Set 12 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Feature Selection 100 90 C T H C C H C T T H T H T T C H T T T T T T L L T H T T T H T H H H H H H H H H H H 84.1 80 75.9 L H T T T T T H H H H H H H H H H H H H H Classification Accuracy (%) T T H C T C T C T T T T T T H L T T D H C H T H T T T H C T H T C T H T T C L L H T T H T T H T T T T H H H H H H H H 70 C 66.3 C T L L H T H T H H 65.1 H H C T H H H H T T H 60 H T C H T T T T H H T T T C H C T C H H D H C T 50 H T C D D H T T T T T C L T H L T T C T H T T T T H H H H C H H H H C H H H H H H H D C 44.4 40 � α ( N + 1 − n ) n ∗ = arg max 1 α = 5 T C T C T C C � Acc ( n ) β T β = 2 n ∈ { 1 ,...,N } 30 D Selected Sowerby eTrims Corel NYUv1 NYUv2 20 5 10 15 20 25 30 35 40 45 Size of Feature Set 13 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Number of Selected Features 3 Selected Sowerby eTrims Corel NYUv1 NYUv2 � α ( N + 1 − n ) n ∗ = arg max 1 8 � Acc ( n ) β 2.5 n ∈ { 1 ,...,N } 2 Acc(n) α (N+1-n) 1/ β 1.5 23 1 10 27 0.5 8 0 5 10 15 20 25 30 35 40 45 Size of Feature Set 14 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Semantic Image Segmentation Train Classification with 
 Validation Feature Ranking selected features Test Feature Set Feature Selection Segmentation min E = min { λ · D + R } x I I y h 15 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Classification & Segmentation arg max D Rhino P. Bear Water Snow Vegetation Ground Sky min E = min { λ · D + R } 16 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Improved Runtime Training Time (in seconds) Testing Time (in seconds) eTrims Corel Sowerby NYUv1 NYUv2 eTrims Corel Sowerby NYUv1 NYUv2 Shotton et al. — 1800 1200 — — — 1.10 2.50 — — Fröhlich et al. — — — — — 17.0 — — — — Couprie et al. — — — — 172800 — — — — 0.70 Hermans et al. — — — — — — — — 0.38 0.38 Proposed 143 20 2 133 183 6.6 0.27 0.07 0.32 0.26 On average we improve the runtime by a factor of 7.7 17 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Competitive Results Classification Segmentation eTrims Corel Sowerby NYUv1 NYUv2 eTrims Corel Sowerby NYUv1 NYUv2 Shotton et al. — 68.4 85.6 — — — 74.6 88.6 — — Fröhlich et al. — — — — — 77.2 — — — — Couprie et al. — — — — — — — — — 52.4 Hermans et al. — — — 65.0 — — — — 71.5 54.2 Proposed 77.1 74.4 87.1 65.0 44.0 77.9 78.2 88.8 66.5 45.0 Image Others Class. Segm. GT 18 Caner Hazırba ş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation