A new metric for evaluating semantic segmentation: leveraging - PowerPoint PPT Presentation

Introduction Semantic segmentation Accuracy evaluation Conclusions A new metric for evaluating semantic segmentation: leveraging global and contour accuracy Eduardo Fernandez-Moral 1 , Renato Martins 1 , Denis Wolf 2 , and Patrick Rives 1 1 Lagadic team. INRIA Sophia Antipolis - M´ editerran´ ee. France. 2 University of Sao Paulo - ICMC/USP, Brazil. 24/09/2017 E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Accuracy evaluation Conclusions Table of contents Introduction 1 Semantic segmentation 2 CNN models Training data Accuracy evaluation 3 Comparison of metrics New BJ metric Conclusions 4 E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Accuracy evaluation Conclusions Introduction E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Accuracy evaluation Conclusions Context: semantic-based urban navigation Create a semantical, textured 3D mesh of the environment to help for guidance and automatic navigation of different types of agents Stereopolis-II E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Accuracy evaluation Conclusions Context: semantic-based urban navigation Create semantical, textured 3D meshes of the environment to help for guidance and automatic navigation of different types of agents E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Accuracy evaluation Conclusions Context: semantic-based urban navigation Create semantical, textured 3D meshes of the environment to help for guidance and automatic navigation of different types of agents E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Accuracy evaluation Conclusions Context: semantic-based urban navigation The problem of semantic segmentation consists of associating a class label to each pixel of the given image: Source: A. Geiger et al. , Vision meets Robotics: The KITTI Dataset. IJRR 2013 G. Ros et al. , Vision-based Offline-Online Perception Paradigm for Autonomous Driving. WACV 2015 E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Accuracy evaluation Conclusions Context: semantic-based urban navigation The problem of semantic segmentation consists of associating a class label to each pixel of the given image: Source: M. Cordts et al. , The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR 2013 E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Accuracy evaluation Conclusions Semantic segmentation approaches Traditional approaches : Classification of hand-crafted visual features (e.g. SIFT) taking into account the spatial distribution and the local neighborhood Support Vector Machines (SVM) Random Forest (RF) Conditional Random Fields (CRF) E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Accuracy evaluation Conclusions Semantic segmentation approaches Traditional approaches : Classification of hand-crafted visual features (e.g. SIFT) taking into account the spatial distribution and the local neighborhood Support Vector Machines (SVM) Random Forest (RF) Conditional Random Fields (CRF) Convolutional Neural Networks (CNN) : used for features extraction and classification. Faster and more accurate than traditional methods Encoder-Decoder CNN CNN + CRF CNN cascades E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Accuracy evaluation Conclusions Our work Our work explores the problem of semantic segmentation from accurate RGB-D images We evaluate different network models and input data combinations We analyze different semantic segmentation metrics , with a particular interest on object boundary segmentation E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation CNN models Accuracy evaluation Training data Conclusions Semantic segmentation E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation CNN models Accuracy evaluation Training data Conclusions Encoder-Decoder CNN SegNet 1 : Encoder-decoder architecture 1 Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. “Segnet: A deep convolutional encoder-decoder architecture for scene segmentation”. In: IEEE transactions on pattern analysis and machine intelligence (2017). E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation CNN models Accuracy evaluation Training data Conclusions Encoder-Decoder CNN SegNet 1 : Encoder-decoder architecture SegNet2 : double pipeline SegNet (for color and geometry) 1 Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. “Segnet: A deep convolutional encoder-decoder architecture for scene segmentation”. In: IEEE transactions on pattern analysis and machine intelligence (2017). E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation CNN models Accuracy evaluation Training data Conclusions Encoder-Decoder CNN CEDCNN : compact model focused on real-time E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation CNN models Accuracy evaluation Training data Conclusions Encoder-Decoder CNN CEDCNN : compact model focused on real-time CEDCNN2 : double pipeline CEDCNN (for color and geometry) E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation CNN models Accuracy evaluation Training data Conclusions Training data Trained and tested on public urban datasets: Virtual Kitti 2 Kitti 3 Results verified on: Cityscapes 4 Our own data 2 Adrien Gaidon et al. “Virtual worlds as proxy for multi-object tracking analysis”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2016, pp. 4340–4349. 3 Andreas Geiger et al. “Vision meets robotics: The KITTI dataset”. In: The International Journal of Robotics Research 32.11 (2013), pp. 1231–1237. 4 Marius Cordts et al. “The cityscapes dataset for semantic urban scene understanding”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2016, pp. 3213–3223. E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation CNN models Accuracy evaluation Training data Conclusions Data preprocessing RGB Raw depth Elevation map Surface normals E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Comparison of metrics Accuracy evaluation New BJ metric Conclusions Accuracy evaluation E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Comparison of metrics Accuracy evaluation New BJ metric Conclusions Metrics Traditional metrics Global accuracy (GA) F1-measure Jaccard index (Intersection over union [IoU]) TP = True positives, FP = False positives TP JI = TP + FN + FP TN = True negatives, FN = False negatives 5 Gabriela Csurka et al. “What is a good evaluation measure for semantic segmentation?.” In: BMVC . vol. 27. 2013, p. 2013. E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Comparison of metrics Accuracy evaluation New BJ metric Conclusions Metrics Traditional metrics Global accuracy (GA) F1-measure Jaccard index (Intersection over union [IoU]) TP = True positives, FP = False positives TP JI = TP + FN + FP TN = True negatives, FN = False negatives Boundary metrics Total boundary accuracy (TO) Jaccard index boundary (TJ) Boundary F-measure (BF) 5 BF c = 2 · P c · R c P c = class precision, R c = class recall P c + R c 5 Gabriela Csurka et al. “What is a good evaluation measure for semantic segmentation?.” In: BMVC . vol. 27. 2013, p. 2013. E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Comparison of metrics Accuracy evaluation New BJ metric Conclusions Boundary metrics Simple contour score (TO, TJ) E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation

Introduction Semantic segmentation Comparison of metrics Accuracy evaluation New BJ metric Conclusions Boundary metrics Simple contour score (TO, TJ) Distance among contours (BF) GT prediction contours E. Fernandez-Moral et al . A new metric for evaluating semantic segmentation