Deep Watershed Transform for Instance Segmentation Min Bai & - PowerPoint PPT Presentation

Deep Watershed Transform for Instance Segmentation Min Bai & Raquel Urtasun To appear at IEEE CVPR 2017 in Hawaii Presented at NVIDIA GTC 2017

Semantic Segmentation ● Input: RGB Image ● Output at each pixel: ○ Semantic label

Instance Segmentation ● Input: RGB Image ● Output at each pixel: ○ Semantic label ○ Instance label ■ Same for each px in object ■ Different among objects ○ Difficulty: How to phrase the problem?

Applications ● Object tracking Image credit: Davi Frossard

Applications ● Interacting with the environment Image credit: http://www.rethinkrobotics.com/build-a-bot/

Applications ● Useful information for other algorithms such as optical flow, etc Image credit: Shenlong Wang

Semantic Segmentation ● Semantic segmentation is a well studied problem ○ Our instance segmentation method leverages an existing technique ○ H. Zhao et al, Pyramid Scene Parsing Network , https://arxiv.org/abs/1612.01105 Image credit: H. Zhao et al.

Watershed Transform ● Classical image segmentation technique Image (left) credit: Adrian Fisher

Scalar Field and Gradient ● Scalar field: single number at each pixel ● Gradient: vector at each pixel, pointing toward direction of greatest ascent Image source: Wikipedia: byVivekj78 - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=15346899

Overview of Approach Input Image Gradient of Energy Landscape Energy Landscape Predicted Instances Semantic Segmentation

Why Predict Direction First? Input Image Direction of Gradient Energy Landscape Much sharper difference in the direction label at the boundary!

Overall Network

Direction Prediction Network Input Image Ground Truth Directions Predicted Directions Semantic Segmentation

Energy Prediction Network Ground Truth Energy Ground Truth Instances Predicted Energy Predicted Instances

Training and Inference ● Pre-train both networks ● End-to-end fine-tuning ● Network trained on NVIDIA DGX-1 ○ Approximately 25 hours total for training on one GP100 core ○ ~0.1s per image for forward pass ○ Thank you NVIDIA for the generous gift! Image source: www.nvidia.com

Cityscapes Dataset ● 2975 training / 500 validation / 1525 testing images ● Instances: car, truck, bus, train, person, rider, motorcycle, bicycle

Cityscapes Instance Segmentation Leaderboard AP* AP* @ 50% AP* @ 50m AP* @ 100m van den Brand et al. 2.3% 3.7% 3.9% 4.9% Cordts et al. 4.6% 12.9% 7.7% 10.3% Uhrig et al. 8.9% 21.1% 15.3% 16.7% Ours 19.4% 35.3% 31.4% 36.8% * Average Precision (AP): higher is better Recently, new approaches have achieved even higher performance.

Sample Output Input RGB Direction Prediction Energy Prediction Semantic Segmentation Predicted Instances Ground Truth Instances

Preliminary TorontoCity Aerial Instance Segmentation Semantic Segmentation (ResNet) Input RGB Predicted Building Instances

Preliminary TorontoCity Aerial Instance Segmentation Weighted AP* Recall* @ Precision* @ Coverage* 50% 50% FCN-8 41.92% 11.37% 21.50% 36.00% ResNet-56 40.65% 12.13% 18.90% 45.36% Ours 56.22% 21.22% 67.16% 63.67% * higher is better

In Summary... ● Simple technique for instance segmentation ● Encodes object instances as energy map ● Predicts gradient direction as intermediate task for better supervision

Deep Watershed Transform for Instance Segmentation Min Bai & - PowerPoint PPT Presentation

Deep Watershed Transform for Instance Segmentation Min Bai & Raquel Urtasun To appear at IEEE CVPR 2017 in Hawaii Presented at NVIDIA GTC 2017 Semantic Segmentation Input: RGB Image Output at each pixel: Semantic label

Semantic Segmentation / Instance Segmentation Based on Deep learning Yiding Liu 2018.12.08

Interactive image segmentation with integrated use of the markers and the hierarchical watershed

Segmentation Bottom-up Segmentation Semantic / instance segmentation Many Slides from L.

Budget-aware Semi-Supervised Semantic and Instance Segmentation Miriam Bellver, Amaia Salvador,

Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds B. Yang, J. Wang,

Video Segmentation for Video Segmentation for Surveillance Surveillance -- A Transform Domain

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Detection and Segmentation CS60010: Deep Learning Abir Das IIT Kharagpur Feb 28, 2020

Image Segmentation Machine Learning Study Group Presented by Yaochen Xie Jan 25, 2018 Outline

An Overview of Semantic Image Segmentation with Deep Learning Simone Bonechi Outline

Other Related Problems Applications Classification Instance Classification Object Detection +

Detection and Segmentation CS60010: Deep Learning Abir Das IIT Kharagpur March 04 and 05, 2020

Lecture 8: Image Segmentation Peng Chao Face++ Researcher pengchao@megvii.com Nov. 2017

Deep learning 8.4. Networks for semantic segmentation Fran cois Fleuret

How Distance Transform Maps Boost Segmentation CNNs: An Empirical Study Jun Ma Department of

Region Merging Driven by Deep Learning for RGB-D Segmentation and Labeling U. Michieli, M.

Segmentation by discrete watersheds Part 1: Watershed cuts Jean Cousty Four-Day Course on

Hang Zhao Massachusetts Institute of Technology Last years challenge at ECCV ADE Dataset New

Segmentation by discrete watersheds Part 2: Algorithms and Seeded watershed cuts Jean Cousty

Localization, and instance segmentation Fang Wan, Yi Zhu, Yanzhao Zhou, Qixian ang Ye Ye

JOINT DETECTION AND SEGMENTATION WITH DEEP HIERARCHICAL NETWORKS Zhao Chen Machine Learning

Seesaw Loss for Long-Tailed Instance Segmentation Jiaqi Wang 1 , Wenwei Zhang 2 , Yuhang Zang 2 ,

Mask R-CNN OBJECT INSTANCE SEGMENTATION AND HUMAN POSE ESTIMATION Kaiming He Georgia Gkioxari

Single-shot Instance Segmentation Chunhua Shen, June 2020 (majority of work done by my students: