JOINT DETECTION AND SEGMENTATION WITH DEEP HIERARCHICAL NETWORKS - PowerPoint PPT Presentation

JOINT DETECTION AND SEGMENTATION WITH DEEP HIERARCHICAL NETWORKS Zhao Chen Machine Learning Intern, NVIDIA

ABOUT ME • 5th year PhD student in physics @ S tanford by day, deep learning computer vision scientist by night. • Intern with Deep Learning Applied Research (Autonomous Vehicles) @ NVIDIA, Oct-Dec 2016. Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 2

TALK OVERVIEW (1) Problem statement and summary. (2) Dataset and preliminaries. (3) Model motivation. (4) Results and visualizations. Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 3

TALK OVERVIEW (1) Problem statement and summary. (2) Dataset and preliminaries. (3) Model motivation. (4) Results and visualizations. Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 4

FROM SINGLE TO MULTITASK LEARNING Putting deep learning to work in the real world . . . Detection Model Obj ect Bounding Boxes . . . S egmentation Model S egmentation Mask 5

FROM SINGLE TO MULTITASK LEARNING Putting deep learning to work in the real world . . . Detection Model Obj ect Bounding Boxes . . . S egmentation Model S egmentation Mask Poor scalability + inefficient use of information! 6

FROM SINGLE TO MULTITASK LEARNING Putting deep learning to work in the real world How do we use one model to perform multiple tasks faster and better? Obj ect Bounding Boxes . . . S hared Model S egmentation Mask 7

FROM SINGLE TO MULTITASK LEARNING Putting deep learning to work in the real world How do we use one model to perform multiple tasks faster and better? Obj ect Bounding Boxes . . . S hared Model S egmentation Mask + edge detection, + surface normals, + distance estimation… 8

FROM SINGLE TO MULTITASK LEARNING Putting deep learning to work in the real world How do we use one model to perform multiple tasks faster and better? Obj ect Bounding Boxes . . . S hared Model S egmentation Mask How do you relate various tasks to each other in a multi-task neural network? 9

WHAT WE WILL SHOW • By ordering tasks based on receptive field and information density , we improve segmentation and detection accuracy by ~2% and ~8% over single networks, respectively. • The j oint network is robust and easy to tune compared to non-hierarchical baselines. Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 10

TALK OVERVIEW (1) Problem statement and summary. (2) Dataset and preliminaries. (3) Model motivation. (4) Results and visualizations. Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 11

CITYSCAPES DATASET • 2975 Training Images @ resolution 1024 x 2048. • 20 classes for semantic segmentation, including 8 obj ect classes. Of these 8, 4 are much more represented (car, bicycle, person, rider): the “ easy classes.” • Both segmentation, bounding box, and edge ground truth can be generated. S emantic Raw S eg. Image Edge Bounding Detection Box 12

HOW TO TRAIN A SEGMENTATION NETWORK • S tandard FCN (S helhamer 2015) Architecture: Convolutions followed by a deconvolution to retrieve a pixel-dense prediction mask. Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 13

HOW TO TRAIN A DETECTION NETWORK • Network outputs confidence that a pixel lies near the center of an obj ect. • Points of high confidence produce bounding box coordinates. • Confidences are rougher than full segmentation but robust to occlusion. Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 14

TALK OVERVIEW (1) Problem statement and summary. (2) Dataset and preliminaries. (3) Model motivation. (4) Results and visualizations. Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 15

Input (1024 x S hared Feature Map (from base CNN) 2048) Low-Res S eg Obj . Confidence Bbox Coordinate Predictions Positions Positions (W x H x 20) Deconv L = α L seg + (1- α )L det Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 16

OUR BASELINE MODEL PERFORMANCE S eg. Weight = α Det. Weight ( α controls how much attention we pay to segmentation vs detection at training) Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 17

OUR BASELINE MODEL PERFORMANCE S eg. Weight = α Det. Weight ( α controls how much attention we pay to segmentation vs detection at training) Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 18

OUR BASELINE MODEL PERFORMANCE S eg. Weight = α Det. Weight ( α controls how much attention we pay to segmentation vs detection at training) Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 19

OUR BASELINE MODEL PERFORMANCE S eg. Weight = α Det. Weight ( α controls how much attention we pay to segmentation vs detection at training) Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 20

OUR BASELINE MODEL PERFORMANCE S eg. Weight = α Det. Weight ( α controls how much attention we pay to segmentation vs detection at training) Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 21

OUR BASELINE MODEL PERFORMANCE S eg. Weight = α Det. Weight ( α controls how much attention we pay to segmentation vs detection at training) Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 22

OUR BASELINE MODEL PERFORMANCE S eg. Weight = α Det. Weight ( α controls how much attention we pay to segmentation vs detection at training) Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 23

OUR BASELINE MODEL PERFORMANCE S eg. Weight = α Det. Weight ( α controls how much attention we pay to segmentation vs detection at training) Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 24

A LABEL HIERARCHY ALONG TWO AXES Required Receptive Field Obj ect Bounding Boxes Density of Information Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 25

A LABEL HIERARCHY ALONG TWO AXES Required Receptive Field Obj ect Bounding Boxes Obj ect Confidence Density of Information Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 26

A LABEL HIERARCHY ALONG TWO AXES Required Receptive Field Obj ect Bounding Boxes Obj ect Confidence S emantic S egmentation Density of Information Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 27

A LABEL HIERARCHY ALONG TWO AXES Required Receptive Field Obj ect Bounding Boxes Edge Detection (plus) Obj ect Confidence S emantic S egmentation Density of Information Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 28

Input (1024 x S hared Feature Map (from base CNN) 2048) Low-Res S eg Obj . Confidence Bbox Coordinate Predictions Positions Positions (W x H x 20) Deconv Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 29

Input (1024 x S hared Feature Map (from base CNN) 2048) S egmentation Obj . Confidence Obj . BBox Features Features Features Low-Res S eg Obj . Confidence Bbox Coordinate Predictions Positions Positions (W x H x 20) Deconv Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 30

Input (1024 x S hared Feature Map (from base CNN) 2048) S egmentation Obj . Confidence Obj . BBox Features Features Features Low-Res S eg Obj . Confidence Bbox Coordinate Predictions Positions Positions (W x H x 20) Deconv Decreasing information density Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 31

Input (1024 x S hared Feature Map (from base CNN) 2048) Edge S egmentation Obj . Confidence Obj . BBox Features Features Features Features Low-Res Edge Low-Res S eg Obj . Confidence Bbox Coordinate Predictions Predictions Positions Positions (W x H x 3) (W x H x 20) Deconv Deconv Decreasing information density Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 32

Input (1024 x S hared Feature Map (from base CNN) 2048) Edge S egmentation Obj . Confidence Obj . BBox Features Features Features Features Low-Res Edge Low-Res S eg Obj . Confidence Bbox Coordinate Predictions Predictions Positions Positions (W x H x 3) (W x H x 20) Deconv Deconv Decreasing information density Zhao Chen, Joint Det ect ion and S egment at ion with Deep Hierarchical Net works, GTC 2017. 33