Image Segmentation Machine Learning Study Group Presented by - PowerPoint PPT Presentation

Image Segmentation Machine Learning Study Group Presented by Yaochen Xie Jan 25, 2018

Outline • Overview • Three Levels of Segmentation • Basic Segmentation • Semantic Segmentation (FCN, DeepLab) • Instance Segmentation (Mask-RCNN)

Image Segmentation • Process of partitioning a digital image into multiple segments. • Typically used to locate objects and boundaries. • More precisely, image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain visual characteristics. http://imagej.net/Segmentation

Image Segmentation • Definition : Image segmentation partitions an image into regions called segments • Image segmentation creates segments of connected pixels by analyzing some similarity criteria: Intensity, color, texture, histogram, features…

Gestalt Theory • Gestalt: whole or group • The whole is greater than the sum of its parts • Relationships between parts can yield new properties or features • Psychologists identified series of factors that predispose set of elements to be grouped (by human visual system) • Max Wertheimer (1880-1943) ： “I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnesses and nuances of color. Do I have ”327”? No. I have sky, house, and trees.”

Image Segmentation • Applications

Segmentation at Three Levels ❖ Basic Segmentation (Mostly unsupervised learning) ❖ Semantic Segmentation ❖ Instance Segmentation • Automatically & Interactively • Many approaches: • Edge-based • Graph-based • Contour-based • Energy-based • ……

Segmentation at Three Levels • Segmentation as a cluster ❖ Basic Segmentation ❖ Semantic Segmentation 5 dimensions: 3 color channels & 2 space location ❖ Instance Segmentation

Segmentation at Three Levels • Intelligent Scissors [Mortensen 95] ❖ Basic Segmentation ❖ Semantic Segmentation Approach answers a basic question - ❖ Instance Segmentation Q: how to find a path from seed to mouse that follows object boundary as closely as possible?

Segmentation at Three Levels • Intelligent Scissors [Mortensen 95] ❖ Basic Segmentation ❖ Semantic Segmentation Basic Idea ❖ Instance Segmentation • Define edge score for each pixel -edge pixels have low cost • Find lowest cost path from seed to mouse Q: How to define cose? How to find the path? (Dijkstra)

Segmentation at Three Levels • Intelligent Scissors [Mortensen 95] ❖ Basic Segmentation ❖ Semantic Segmentation How does really work? ❖ Instance Segmentation Treat the image as a graph: Graph •node for every pixel p 
 link between every adjacent •pair of pixels p, q •cost c for each link • The link should follow the intensity edge: • want intensity to change rapidly cross the link • c related to di ff erence of intensity cross the link

Segmentation at Three Levels • Graph based segmentation ❖ Basic Segmentation ❖ Semantic Segmentation Felzenszwalb P F , Huttenlocher D P . E ffi cient Graph-Based Image Segmentation[J]. International Journal of Computer Vision, 2004, 59(2):167-181. ❖ Instance Segmentation Basic Idea ： Consider image as a graph, pixels as nodes and similarities as weights of edges. Cut edges with less weight in each iteration. Source: Forsyth & Ponce

Segmentation at Three Levels • Graph based segmentation ❖ Basic Segmentation ❖ Semantic Segmentation Break graph into segments ❖ Instance Segmentation To delete links that cross between segments (with low weight) • Similar pixels should be in the same segment • Dissimilar pixels should be in di ff erent segments Source: Forsyth & Ponce

Segmentation at Three Levels • Graph based segmentation ❖ Basic Segmentation ❖ Semantic Segmentation Measuring A ffi nity (weight): ❖ Instance Segmentation Source: Forsyth & Ponce

Segmentation at Three Levels • Graph based segmentation ❖ Basic Segmentation ❖ Semantic Segmentation Measuring A ffi nity (weight): ❖ Instance Segmentation Source: Forsyth & Ponce

Segmentation at Three Levels • Graph based segmentation ❖ Basic Segmentation ❖ Semantic Segmentation Measuring A ffi nity (weight): ❖ Instance Segmentation Source: Forsyth & Ponce

Segmentation at Three Levels • Graph based segmentation ❖ Basic Segmentation ❖ Semantic Segmentation Measuring A ffi nity (weight): ❖ Instance Segmentation Source: Forsyth & Ponce

Segmentation at Three Levels • Graph based segmentation ❖ Basic Segmentation ❖ Semantic Segmentation Steps: ❖ Instance Segmentation 1.Compute all the weights of edges. Put each node into separate components. 2.Sort edges by their weights. 3.In each iteration, for edge with the least weight: If its 2 ends belongs to di ff erent components and its weight is less than the threshold of both components, merge two ends. Set new threshold as min{weights}.

Segmentation at Three Levels • Fully Convolutional Networks [CVPR 2015] ❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

Segmentation at Three Levels • Convolutionalization - why fully convolution? ❖ Basic Segmentation ❖ Semantic Segmentation Transforming fully connected layers into convolution layers enables a classification net to output a heat-map ❖ Instance Segmentation Fully Convolutional Networks [CVPR 2015]

Segmentation at Three Levels • Upsampling (Deconvolution) ❖ Basic Segmentation ❖ Semantic Segmentation Fully Convolutional Networks [CVPR 2015] ❖ Instance Segmentation Deconvolution Convolution

Segmentation at Three Levels • Skip Architecture - combining WHAT and WHERE ❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation Fully Convolutional Networks [CVPR 2015]

Segmentation at Three Levels • DeepLab [ICLR 2015] ❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

Segmentation at Three Levels • Atrous convolutions ❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation Atrous convolution

Segmentation at Three Levels • Atrous convolutions ❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation 500 x 500 16 x 16 Fully Convolutional Network DeepLab 514 x 514 67 x 67

Segmentation at Three Levels • Conditional Random Field (CRF) ❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation From output feature map From raw image

Segmentation at Three Levels • Conditional Random Field (CRF) ❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation

Segmentation at Three Levels • Instance Segmentation ❖ Basic Segmentation ❖ Semantic Segmentation ❖ Instance Segmentation