Recurrent Pixel Embedding for Grouping Shu Kong CS, ICS, UCI - PowerPoint PPT Presentation

Recurrent Pixel Embedding for Grouping Shu Kong CS, ICS, UCI

Outline 1. Problem Statement -- Pixel Grouping 2. Pixel-Pair Spherical Max-Margin Embedding 3. Recurrent Mean Shift Grouping 4. Experiment 5. Conclusion and Extension Note: the slides were made before paper submission, please treat them as supplemental material and refer to the paper for updated content.

Pixel Labeling Tasks diving into pixels --

Pixel Labeling: Low-Level Vision Tasks diving into pixels -- Low-level vision: edge, boundary, contour

Pixel Labeling: Mid-Level Vision Tasks diving into pixels -- Low-level vision: edge, boundary, contour Mid-level vision: object proposal

Pixel Labeling: High-Level Vision Tasks diving into pixels -- Low-level vision: edge, boundary, contour Mid-level vision: object proposal High-level vision: semantic segmentation instance-level semantic segmentation

Pixel Labeling: Learning Tasks diving into pixels -- Low-level vision: logistic loss edge, boundary, contour Mid-level vision: logistic loss for score object proposal regression for location High-level vision: semantic segmentation logistic loss for mask&score instance-level semantic segmentation cross-entropy for category

Pixel Labeling: New Framework A new framework consisting of two novel modules --

Pixel Labeling: New Framework This framework is A new framework consisting of two novel modules -- agnostic to architecture, so ignore deep learning for now!

Pixel Labeling: New Framework A new framework consisting of two novel modules -- 1. pixel-pair spherical max-margin regression 2. recurrent mean shift grouping

Pixel Labeling: New Framework A new framework consisting of two novel modules -- 1. pixel-pair spherical max-margin regression l learning an embedding space on the hyper-sphere such that • if meeting the pair-wise criterion , learn to push pixels to be close to each other, e.g. both are boundaries, from same instance; • if not, learn to pull them apart. 2. recurrent mean shift grouping

Pixel Labeling: New Framework A new framework consisting of two novel modules -- 1. pixel-pair spherical max-margin regression l learning an embedding space on the hyper-sphere such that • if meeting the pair-wise criterion , learn to push pixels to be close to each other; e.g. both are boundaries, from same instance; • if not, learn to pull them apart. 2. recurrent mean shift grouping iteratively group the pixels into discrete clusters, such as criteria: l boundary vs. non-boundary; object proposals; semantic segments

Pixel-Pair Spherical Max-Margin Regression

Pixel-Pair Spherical Max-Margin Regression date back to Fisher Linear discriminant analysis (LDA)

Pixel-Pair Spherical Max-Margin Regression date back to Fisher Linear discriminant analysis (LDA) To utilize the label information in finding informative projection, maximizing the following objective where

Pixel-Pair Spherical Max-Margin Regression What loss functions can we use at pixel-level?

Pixel-Pair Spherical Max-Margin Regression What loss functions can we use at pixel-level? Principle -- 1. for positive pairs of pixels (meeting the criterion), minimizing the pair-wise discrepancy/distance; 2. for negative pairs, minimizing the similarity.

Pixel-Pair Spherical Max-Margin Regression What loss functions can we use at pixel-level? Principle -- 1. for positive pairs of pixels (meeting the criterion), minimizing the pair-wise discrepancy/distance; 2. for negative pairs, minimizing the similarity. Bert De Brabandere, Davy Neven, Luc Van GoolSemantic Instance Segmentation with a Discriminative Loss Function, arxiv, 2017

Pixel-Pair Spherical Max-Margin Regression What loss functions can we use at pixel-level? Principle -- 1. for positive pairs of pixels (meeting the criterion), minimizing the pair-wise discrepancy/distance; 2. for negative pairs, minimizing the similarity. for example: Euclidean distance between pixel feature vectors for measuring distance. Its inverse, or Gaussian transform, can measure the similarity. ..... Bert De Brabandere, Davy Neven, Luc Van GoolSemantic Instance Segmentation with a Discriminative Loss Function, arxiv, 2017 Alejandro Newell, Jia Deng, Associative Embedding: End-to-End Learning for Joint Detection and Grouping, NIPS, 2017 Alireza Fathi, Zbigniew Wojna, Vivek Rathod, Peng Wang, Hyun Oh Song, Sergio Guadarrama, Kevin P. Murphy, Semantic Instance Segmentation via Deep Metric Learning

Pixel-Pair Spherical Max-Margin Regression We propose the module to learn a hyper-sphere (embedding space), such that positive pairs have high cosine similarity; negative pairs have low cosine similarity.

Pixel-Pair Spherical Max-Margin Regression Why cosine similarity? E. B. Saff and A. B. Kuijlaars. Distributing many points on a sphere. The mathematical intelligencer, 19(1):5–11, 1997. L. Lovisolo ; E.A.B. da Silva, uniform distribution of points on a hyper-sphere with applications to vector bit-plane encoding, IEE Proc. Vision, Image and Signal Processing, 2001

Pixel-Pair Spherical Max-Margin Regression Why cosine similarity? 1. scale-invariant to the length of feature vector; E. B. Saff and A. B. Kuijlaars. Distributing many points on a sphere. The mathematical intelligencer, 19(1):5–11, 1997. L. Lovisolo ; E.A.B. da Silva, uniform distribution of points on a hyper-sphere with applications to vector bit-plane encoding, IEE Proc. Vision, Image and Signal Processing, 2001

Pixel-Pair Spherical Max-Margin Regression Why cosine similarity? 1. scale-invariant to the length of feature vector; 2. easy to analyze how to set hyper-parameters; E. B. Saff and A. B. Kuijlaars. Distributing many points on a sphere. The mathematical intelligencer, 19(1):5–11, 1997. L. Lovisolo ; E.A.B. da Silva, uniform distribution of points on a hyper-sphere with applications to vector bit-plane encoding, IEE Proc. Vision, Image and Signal Processing, 2001

Pixel-Pair Spherical Max-Margin Regression Why cosine similarity? 1. scale-invariant to the length of feature vector; 2. easy to analyze how to set hyper-parameters; E. B. Saff and A. B. Kuijlaars. Distributing many points on a sphere. The mathematical intelligencer, 19(1):5–11, 1997. L. Lovisolo ; E.A.B. da Silva, uniform distribution of points on a hyper-sphere with applications to vector bit-plane encoding, IEE Proc. Vision, Image and Signal Processing, 2001

Pixel-Pair Spherical Max-Margin Regression We use the calibrated cosine similarity as below

Pixel-Pair Spherical Max-Margin Regression We use the calibrated cosine similarity as below loss function contains postive and negative pairs

Pixel-Pair Spherical Max-Margin Regression We use the calibrated cosine similarity as below loss function contains postive and negative pairs alpha is the margin, hyper parameter to be set.

Pixel-Pair Spherical Max-Margin Regression We use the calibrated cosine similarity as below loss function contains postive and negative pairs alpha is the margin, hyper parameter to be set. Gradient is one, didn't penalize hard pixels in sensitive regions, say nearby boundary, segments, etc.

Pixel-Pair Spherical Max-Margin Regression Important theories 1. the loss has a lower bound, minimum; 2. the lower bound does not depend on the dimension of the embedding space.

Pixel-Pair Spherical Max-Margin Regression 2D case

Pixel-Pair Spherical Max-Margin Regression 3D case

Pixel-Pair Spherical Max-Margin Regression https://en.wikipedia.org/wiki/N-sphere

Pixel-Pair Spherical Max-Margin Regression https://en.wikipedia.org/wiki/N-sphere

Pixel-Pair Spherical Max-Margin Regression https://en.wikipedia.org/wiki/N-sphere

Pixel-Pair Spherical Max-Margin Regression One more

Pixel-Pair Spherical Max-Margin Regression Last one -- Combination-aware Weighting

Recurrent Mean Shift Grouping From good embedding space to pixel labeling How to get the instances? How to group the pixels?

Recurrent Mean Shift Grouping From good embedding space to pixel labeling How to get the instances? How to group the pixels? k-means, k-medoids?

Recurrent Mean Shift Grouping From good embedding space to pixel labeling How to get the instances? How to group the pixels? k-means, k-medoids? mean shift

Recurrent Mean Shift Grouping mean shift R.Collins, CSE, PSU, CSE598G Spring 2006

Recurrent Mean Shift Grouping mean shift R.Collins, CSE, PSU, CSE598G Spring 2006

Recurrent Mean Shift Grouping mean shift K. Fukunaga, L. Hostetler, The Estimation of the Gradient of a Density Function, with Applications in Pattern - Recognition, PAMI, 1975

Recurrent Mean Shift Grouping mean shift Other than estimating the PDF directly, estimating the gradient --

Recurrent Mean Shift Grouping mean shift then

Recurrent Mean Shift Grouping mean shift: iteratively updating by shifting the data by such an amount

Recurrent Mean Shift Grouping mean shift: iteratively updating by shifting the data by such an amount

Recurrent Mean Shift Grouping mean shift: iteratively updating by shifting the data by such an amount Gaussian blurring mean-shift (GBMS) algorithm the new iterate is the data average under the posterior probabilities given the current iterate: