Goal Goal: Identify groups of pixels that go together image credit: - PowerPoint PPT Presentation

Lecture 2: Introduction to Segmentation Jonathan Krause Fei-Fei Li, Jonathan Krause Lecture 2 - 1 Goal • Goal: Identify groups of pixels that go together image credit: Steve Seitz, Kristen Grauman Fei-Fei Li, Jonathan Krause Lecture 2 - 2

We’re going to learn a number of techniques that are useful for Types of Segmentation semantic segmentation, but will focus on techniques that are more generally applicable to several types of segmentation problems. For • Semantic Segmentation: Assign labels example, most semantic segmentation methods will learn appearance models for each class, which is not something we’re going to talk about much. Tiger Grass Water Dirt image credit: Steve Seitz, Kristen Grauman Fei-Fei Li, Jonathan Krause Lecture 2 - 3 This is the type of segmentation done by GrabCut, the focus of project Types of Segmentation 1. • Figure-ground segmentation: Foreground/background image credit: Carsten Rother Fei-Fei Li, Jonathan Krause Lecture 2 - 4

Co-segmentation is a relatively recent line of work in segmentation. Types of Segmentation Historically, it started with segmenting the same object instance in multiple images, but eventually grew to segmenting out instances of • Co-segmentation: Segment common object the same category. image credit: Armand Joulin Fei-Fei Li, Jonathan Krause Lecture 2 - 5 Useful in graphics applications, e.g. image matting Application: As a result Rother et al. 2004 Fei-Fei Li, Jonathan Krause Lecture 2 - 6

Superpixels (also see last slide) make computation faster. Region Application: Speed up Recognition proposals speed up detection. Fei-Fei Li, Jonathan Krause Lecture 2 - 7 By getting rid of the background we can remove irrelevant information Application: Better Classification Angelova and Zhu, 2013 Fei-Fei Li, Jonathan Krause Lecture 2 - 8

History: Before Computer Vision Fei-Fei Li, Jonathan Krause Lecture 2 - 9 Gestalt Theory • Gestalt: whole or group – Whole is greater than sum of its parts – Relationships among parts can yield new properties/features • Psychologists identified series of factors that predispose set of elements to be grouped (by human visual system) “I stand at the window and see a house, trees, sky. 
 Theoretically I might say there were 327 brightnesses 
 and nuances of colour. Do I have "327"? No. I have sky, house, and trees.” Max Wertheimer 
 (1880-1943) Fei-Fei Li, Jonathan Krause Lecture 2 - 10

Gestalt Factors Image source: Forsyth & Ponce • These factors make intuitive sense, but are very difficult to translate into algorithms. Fei-Fei Li, Jonathan Krause Lecture 2 - 11 Outline 1. Segmentation as clustering CS 131 review 2. Graph-based segmentation 3. Segmentation as energy minimization new stuff Fei-Fei Li, Jonathan Krause Lecture 2 - 12

Outline 1. Segmentation as clustering 1. K-Means 2. GMMs and EM 3. Mean Shift 2. Graph-based segmentation 3. Segmentation as energy minimization Fei-Fei Li, Jonathan Krause Lecture 2 - 13 Clustering isn’t used as a segmentation approach too much anymore, Segmentation as Clustering but highlights many of the key ideas still used in modern algorithms in • Pixels are points in a high-dimensional space terms of modeling appearance. - color: 3d - color + location: 5d • Cluster pixels into segments Fei-Fei Li, Jonathan Krause Lecture 2 - 14

Clustering: K-Means Algorithm: 1. Randomly initialize the cluster centers, c 1 , ..., c K 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i . Put p into cluster i 3. Given points in each cluster, solve for c i • Set c i to be the mean of points in cluster i 4. If c i have changed, repeat Step 2 • Properties – Will always converge to some solution – Can be a “local minimum” • Does not always find the global minimum of objective function: slide credit: Steve Seitz Fei-Fei Li, Jonathan Krause Lecture 2 - 15 Clustering: K-Means k=2 k=3 slide credit: Kristen Grauman Fei-Fei Li, Jonathan Krause Lecture 2 - 16

Clustering: K-Means Note: Visualize segment with average color Fei-Fei Li, Jonathan Krause Lecture 2 - 17 K-Means Pro: - Extremely simple - Efficient Con: - Hard quantization in clusters - Can’t handle non-spherical clusters Fei-Fei Li, Jonathan Krause Lecture 2 - 18

Gaussian Mixture Model • Represent data distribution as mixture of multivariate Gaussians. How do we actually fit this distribution? Fei-Fei Li, Jonathan Krause Lecture 2 - 19 CS229’s treatment of EM is quite nice. Look it up if you don’t know Expectation Maximization (EM) this! • Goal – Find parameters θ (for GMMs: ) that maximize the likelihood function: • Approach: 1. E-step: given current parameters, compute ownership of each point 2. M-step: given ownership probabilities, update parameters to maximize likelihood function 3. Repeat until convergence See CS229 material if this is unfamiliar! Fei-Fei Li, Jonathan Krause Lecture 2 - 20

Clustering: Expectation Maximization (EM) Fei-Fei Li, Jonathan Krause Lecture 2 - 21 GMMs Pro: - Still fairly simple and efficient - Model more complex distributions Con: - Need to know number of components in advance — hard to know unless you’re looking at the data yourself! Fei-Fei Li, Jonathan Krause Lecture 2 - 22

The original mean shift paper generalizes all of this. Clustering: Mean-shift 1. Initialize random seed, and window W 2. Calculate center of gravity (the “mean”) of W: Can generalize to arbitrary windows/kernels • 3. Shift the search window to the mean 4. Repeat Step 2 until convergence Only parameter: window size slide credit: Steve Seitz Fei-Fei Li, Jonathan Krause Lecture 2 - 23 Mean-Shift Region of interest Center of mass Mean Shift vector Slide by Y . Ukrainitz & B. Sarel Fei-Fei Li, Jonathan Krause Lecture 2 - 24

Mean-Shift Region of interest Center of mass Mean Shift vector Slide by Y . Ukrainitz & B. Sarel Fei-Fei Li, Jonathan Krause Lecture 2 - 25 Mean-Shift Region of interest Center of mass Mean Shift vector Slide by Y . Ukrainitz & B. Sarel Fei-Fei Li, Jonathan Krause Lecture 2 - 26

Mean-Shift Region of interest Center of mass Slide by Y . Ukrainitz & B. Sarel Fei-Fei Li, Jonathan Krause Lecture 2 - 27 Clustering: Mean-shift • Cluster: all data points in the attraction basin of a mode • Attraction basin: the region for which all trajectories lead to the same mode slide credit: Y . Ukrainitz & B. Sarel Fei-Fei Li, Jonathan Krause Lecture 2 - 28

Mean-shift for segmentation • Find features (color, gradients, texture, etc) • Initialize windows at individual pixel locations • Perform mean shift for each window until convergence • Merge windows that end up near the same “peak” or mode Fei-Fei Li, Jonathan Krause Lecture 2 - 29 Mean-shift for segmentation Fei-Fei Li, Jonathan Krause Lecture 2 - 30

Mean Shift Pro: - No number of clusters assumption - Handle unusual distributions - Simple Con: - Choice of window size - Can be somewhat expensive Fei-Fei Li, Jonathan Krause Lecture 2 - 31 Clustering Pro: - Generally simple - Can handle most data distributions with sufficient effort. Con: - Hard to capture global structure - Performance is limited by simplicity Fei-Fei Li, Jonathan Krause Lecture 2 - 32

Outline 1. Segmentation as clustering 2. Graph-based segmentation 1. General Properties 2. Spectral Clustering 3. Min Cuts 4. Normalized Cuts 3. Segmentation as energy minimization Fei-Fei Li, Jonathan Krause Lecture 2 - 33 Images as Graphs q w pq w p – Node (vertex) for every pixel – Edge between pairs of pixels, (p,q) – Affinity weight w pq for each edge • w pq measures similarity • Similarity is inversely proportional to difference 
 (in color and position…) slide credit: Steve Seitz Fei-Fei Li, Jonathan Krause Lecture 2 - 34

Images as Graphs Which edges to include? Fully connected: w - Captures all pairwise similarities - Infeasible for most images Neighboring pixels: - Very fast to compute - Only captures very local interactions Local neighborhood: - Reasonably fast, graph still very sparse - Good tradeoff Fei-Fei Li, Jonathan Krause Lecture 2 - 35 Measuring Affinity • In general: • Examples: - Distance: - Intensity: - Color: - Texture: • Note: Can also modify distance metric slide credit: Forsyth & Ponce Fei-Fei Li, Jonathan Krause Lecture 2 - 36

Measuring Affinity Distance: slide credit: Forsyth & Ponce Fei-Fei Li, Jonathan Krause Lecture 2 - 37 Measuring Affinity Intensity: slide credit: Forsyth & Ponce Fei-Fei Li, Jonathan Krause Lecture 2 - 38

Measuring Affinity Color: slide credit: Forsyth & Ponce Fei-Fei Li, Jonathan Krause Lecture 2 - 39 In practice you’d combine several of these a ffi nity measures in one. Measuring Affinity Texture: slide credit: Forsyth & Ponce Fei-Fei Li, Jonathan Krause Lecture 2 - 40

Segmentation as Graph Cuts w A B C • Break Graph into Segments – Delete links that cross between segments – Easiest to break links that have low similarity (low weight) • Similar pixels should be in the same segments • Dissimilar pixels should be in different segments slide credit: Steve Seitz Fei-Fei Li, Jonathan Krause Lecture 2 - 41 Graph Cut with Eigenvalues • Given: Affinity matrix W • Goal: Extract a single good cluster v - v(i): score for point i for cluster v Fei-Fei Li, Jonathan Krause Lecture 2 - 42