Lecture 21: Grouping 1
Announcements • Final project guidelines and sign-up sheet out on Piazza Writeup (70%) • Due 4/26 midnight • Grading rubric provided Presentation (20%) • Sign-up sheet for 4/27 - 4/28. • If no times work, send private Piazza post ASAP . • 5 mins + 2 mins of questions. • Recommend splitting time between teammates
Announcements • Final project guidelines and sign-up sheet out Writeup (70%) • Should have intro, related work, method, experiments, conclusion. • At most 6 pages in CVPR format • Evaluated on: • Background: how well do you explain the related work? • Completeness: results quality, what worked or didn’t work, thoroughness of evaluation • Format and clarity
Today • Image segmentation • Edge-aware image processing 4
<latexit sha1_base64="b4HLEbhr7TEtaehBb4ygFYyuiV8=">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm/GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipGQ7KFbfqLkDWiZeTCuRoDMpf/WHM0gilYJq3fPcxPgZVYzgbNSP9WYUDahI+xZKmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUWZsNiUbgrf68jpX1U9t+o1ryv12zyOIpzBOVyCBzWowz0oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8AyvGM6g=</latexit> <latexit sha1_base64="b4HLEbhr7TEtaehBb4ygFYyuiV8=">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm/GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipGQ7KFbfqLkDWiZeTCuRoDMpf/WHM0gilYJq3fPcxPgZVYzgbNSP9WYUDahI+xZKmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUWZsNiUbgrf68jpX1U9t+o1ryv12zyOIpzBOVyCBzWowz0oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8AyvGM6g=</latexit> <latexit sha1_base64="b4HLEbhr7TEtaehBb4ygFYyuiV8=">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm/GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipGQ7KFbfqLkDWiZeTCuRoDMpf/WHM0gilYJq3fPcxPgZVYzgbNSP9WYUDahI+xZKmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUWZsNiUbgrf68jpX1U9t+o1ryv12zyOIpzBOVyCBzWowz0oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8AyvGM6g=</latexit> <latexit sha1_base64="b4HLEbhr7TEtaehBb4ygFYyuiV8=">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm/GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipGQ7KFbfqLkDWiZeTCuRoDMpf/WHM0gilYJq3fPcxPgZVYzgbNSP9WYUDahI+xZKmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUWZsNiUbgrf68jpX1U9t+o1ryv12zyOIpzBOVyCBzWowz0oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8AyvGM6g=</latexit> Recall: semantic segmentation problem f (Colors represent categories) 5 Source: Torralba, Freeman, Isola
Bottom-up segmentation • Group together similar-looking pixels • “Bottom-up” process • Unsupervised 6 Source: A. Oliva
A “simple” segmentation problem 7 Source: Torralba and Freeman
Segmentation is a global process What are the occluded numbers? 8 Source: A. Oliva
Segmentation is a global process What are the occluded numbers? Occlusion is an important cue in grouping. 9 Source: A. Oliva
Groupings by Invisible Completions 10 * Images from Steve Lehar’s Gestalt papers Source: A. Oliva
… but not too global 11 Source: A. Oliva
12 Photo credit: R. C. James
Perceptual organization “…the processes by which the bits and pieces of visual information that are available in the retinal image are structured into the larger units of perceived objects and their interrelations” Stephen E. Palmer, Vision Science, 1999 13 Source: A. Oliva
Gestalt principles There are hundreds of different grouping laws 14
15 Source: A. Oliva
Familiar configuration 16 Source: A. Oliva
Clustering 17 Source: K. Grauman
A simple segmentation algorithm • Each pixel is described by a vector z = [r, g, b] or [Y u v], … • Run a clustering algorithm (e.g. k-means) using some distance between pixels: D(pixel i , pixel j ) = || z i – z j || 2 18 Source: A. Oliva
Method 1: K-Means Idea: find K centroids that cover the points 19 Example from: https://en.wikipedia.org/wiki/K-means_clustering
<latexit sha1_base64="stxZofKDEFZTyhgykY20wKv2T5Q=">ACknicbVHLbhMxFPUMrxIeDY8dG4sIiQVEMwUEQgIK3bBgUVTSVorTyOPxJFb8GNl3ECMzH8TvsONvuJMGVFquZPn4nHPte6+LWqsAWfYrS9dvnL12tb1wY2bt25vD+/cPQyu8UJOhNPOHxc8SK2snIACLY9rL7kptDwqVnu9fvRV+qCc/QJtLWeGL6yqlOCA1Hz4I7JShVrzNkCrJY2sSX6JSA0HJZFReNBR7vIXC09B+ctN+jfsGeGVR6lhoDJ1H9SbvTuLqz/Fv+reOMmXpATrQq2UF7PtZ8WlkhdNlaA1u/bMN3tmbmVeLJZpP4g4Sw1E2ztZBL4J8A0ZkE/vz4U9WOtEYaUFoHsI0z2qYeGghJbdgDVB1lys+EJOEfZthVlcj7Sj5ApaeU8Lgt0zZ7NiNyEvmB09p2E81pP/k+bNlC9mkVl6wakFacPVY2m4Gj/P7RUXgrQLQIuvMJaqVhyzwXgrwxwCPn5li+Cw51x/mycfX4+2v2wGcWeUAeksckJy/JLvlI9smEiGQ7eZG8Td6l9PX6ft079SaJpuce+SfSD/9Bn+YzKo=</latexit> K-Means • Points are assigned to nearest centroid • Find centroids µ i that minimize distance to points assigned to it Minimize: k k x � µ i k 2 X X arg min S i =1 x ∈ S i 20 Example from: https://en.wikipedia.org/wiki/K-means_clustering
K-Means 1. Initialize centroids randomly 21 Example from: https://en.wikipedia.org/wiki/K-means_clustering
K-Means 2. Assign points to nearest cluster centroid 22 Example adapted from: https://en.wikipedia.org/wiki/K-means_clustering
K-Means 3. Update centroids 23 Example adapted from: https://en.wikipedia.org/wiki/K-means_clustering
K-Means Repeat until change is very small 24 Example adapted from: https://en.wikipedia.org/wiki/K-means_clustering
K-Means Clustering • Given k, the k-means algorithm consists of four steps: – Select initial centroids at random. – Assign each point to the cluster with the nearest centroid. – Compute each centroid as the mean of the points assigned to it. – Repeat previous 2 steps until no change. 25 Source: A. Oliva
• K-means (k=5) clustering based on intensity (middle) or color (right) is essentially vector quantization of the image attributes – Clusters don’t have to be spatially coherent Image Intensity-based clusters Color-based clusters 26 each pixel is replaced with the mean value of its cluster Source: A. Oliva
K-means using color alone (k=11 clusters) Showing 4 of the segments, (not necessarily connected) Some are good, some 27 meaningless Source: A. Oliva
Including spatial relationships Augment data to be clustered with spatial coordinates. Y ⎛ ⎞ ⎜ ⎟ u color coordinates ⎜ ⎟ (or r,g,b) z ⎜ ⎟ = v ⎜ ⎟ x ⎜ ⎟ spatial coordinates ⎜ ⎟ y ⎝ ⎠ 28 Source: A. Oliva
• Clustering based on (r,g,b,x,y) values enforces more spatial coherence K-means using color and position, 20 segments Still misses goal of perceptually pleasing or useful segmentation No measure of texture 29 Hard to pick K… Source: A. Oliva
K-Means for segmentation • Pros – Very simple method – Converges to a local minimum of the error function • Cons – Memory-intensive – Need to pick K – Sensitive to initialization – Sensitive to outliers – Parametric 30 Source: S. Lazebnik
Method 2: Mean shift clustering • A versatile technique for clustering-based segmentation http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html 31 Source: A. Oliva D. Comaniciu and P . Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, PAMI 2002.
Mean shift mode-finding algorithm The mean shift algorithm seeks modes or local maxima of density in the feature space Feature space image (L*u*v* color values) 32
Search window Center of mass Mean Shift vector 33 Slide by Y. Ukrainitz & B. Sarel
Search window Center of mass Mean Shift vector 34 Slide by Y. Ukrainitz & B. Sarel
Search window Center of mass Mean Shift vector 35 Slide by Y. Ukrainitz & B. Sarel
Search window Center of mass Mean Shift vector 36 Slide by Y. Ukrainitz & B. Sarel
Search window Center of mass Mean Shift vector 37 Slide by Y. Ukrainitz & B. Sarel
Search window Center of mass Mean Shift vector 38 Slide by Y. Ukrainitz & B. Sarel
Search window Center of mass 39 Slide by Y. Ukrainitz & B. Sarel
Mean shift clustering • Cluster: all data points in the attraction basin of a mode 40 Slide by Y. Ukrainitz & B. Sarel
Mean Shift Segmentation 1. Convert the image into tokens (via color, gradients, texture measures etc). 2. Choose initial search window locations uniformly in the data. 3. Compute the mean shift window location for each initial position. 4. Merge windows that end up on the same “peak” or mode. 5. The data these merged windows traversed are clustered together. Pixels in L*u* space Clustering results after ~160 mean shift procedures 41 Corresponding trajectories with peaks marked as red dots Source: A. Oliva Szeliski – Chapter 5 – Segmentation, 5.3 Mean shift
Mean Shift Segmentation Results 42 Source: A. Oliva
Recommend
More recommend
