� Image segmentation BBM 413 � Fundamentals of � • Goal: identify groups of pixels that go together Image Processing Erkut Erdem � Dept. of Computer Engineering � Hacettepe University � Segmentation – Part 1 Slide credit: S. Seitz, K. Grauman The goals of segmentation The goals of segmentation • Separate image into coherent “objects” • Separate image into coherent “objects” • Group together similar-looking pixels for efficiency of further processing “superpixels” image human segmentation http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/ X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003. Slide credit: S. Lazebnik Slide credit: S. Lazebnik
The goals of segmentation Segmentation • Separate image into coherent “objects” • Compact representation for image data in terms of a set of • Group together similar-looking pixels for efficiency of further components processing “superpixels” • Components share “common” visual properties • Properties can be defined at different level of abstractions R. Achanta et al.. SLIC Superpixels Compared to State-of-the-art Superpixel Methods. TPAMI 2012. Slide credit: Fei-Fei Li What is segmentation? Segmentation is a global process • Clustering image elements that “belong together” – Partitioning • Divide into regions/sequences with coherent internal properties – Grouping • Identify sets of coherent tokens in image What are the occluded numbers? Slide credit: Fei-Fei Li Slide credit: B. Freeman and A. Torralba
Segmentation is a global process Segmentation is a global process … but not too global What are the occluded numbers? Occlusion is an important cue in grouping. Slide credit: B. Freeman and A. Torralba Slide credit: B. Freeman and A. Torralba Groupings by Invisible Completions Magritte, 1957 * Images from Steve Lehar � s Gestalt papers Slide credit: B. Freeman and A. Torralba Slide credit: B. Freeman and A. Torralba
Groupings by Invisible Completions Groupings by Invisible Completions 1970s: R. C. James 2000s: Bev Doolittle Slide credit: B. Freeman and A. Torralba Slide credit: B. Freeman and A. Torralba Gestalt Psychology Perceptual organization � • German: Gestalt - "form" or "whole ” � …the processes by which the bits and • Berlin School, early 20th century pieces of visual information that are – Kurt Koffka, Max Wertheimer, and Wolfgang Köhler � � � available in the retinal image are • Gestalt: whole or group � � � � � � � structured into the larger units of – Whole is greater than sum of its parts perceived objects and their � � � � � � � – Relationships among parts can yield new properties/features interrelations � � � � � � � � � � • Psychologists identified series of factors that predispose set of � � � � � � � m) elements to be grouped (by human visual system) Stephen E. Palmer, Vision Science, 1999 “I stand at the window and see a house, trees, sky. � Theoretically I might say there were 327 brightnesses and nuances of colour. Do I have “327”? No. I have sky, house, and trees.” � Max Wertheimer (1880-1943) orms/forms.htm Slide credit: B. Freeman and A. Torralba Slide credit: J. Hays and Fei-Fei Li � �
Gestalt Psychology Laws of Seeing, Wolfgang Metzger, 1936 (English translation by Lothar Spillmann, MIT Press, 2006) Slide credit: B. Freeman and A. Torralba Similarity Familiarity Slide credit: K. Grauman Slide credit: B. Freeman and A. Torralba http://chicagoist.com/attachments/chicagoist_alicia/GEESE.jpg, http://wwwdelivery.superstock.com/WI/223/1532/PreviewComp/SuperStock_1532R-0831.jpg
Symmetry Common fate Image credit: Arthus-Bertrand (via F. Durand) Slide credit: K. Grauman Slide credit: K. Grauman http://seedmagazine.com/news/2006/10/beauty_is_in_the_processingtim.php Proximity Familiarity Slide credit: K. Grauman http://www.capital.edu/Resources/Images/outside6_035.jpg Slide credit: B. Freeman and A. Torralba
Familiarity Influences of grouping Grouping influences other � perceptual mechanisms such as lightness perception Slide credit: B. Freeman and A. Torralba http://web.mit.edu/persci/people/adelson/publications/gazzan.dir/koffka.html Slide credit: B. Freeman and A. Torralba Emergence Gestalt cues • Good intuition and basic principles for grouping • Basis for many ideas in segmentation and occlusion reasoning • Some (e.g., symmetry) are difficult to implement in practice http://en.wikipedia.org/wiki/Gestalt_psychology Slide credit: S. Lazebnik Slide credit: J. Hays
A simple segmentation technique: Segmentation methods Background Subtraction • Segment foreground from background • If we know what the • Approach: • Histogram-based segmentation background looks like, it is – use a moving average to easy to identify “interesting estimate background image • Segmentation as clustering bits – subtract from current frame – K-means clustering – large absolute values are – Mean-shift segmentation interesting pixels • Graph-theoretic segmentation • trick: use morphological • Applications operations to clean up pixels – Min cut – Person in an office – Normalized cuts – Tracking cars on a road • Interactive segmentation – surveillance � Slide credit: B. Freeman Two different background removal models Background estimate Foreground estimate Foreground estimate Movie frames from which we want to extract the foreground subject Average over frames low thresh high thresh EM background estimate low thresh high thresh EM background estimate Images: Forsyth and Ponce, Computer Vision: A Modern Approach EM Images: Forsyth and Ponce, Computer Vision: A Modern Approach Slide credit: B. Freeman Slide credit: B. Freeman
Image segmentation: toy example Segmentation methods white black • Segment foreground from background pixels 3 pixels pixel count gray • Histogram-based segmentation pixels 2 1 • Segmentation as clustering – K-means clustering – Mean-shift segmentation input image • Graph-theoretic segmentation intensity – Min cut – Normalized cuts • These intensities define the three groups. • Interactive segmentation • We could label every pixel in the image according to which of these primary intensities it is. • i.e., segment the image based on the intensity feature. • What if the image isn’t quite so simple? Slide credit: K. Grauman pixel count pixel count input image input image intensity intensity • Now how to determine the three main intensities that define our groups? pixel count • We need to cluster. input image intensity Slide credit: K. Grauman Slide credit: K. Grauman
Segmentation methods • Segment foreground from background 190 255 0 intensity • Histogram-based segmentation 3 • Segmentation as clustering 2 1 – K-means clustering – Mean-shift segmentation • Graph-theoretic segmentation • Goal: choose three “centers” as the representative – Min cut intensities, and label every pixel according to which of – Normalized cuts these centers it is nearest to. • Interactive segmentation • Best cluster centers are those that minimize SSD between all points and their nearest cluster center c i : Slide credit: K. Grauman Clustering Segmentation as clustering • With this objective, it is a “chicken and egg” problem: • Cluster together (pixels, tokens, etc.) that belong together... – If we knew the cluster centers , we could allocate points • Agglomerative clustering to groups by assigning each to its closest center. – attach closest to cluster it is closest to – repeat • Divisive clustering – split cluster along best boundary – repeat – If we knew the group memberships , we could get the • Dendrograms centers by computing the mean per group. – yield a picture of output as clustering process continues Slide credit: K. Grauman Slide credit: B. Freeman
Greedy Clustering Algorithms Agglomerative clustering Slide credit: D. Hoiem Slide credit: B. Freeman Agglomerative clustering Agglomerative clustering Slide credit: D. Hoiem Slide credit: D. Hoiem
Agglomerative clustering Agglomerative clustering Slide credit: D. Hoiem Slide credit: D. Hoiem Common similarity/distance Dendograms measures • P-norms – City Block (L1) Here x i is the – Euclidean (L2) distance btw. � – L-infinity two points • Mahalanobis – Scaled Euclidean • Cosine distance Dendogram formed by Data set agglomerative clustering using single-link clustering. Slide credit: D. Hoiem Slide credit: B. Freeman
Recommend
More recommend
