Announcements Next week : guest lectures Tuesday : Background - PDF document

9/24/2009 Announcements • Next week : guest lectures – Tuesday : Background modeling Fitting: – Thursday : Image formation Deformable contours Deformable contours • Yong Jae and I are not available for office hours next week. Jaechul is available as usual. Thursday, Sept 24 Kristen Grauman UT-Austin Announcements • Matlab issues: ask us about Matlab coding problems. – e.g., “How do I remove a different pixel from each row? When I try to delete them this way (XYZ), I get a size error…” – (but not: “What does the function imfilter do?”) • Check the functions listed in the psets – help <function name> Some seam carving results f from Pset 1 P t 1 1

9/24/2009 Results from Michael Yao Results from Eunho Yang ∙ Left, “chinese_opera.jpg” (768x600), Original ∙ Top Right, “chinese_opera ‐ dumb ‐ resize.jpg”(400x768), Regular Resize ∙ Bottom Right, “chinese_opera ‐ seamcarving ‐ resize.jpg”(400x768), Content Aware Resize Results from Larry Lindsey Results from Donghyuk Shin Removal of a marked object Results from Donghyuk Shin Results from Michael Fairley Seam carving using HSV (300 by 268) Convention resize(300 by 268) Seam carving (300 by 300) Seam carving (300 by 300) Original image (500 by 368) Original image (500 by 500) This example shows a hue ‐ based skin detector works well to preserve the face in seam carving. However, we can see the body is largely removed, producing a undesirable artifact in the *This example shows a failure case of seam Seam carving using a gradient proportion between face and body. energy (300 by 268) carving in an image with a regular texture Conventional resize (300 by 300) pattern. 2

9/24/2009 Results from Dustin Ho Results from Jay Hennig Conventional resize (399 by 599) Conventional resize O i i Original image (599 by 799) l i ( 99 b 99) Seam carving (399 by 599) Original image Seam carving Last time: Fitting with “voting” Grouping and Fitting • Hough transform for fitting lines, circles y b ( x 1 , y 1 ) y 0 ( x 0 , y 0 ) x 0 x m Goal: move from array of pixel values (or Hough space image space filter outputs) to a collection of regions, objects, and shapes. Edges vs. boundaries Pixels vs. regions By grouping pixels based on Gestalt- inspired attributes, we can map the pixels into a set of image clusters on intensity regions. g Each region is consistent according to the Edges useful signal to features and indicate occluding similarity metric we boundaries, shape. used to do the clustering . …but quite often boundaries of interest Here the raw edge are fragmented, and we have extra output is not so bad… image clusters on color “clutter” edge points. Images from D. Jacobs 3

9/24/2009 Edges vs. boundaries Today Given a model of interest, we can • Fitting an arbitrary shape model with overcome some of the Generalized Hough Transform missing and noisy edges using fitting techniques. • Fitting an arbitrary shape with “active” Fitting an arbitrary shape with active deformable contours With voting methods like the Hough transform , detected points vote on possible model parameters. Previously, we focused on the case where a line or circle was the model… Generalized Hough transform Generalized Hough transform • What if want to detect arbitrary shapes defined by To detect the model shape in a new image: boundary points and a reference point? • For each edge point – Index into table with its gradient orientation θ At each boundary point, compute displacement – Use retrieved r vectors to vote for position of Use retrieved r vectors to vote for position of vector: r = a – p i . t x a reference point For a given model shape: θ θ • Peak in this Hough space is reference point with p 1 p 2 store these vectors in a most supporting edges table indexed by gradient Image space orientation θ . Assuming translation is the only transformation here, i.e., orientation and scale are fixed. [Dana H. Ballard, Generalizing the Hough Transform to Detect Arbitrary Shapes, 1980] Example Example Say we’ve already stored a table of displacement vectors displacement vectors for model points as a function of edge orientation for this model shape. model shape Adapted from Lana Lazebnik 4

9/24/2009 Example Example Now we want to look at some edge points detected in a new image, and vote on the position of that shape. range of voting locations for test point range of voting locations for test point Example Example votes for points with θ = Recall: displacement vectors for model points Example Example votes for points with θ = votes for points with θ = range of voting locations for test point 5

9/24/2009 Application of Generalized Hough for recognition Application of Generalized Hough for recognition • Instead of indexing displacements by gradient • Instead of indexing displacements by gradient orientation, index by “visual codeword” orientation, index by “visual codeword” visual codeword with displacement vectors training image test image B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004 Learning in Computer Vision 2004 Slide credit: L. Lazebnik Slide credit: L. Lazebnik Today Deformable contours a.k.a. active contours, snakes • Fitting an arbitrary shape model with Given : initial contour (model) near desired object Generalized Hough Transform Goal : evolve the contour to fit exact object boundary Main idea : elastic band is • Fitting an arbitrary shape with “active” Fitting an arbitrary shape with active iteratively adjusted so as to iteratively adjusted so as to deformable contours • be near image positions with high gradients, and • satisfy shape “preferences” or contour priors [Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV1987] Figure credit: Yuri Boykov Deformable contours: intuition Deformable contours vs. Hough Like generalized Hough transform, useful for shape fitting; but initial intermediate final Hough Deformable contours Rigid model shape Prior on shape types, but shape iteratively adjusted ( deforms ) Single voting pass can detect multiple instances Requires initialization nearby One optimization “pass” to fit a single contour Image from http://www.healthline.com/blogs/exercise_fitness/uploaded_images/HandBand2-795868.JPG 6

9/24/2009 Why do we want to fit Why do we want to fit deformable shapes? deformable shapes? • Non-rigid, deformable objects can change their shape over h • Some objects have similar basic form but time, e.g. lips, some variety in the contour shape. hands… Figure from Kass et al. 1987 Why do we want to fit Why do we want to fit deformable shapes? deformable shapes? • Non-rigid, deformable objects can change their shape over h time, e.g. lips, hands… • Non-rigid, deformable objects can change their shape over time. Figure credit: Julien Jomier Representation Aspects we need to consider • We’ll consider a discrete representation of the contour, • Representation of the contours consisting of a list of 2d point positions (“vertices”). • Defining the energy functions ν = ( , ), x y ( , ) – External x 0 y i i i 0 – Internal = = − 0 0 , 1 1 , , 1 1 K for for i i n n • Minimizing the energy function ( , ) x 19 y 19 • Extensions: – Tracking • At each iteration, we’ll have the – Interactive segmentation option to move each vertex to another nearby location (“state”). 7

9/24/2009 Fitting deformable contours Energy function How should we adjust the current contour to form the new The total energy (cost) of the current snake is contour at each iteration? defined as: • Define a cost function (“energy” function) that says how = + E E E good a candidate configuration is. total internal external • Seek next configuration that minimizes that cost function. Internal energy: encourage prior shape preferences: Internal energy: encourage prior shape preferences: e.g., smoothness, elasticity, particular known shape. External energy (“image” energy): encourage contour to fit on places where image structures exist, e.g., edges. A good fit between the current deformable contour initial intermediate final and the target shape in the image will yield a low value for this cost function. External energy: intuition External image energy • Measure how well the curve matches the image data How do edges affect “snap” of • “Attract” the curve toward different image features rubber band? – Edges, lines, texture gradient, etc. Think of external energy from image as gravitational pull towards areas of high contrast towards areas of high contrast Magnitude of gradient - (Magnitude of gradient) 2 + 2 ( ) ( ) ( ) G I G I x y − + ( ) 2 ( ) 2 G I G I x y External image energy Internal energy: intuition ( , ) • Gradient images and ( , ) G x x y G y x y • External energy at a point on the curve is: ν = − ν + ν ( ) ( | ( ) | 2 | ( ) | 2 ) E G G What are the underlying And in this one? external x y boundaries in this fragmented • External energy for the whole curve: edge image? − 1 n ∑ = − 2 + 2 | ( , ) | | ( , ) | E G x y G x y external x i i y i i = 0 i 8