Fitting: Deformable contours Thurs Oct 1 Kristen Grauman UT - PDF document

9/30/2015 Fitting: Deformable contours Thurs Oct 1 Kristen Grauman UT Austin Recap so far: Grouping and Fitting Goal: move from array of pixel values (or filter outputs) to a collection of regions, objects, and shapes. Grouping: Pixels vs. regions By grouping pixels based on Gestalt- inspired attributes, we can map the pixels into a set of image clusters on intensity regions. Each region is consistent according to the features and similarity metric we used to do the clustering . image clusters on color 1

9/30/2015 Fitting: Edges vs. boundaries Edges usef ul signal to indicate occluding boundaries, shape. …but quite of ten boundaries of interest Here the raw edge are f ragmented, and we hav e extra output is not so bad… “clutter” edge points. Images from D. Jacobs Fitting: Edges vs. boundaries Given a model of interest, we can overcome some of the missing and noisy edges using fitting techniques. With voting methods like the Hough transform , detected points vote on possible model parameters. Voting with Hough transform • Hough transform for fitting lines, circles, arbitrary shapes b y ( x 1 , y 1 ) y 0 ( x 0 , y 0 ) m x 0 x Hough space image space In all cases, we knew the explicit model to fit. 2

9/30/2015 T oday • Fitting an arbitrary shape w ith “active” deformable contours Deformable contours a.k.a. active contours, snakes Given : initial contour (model) near desired object [Snakes: Active contour models, K ass, Witkin, & Terzopoulos, ICCV1987] Figure credit: Yuri Boykov Deformable contours a.k.a. active contours, snakes Given : initial contour (model) near desired object Goal : evolve the contour to fit exact object boundary Main idea : elastic band is iteratively adjusted so as to • be near image positions with high gradients, and • satisfy shape “preferences” or contour priors [Snakes: Active contour models, K ass, Witkin, & Terzopoulos, ICCV1987] Figure credit: Yuri Boykov 3

9/30/2015 Deformable contours: intuition Im age from http://www.healthlin e.c o m /blo gs /ex erc i s e_f itnes s /up loade d_im ages /HandBand2 -7 95868 .J PG Deformable contours vs. Hough Like generalized Hough transform, useful for shape fitting; but initial intermediate final Hough Deformable contours Rigid model shape Prior on shape types, but shape iteratively adjusted ( deforms ) Single voting pass can detect multiple instances Requires initialization nearby One optimization “pass” to fit a single contour Why do we want to fit deformable shapes? • Some objects have similar basic form but some variety in the contour shape. 4

9/30/2015 Why do we want to fit deformable shapes? • Non-rigid, deformable objects can change their shape over time, e.g. lips, hands… Figure from Kass et al. 1987 Why do we want to fit deformable shapes? • Non-rigid, deformable objects can change their shape over time, e.g. lips, hands… Why do we want to fit deformable shapes? • Non-rigid, deformable objects can change their shape over time. Figure credit: Julien Jomier 5

9/30/2015 Aspects we need to consider • Representation of the contours • Defining the energy functions – External – Internal • Minimizing the energy function • Extensions: – Tracking – Interactive segmentation Representation • We’ll consider a discrete representation of the contour, consisting of a list of 2d point positions (“vertices”).   ( x , y ), ( x 0 y , ) i i i 0   0 , 1 ,  , 1 i n for ( x 19 y , ) 19 • At each iteration, we’ll have the option to move each vertex to another nearby location (“state”). Fitting deformable contours How should we adjust the current contour to form the new contour at each iteration? • Define a cost function (“energy” function) that says how good a candidate configuration is. • Seek next configuration that minimizes that cost function. initial intermediate final 6

9/30/2015 Energy function The total energy (cost) of the current snake is defined as:   E E E total internal external Internal energy: encourage prior shape preferences: e.g., smoothness, elasticity, particular known shape. External energy (“image” energy): encourage contour to fit on places where image structures exist, e.g., edges. A good fit between the current deformable contour and the target shape in the image will yield a low value for this cost function. External energy: intuition • Measure how well the curve matches the image data • “Attract” the curve toward different image features – Edges, lines, texture gradient, etc. External image energy How do edges affect “snap” of rubber band? Think of external energy from image as gravitational pull towards areas of high contrast Magnitude of gradient - (Magnitude of gradient)  2 2 G ( I ) G ( I )   x y   2 2 ( ) ( ) G I G I x y 7

9/30/2015 External image energy • Gradient images and ( , ) ( , ) G x x y G y x y • External energy at a point on the curve is:       2 2 E ( ) ( | G ( ) | | G ( ) | ) external x y • External energy for the whole curve:  n 1     2 2 | ( , ) | | ( , ) | E G x y G x y external x i i y i i  0 i Internal energy: intuition What are the underlying And in this one? boundaries in this fragmented edge image? Internal energy: intuition A priori , we want to favor smooth shapes, contours with low curvature , contours similar to a known shape , etc. to balance what is actually observed (i.e., in the gradient image). 8

9/30/2015 Internal energy For a continuous curve, a common internal energy term is the “bending energy”. At some point v(s) on the curve, this is: 2   2 2 d d      E internal ( ( s )) 2 ds d s Tension, Stiffness, Elasticity Curvature Internal energy • For our discrete representation,     ( , ) 0  1 x y i … n i i i   2 d d                  ( ) ( ) 2 v      1 1 1 1 i 1 i 2 i i i i i i i ds ds Note these are derivatives relative to position- --not spatial • Internal energy for the whole curve: image gradients.  n 1       2        2 E 2    1 1 1 internal i i i i i  i 0 Why do these reflect tension and curvature ? Example: compare curvature       2 E curvature v ( ) 2   i i 1 i i 1       2 2 ( x 2 x x ) ( y 2 y y )     i 1 i i 1 i 1 i i 1 (2,5) (2,2) (3,1) (1,1) (3,1) (1,1) 3 − 2 2 + 1 2 + 1 − 2 5 + 1 2 3 − 2 2 + 1 2 + 1 − 2 2 + 1 2 −2 2 = 4 = −8 2 = 64 = 9

9/30/2015 Penalizing elasticity • Current elastic energy definition uses a discrete estimate of the derivative:  1 n  2       E 1 elastic i i  i 0  n 1        2 2 ( ) ( ) x x y y   i 1 i i 1 i  0 i What is the possible problem with this definition? Penalizing elasticity • Current elastic energy definition uses a discrete estimate of the derivative:  n 1  2       E elastic i 1 i  0 i    2 Instead: n 1         2 2 ( ) ( ) x x y y d   i 1 i i 1 i  0 i where d is the average distance between pairs of points – updated at each iteration. Dealing with missing data • The preferences for low-curvature, smoothness help deal with missing data: Illusory contours found! [Figure from Kass et al. 1987] 10

9/30/2015 Extending the internal energy: capture shape prior • If object is some smooth variation on a known shape, we can use a term that will penalize deviation from that shape:  n 1         ˆ 2 E ( ) internal i i  i 0  ˆ { } where are the points of the i known shape. Fig from Y . Boykov Total energy: function of the weights    E E E total internal external  1 n     2 2 E | G ( x , y ) | | G ( x , y ) | external x i i y i i  i 0    n 1        2        2 E d 2    1 1 1 internal i i i i i  i 0 Total energy: function of the weights  • e.g., weight controls the penalty for internal elasticity large  medium  small  Fig from Y . Boykov 11