Snakes: Snakes: Snakes: Snakes: Active Contours Active Contours - - PowerPoint PPT Presentation

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Snakes: Snakes: Snakes: Snakes: Active Contours Active Contours Active Contours Active Contours

Zoltan Zoltan Kato Kato

http://www.cab.u http://www.cab.u-

• szeged.hu/~kato/variational/

szeged.hu/~kato/variational/

2 2

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Introduction Introduction Introduction Introduction

• Proposed by

Proposed by

• Michael

Michael Kass Kass

• Andrew

Andrew Witkin Witkin

• http://

http://www.ri.cmu.edu/people/witkin_andrew.html www.ri.cmu.edu/people/witkin_andrew.html

• Demetri

Demetri Terzopoulos Terzopoulos

• http://

http://mrl.nyu.edu/~dt mrl.nyu.edu/~dt/ /

Snakes: Active Contour Models Snakes: Active Contour Models. . International Journal of Computer Vision, International Journal of Computer Vision,

• Vol. 1, pp 321
• Vol. 1, pp 321-
• 331, 1988.

331, 1988.

3 3

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

What is a snake? What is a snake? What is a snake? What is a snake?

• An energy minimizing

An energy minimizing spline spline guided by external constraint guided by external constraint forces and pulled by image forces toward features: forces and pulled by image forces toward features:

• Edge detection

Edge detection

• Subjective contours

Subjective contours

• Motion tracking

Motion tracking

• Stereo matching

Stereo matching

• ….

….

Images taken from the GVF website: Images taken from the GVF website: http:// http://iacl.ece.jhu.edu/projects/gvf iacl.ece.jhu.edu/projects/gvf/ /

4 4

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Snake behavior Snake behavior Snake behavior Snake behavior

• A snake falls into the closest

A snake falls into the closest local local energy energy minimum. minimum.

• The local minima of the snake energy comprise

The local minima of the snake energy comprise the set of alternative solutions the set of alternative solutions

• A higher level knowledge is needed to choose

A higher level knowledge is needed to choose the the „correct one” „correct one” from these solutions from these solutions

• High

High-

• level reasoning

level reasoning

• User interaction

User interaction

• These high

These high-

• level methods can

level methods can interact interact with the with the contour model by pushing it toward an contour model by pushing it toward an appropriate local minimum appropriate local minimum

5 5

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Snake behavior Snake behavior Snake behavior Snake behavior

• They rely on other mechanisms

They rely on other mechanisms to place them to place them near near the desired the desired contour. contour.

• The existence of such an

The existence of such an initializer initializer is is application dependent. application dependent.

• Even in the case of manual

Even in the case of manual initialization, snakes are quite initialization, snakes are quite powerful in refining the user’s input. powerful in refining the user’s input.

• Basically, snakes are trying to

Basically, snakes are trying to match a deformable model to an match a deformable model to an image by means of energy image by means of energy minimization. minimization.

Image taken from the GVF website: Image taken from the GVF website: http:// http://iacl.ece.jhu.edu/projects/gvf iacl.ece.jhu.edu/projects/gvf/ /

6 6

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Snake energy Snake energy Snake energy Snake energy

• Parametric representation:

Parametric representation: v(s v(s)=( )=(x(s),y(s x(s),y(s)) ))

• E

Eint

int = internal energy due to bending. Serves to impose

= internal energy due to bending. Serves to impose piecewise smoothness constraint. piecewise smoothness constraint.

• E

Eimage

image = image forces pushing the snake toward image

= image forces pushing the snake toward image features (edges, etc…). features (edges, etc…).

• E

Econ

con = external constraints are responsible for putting

= external constraints are responsible for putting the snake near the desired local minimum. It may come the snake near the desired local minimum. It may come from: from:

• Higher level interpretation

Higher level interpretation

• User interaction, etc…

User interaction, etc…

∫

+ + =

1 int

)) ( ( )) ( ( )) ( ( ds s v E s v E s v E E

con image snake

7 7

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Internal energy Internal energy Internal energy Internal energy

• The snake is a

The snake is a controlled continuity controlled continuity spline spline

• Regularizes the problem

Regularizes the problem

• The first order derivative

The first order derivative v vs

s(s

(s) ) makes the makes the spline spline act like act like a a membrane membrane („elasticity”). („elasticity”).

• The second order derivative

The second order derivative v vss

ss(s

(s) ) makes it act like a makes it act like a thin thin-

• plate

plate („rigidity”). („rigidity”).

• α

α(s (s) ) and and β β(s) (s) controls the relative importance of controls the relative importance of membrane and thin membrane and thin-

• plate terms

plate terms

• Setting

Setting β β(s)=0 (s)=0 for a point allows the snake to become second for a point allows the snake to become second-

• rder discontinuous and develop a corner.
• rder discontinuous and develop a corner.

2 / ) | ) ( | ) ( | ) ( | ) ( (

2 2 int

s v s s v s E

ss s

β α + =

8 8

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Image forces Image forces Image forces Image forces

• Attracts the snake to features (data term)

Attracts the snake to features (data term)

• lines

lines: the simplest functional is the image : the simplest functional is the image intensity: intensity: E Eline

line=

=I(x,y I(x,y) )

• Depending on the sign of

Depending on the sign of w wline

line, the snake will be

, the snake will be attracted to the lightest or darkest attracted to the lightest or darkest nerby nerby contour contour

• edges

edges: one can simply set : one can simply set E Eedge

edge=

= -

• |

|∇ ∇ I(x,y)| I(x,y)|2

2

• attracts the snake to large intensity gradients.

attracts the snake to large intensity gradients.

• terminations

terminations: discussed later : discussed later

term term edge edge line line image

E w E w E w E + + =

9 9

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Snake convergence Snake convergence Snake convergence Snake convergence

• If part of a snake finds a low

If part of a snake finds a low-

• energy

energy feature feature the the spline spline term will pull term will pull neighboring parts toward a possible neighboring parts toward a possible continuation of the feature found. continuation of the feature found.

• In fact, this places a large energy well

In fact, this places a large energy well around a good local minimum around a good local minimum

Video taken from the website: Video taken from the website: http://www http://www-

• 2.cs.cmu.edu/afs/cs/user/aw/www/gallery.html

2.cs.cmu.edu/afs/cs/user/aw/www/gallery.html

10 10

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Scale space Scale space Scale space Scale space

• Minimization by scale

Minimization by scale-

• continuation:

continuation:

1. 1. Spatial Spatial smooting smooting the edge or the edge or line functional line functional

• E

Eedge

edge=

= -

• (G

(Gσ

σ∗∇

∗∇2

2 I)

I)2

2, where

, where G Gσ

σ is

is a Gaussian with a Gaussian with σ σ standard standard deviation deviation

• Minima lie on zero crossings

Minima lie on zero crossings

• f
• f G

Gσ

σ∗∇

∗∇2

2 I

I (~edges) (~edges)

2. 2. Snake comes to equilibrium on Snake comes to equilibrium on a blurry energy a blurry energy 3. 3. Slowly reduce the blurring Slowly reduce the blurring

Image taken from Image taken from M.

• M. Kass

Kass & A. & A. Witkin Witkin & D. & D. Terzopoulos Terzopoulos: Snakes: : Snakes: Active Contour Models. Active Contour Models. International Journal of Computer Vision, International Journal of Computer Vision,

• Vol. 1, pp 321
• Vol. 1, pp 321-
• 331, 1988

331, 1988. .

Zero crossings

11 11

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Termination functional Termination functional Termination functional Termination functional

• Attracts the snake toward termination of line

Attracts the snake toward termination of line segments and corners. segments and corners.

• Let

Let C(x,y C(x,y)= )= G Gσ

σ (x,y)

(x,y)∗ ∗I(x,y)) I(x,y))2

2 (smoothed image)

(smoothed image)

• Let

Let θ θ=tan =tan-

• 1

1(C

(Cy

y/C

/Cx

x)

) the gradient angle the gradient angle

• n=(

n=(cos cos θ θ, sin , sin θ θ) ) unit vector along gradient unit vector along gradient

• n

n⊥

⊥=(

=(-

• sin

sin θ θ, , cos cos θ θ) ) perpendicular to gradient perpendicular to gradient

• E

Eterm

term is defined using curvature of level lines in

is defined using curvature of level lines in C(x,y C(x,y) ): :

( )

2 / 3 2 2 2 2 2 2

2 / /

y x y xx y x xy x yy term

C C C C C C C C C n C n C n E + + − = ∂ ∂ ∂ ∂ = ∂ ∂ =

⊥ ⊥

θ

12 12

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Subjective contour Subjective contour Subjective contour Subjective contour

• Combining

Combining E Eedge

edge and

and E Eterm

term, we can create a snake

, we can create a snake attracted to edges and terminations attracted to edges and terminations

• The shape of the snake between the edges and lines in the

The shape of the snake between the edges and lines in the illusion is completely determined by the illusion is completely determined by the spline spline smoothness smoothness term term

• The same snake can find traditional edges in natural

The same snake can find traditional edges in natural images images

Image taken from Image taken from M.

• M. Kass

Kass & A. & A. Witkin Witkin & D. & D. Terzopoulos Terzopoulos: Snakes: : Snakes: Active Contour Models. Active Contour Models. International International Journal of Computer Vision, Vol. 1, Journal of Computer Vision, Vol. 1, pp 321 pp 321-

• 331, 1988

331, 1988. .

13 13

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Subjective contour: Subjective contour: Subjective contour: Subjective contour: hysteresys hysteresys hysteresys hysteresys

• Snake tracking a

Snake tracking a moving subjective moving subjective contour contour

• The snake bends

The snake bends until the internal until the internal spline spline forces forces

• verpower image
• verpower image

forces forces

• Then the snake

Then the snake falls off the line and falls off the line and returns to a returns to a smoother shape smoother shape

Image taken from Image taken from M.

• M. Kass

Kass & A. & A. Witkin Witkin & D. & D. Terzopoulos Terzopoulos: Snakes: Active Contour Models. : Snakes: Active Contour Models. International International Journal of Computer Vision, Vol. 1, pp 321 Journal of Computer Vision, Vol. 1, pp 321-

• 331, 1988

331, 1988. .

14 14

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Motion tracking Motion tracking Motion tracking Motion tracking

• Once a snake finds a

Once a snake finds a feature, it „locks on” feature, it „locks on”

• If the feature begins

If the feature begins to move, the snake to move, the snake will track the same will track the same local minimum local minimum

• Fast motion could

Fast motion could cause the snake to cause the snake to flip into a different flip into a different minimum minimum

Image taken from Image taken from M.

• M. Kass

Kass & A. & A. Witkin Witkin & D. & D. Terzopoulos Terzopoulos: Snakes: Active Contour Models. : Snakes: Active Contour Models. International International Journal of Computer Vision, Vol. 1, pp 321 Journal of Computer Vision, Vol. 1, pp 321-

• 331, 1988

331, 1988. .

15 15

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Snake energy minimization Snake energy minimization Snake energy minimization Snake energy minimization

• When

When α α(s (s) ) and and β β(s) (s) are are constant constant, we get two , we get two independent Euler independent Euler-

• Lagrange equations.

Lagrange equations.

• When

When α α(s (s) ) and and β β(s) (s) are not constant are not constant then then it is simpler to use a discrete formulation: it is simpler to use a discrete formulation:

= ∂ ∂ + + = ∂ ∂ + + y E y y x E x x

ext ssss ss ext ssss ss

β α β α

4 2 1 1 2 2 1 int 1 int

2 / | 2 | 2 / | | ) ( )) ( ), ( ( ) , ( ), ( ) ( h v v v h v v i E ih y ih x y x v i E i E E

i i i i i i i i i i n i ext snake + − − =

+ − + − = = = + = ∑ β α

16 16

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Snake energy minimization Snake energy minimization Snake energy minimization Snake energy minimization

• Let

Let f fx

x(i

(i) = ) = ∂ ∂ E Eext

ext /

/∂ ∂ x xi

i

where derivatives are where derivatives are approximated by finite approximated by finite differences if they differences if they cannot be computed cannot be computed analitically analitically. .

• The corresponding

The corresponding Euler equations Euler equations

• In matrix form where

In matrix form where A A is a is a pentadiagonal pentadiagonal banded matrix: banded matrix:

)) ( ), ( ( ) 2 ( ) 2 ( 2 ) 2 ( ) ( ) (

2 1 1 1 1 1 2 1 1 1 1

= + + − + + − − + − + − − −

+ + + + − − − − + + −

i f i f v v v v v v v v v v v v v

y x i i i i i i i i i i i i i i i i i i

β β β α α

) , ( ) , ( = + = + y x f Ay y x f Ax

y x

17 17

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Snake energy minimization Snake energy minimization Snake energy minimization Snake energy minimization

• Taking into account the derivatives requires

Taking into account the derivatives requires changing changing A A at each iteration. Speed up: at each iteration. Speed up:

• We assume that

We assume that f fx

x and

and f fy

y are constant during a

are constant during a time step time step explicit Euler method explicit Euler method w.r.t w.r.t. the . the external forces. external forces.

• internal forces are specified by

internal forces are specified by A A we can we can evaluate the time derivative at evaluate the time derivative at t t rather than rather than t t-

• 1

1

) ( ) , ( ) ( ) , (

1 1 1 1 1 1 − − − − − −

− − = + − − = +

t t t t y t t t t t x t

y y y x f Ay x x y x f Ax γ γ

γ is the step size

18 18

Zoltan Zoltan Zoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on Variational Variational Variational and Level Set Methods in Image processing and Level Set Methods in Image processing and Level Set Methods in Image processing

Snake energy minimization Snake energy minimization Snake energy minimization Snake energy minimization

• At equilibrium, the time derivative vanishes.

At equilibrium, the time derivative vanishes.

• The Euler equations can be solved by

The Euler equations can be solved by matrix inversion: matrix inversion:

• The inverse can be calculated by

The inverse can be calculated by LU LU decomposition in decomposition in O(n O(n) ) time. time.

)) , ( ( ) ( )) , ( ( ) (

1 1 1 1 1 1 1 1 − − − − − − − −

− + = − + =

t t y t t t t x t t

y x f y I A y y x f x I A x γ γ γ γ