Image Processing using Partial Differential Equations (PDE) - - PowerPoint PPT Presentation

Image Processing using Partial Differential Equations (PDE)

Restoration, segmentation, tracking, optical flow estimation, registration

Pierre Kornprobst

NeuroMathComp project team INRIA Sophia Antipolis - M´ editerran´ ee Vision Student Talks [ViST]

April 2014

Kornprobst (INRIA) PDEs April 2014 1 / 48

My goals today

Introduce methodology

Kornprobst (INRIA) PDEs April 2014 2 / 48

My goals today

Introduce methodology Show links between formulations (min, ∂.

∂t ,

• )

Kornprobst (INRIA) PDEs April 2014 2 / 48

My goals today

Introduce methodology Show links between formulations (min, ∂.

∂t ,

• )

Show a success story: level-sets

Kornprobst (INRIA) PDEs April 2014 2 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

Kornprobst (INRIA) PDEs April 2014 3 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

2 Three solutions to go further

Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem

Kornprobst (INRIA) PDEs April 2014 3 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

2 Three solutions to go further

Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem

3 A success story: Levels-sets

Kornprobst (INRIA) PDEs April 2014 3 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

2 Three solutions to go further

Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem

3 A success story: Levels-sets 4 Conclusion

Kornprobst (INRIA) PDEs April 2014 3 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

2 Three solutions to go further

Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem

3 A success story: Levels-sets 4 Conclusion

Kornprobst (INRIA) PDEs April 2014 4 / 48

An image is seen as a function defined in continuous space

u : Ω ⊂ I R2 → I R

Kornprobst (INRIA) PDEs April 2014 5 / 48

A PDE defines an evolution

u(t, x) / ∂u ∂t = H(t, x, u, ∇u, ∇2u) v(x) ≡ u(∞, x)

Kornprobst (INRIA) PDEs April 2014 6 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

2 Three solutions to go further

Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem

3 A success story: Levels-sets 4 Conclusion

Kornprobst (INRIA) PDEs April 2014 7 / 48

Gaussian filtering

Let u0 an image, we define : uσ(x) = (Gσ ∗ u0)(x) avec Gσ(x) = 1 2π σ2 exp

• − |x|2

2 σ2

• .

σ = 0 σ = 5 σ = 11 σ = 17

Kornprobst (INRIA) PDEs April 2014 8 / 48

Heat equation

A linear PDE ∂u ∂t (t, x) = ∆u(t, x), t ≥ 0, u(0, x) = u0(x). t = 0 t = 12.5 t = 60.5 t = 93.5 A notion of scale

Kornprobst (INRIA) PDEs April 2014 9 / 48

Solution of the heat equation is a convolution

u(t, x) = (G√

2 t ∗ u0)(x)

Gaussian filtering One operation in a large neighbourhood

σ = 0 σ = 5 σ = 11 σ = 17

t = 0 t = 12.5 t = 60.5 t = 93.5 Heat equation A succession of local operations

Kornprobst (INRIA) PDEs April 2014 10 / 48

Take home messages

PDE appear as a natural way to smooth images. When it is linear, a PDE (or equivalently the convolution) do not preserve edges.

Kornprobst (INRIA) PDEs April 2014 11 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

2 Three solutions to go further

Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem

3 A success story: Levels-sets 4 Conclusion

Kornprobst (INRIA) PDEs April 2014 12 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

2 Three solutions to go further

Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem

3 A success story: Levels-sets 4 Conclusion

Kornprobst (INRIA) PDEs April 2014 13 / 48

Solution 1: Make convolution ”nonlinear”

Bilateral filtering (Tomasi, Manduchi [1998])

Given u0, we define u by : u(x) = 1 w(x)

• d(x − ξ)˜

d(u0(x) − u0(ξ))u0(ξ)dξ with w(x) =

• d(x − ξ)˜

d(u0(x) − u0(ξ))dξ

Kornprobst (INRIA) PDEs April 2014 14 / 48

Denoising and Simplification

before after

Kornprobst (INRIA) PDEs April 2014 15 / 48

Denoising and Simplification

before after

Computer Graphics community like it!

Numerous improvements, extensions, efficient implementations and great applications

Kornprobst (INRIA) PDEs April 2014 15 / 48

Do your own comics

Winnemoller, Olsen, Gooch [2006]

before after

(video-spiderman.avi) Kornprobst (INRIA) PDEs April 2014 16 / 48

Tone management for photographic look

Bae, Paris, Durand [2006]

Kornprobst (INRIA) PDEs April 2014 17 / 48

Tone management for photographic look

Bae, Paris, Durand [2006]

”Clearing winter storm”, Ansel Adams

Kornprobst (INRIA) PDEs April 2014 17 / 48

Tone management for photographic look

Bae, Paris, Durand [2006]

Our classical picture

Kornprobst (INRIA) PDEs April 2014 18 / 48

Tone management for photographic look

Bae, Paris, Durand [2006]

model before

Kornprobst (INRIA) PDEs April 2014 19 / 48

Tone management for photographic look

Bae, Paris, Durand [2006]

model after

Kornprobst (INRIA) PDEs April 2014 20 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

2 Three solutions to go further

Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem

3 A success story: Levels-sets 4 Conclusion

Kornprobst (INRIA) PDEs April 2014 21 / 48

Solution 2: Modify the heat equation

Heat equation ∂u ∂t = ∆u = div( ? ∇u)

Kornprobst (INRIA) PDEs April 2014 22 / 48

Solution 2: Modify the heat equation

Heat equation ∂u ∂t = ∆u = div( ? ∇u) Perona and Malik model [1990] ∂u ∂t = div (c(|∇u|2)

scalar

∇u) with c(s) = 1/√1 + s exp(−s)

Kornprobst (INRIA) PDEs April 2014 22 / 48

Solution 2: Modify the heat equation

Heat equation ∂u ∂t = ∆u = div( ? ∇u) Perona and Malik model [1990] ∂u ∂t = div (c(|∇u|2)

scalar

∇u) with c(s) = 1/√1 + s exp(−s) Buades, Coll et Morel [2005]: Bilateral filter is related to Perona and Malik model

Kornprobst (INRIA) PDEs April 2014 22 / 48

Diffusion acts on isophotes

Kornprobst (INRIA) PDEs April 2014 23 / 48

Diffusion can interpreted w.r.t. local image structures

Nonlinear diffusion is non only a ”controlled” diffusion but it is also related to a directional diffusion depending on local image structures Most diffusion operators can be rewritten as:

• . . .
• uTT +
• . . .
• uNN

Kornprobst (INRIA) PDEs April 2014 24 / 48

How to better take into account local image structure?

Limitation of previous models: Estimation of T, N is local

Kornprobst (INRIA) PDEs April 2014 25 / 48

How to better take into account local image structure?

Limitation of previous models: Estimation of T, N is local Solution: Use structure tensor kρ ∗ ∇uσ∇ut

σ = kρ ∗

uσxx uσxy uσxy uσyy

• Kornprobst (INRIA)

PDEs April 2014 25 / 48

How to better take into account local image structure?

Limitation of previous models: Estimation of T, N is local Solution: Use structure tensor kρ ∗ ∇uσ∇ut

σ = kρ ∗

uσxx uσxy uσxy uσyy

• Weickert [1996]

∂u ∂t = div (D(kρ ∗ ∇uσ∇ut

σ)

matrix

∇u)

Kornprobst (INRIA) PDEs April 2014 25 / 48

Example

Original Chaleur Perona-Malik Weickert

Kornprobst (INRIA) PDEs April 2014 26 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

2 Three solutions to go further

Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem

3 A success story: Levels-sets 4 Conclusion

Kornprobst (INRIA) PDEs April 2014 27 / 48

When a PDE is a gradient descent of an optimisation problem (also called variational problem)

v(x) = Argmin

u(x)

E(u) =

• Ω

F(x, u, ∇u, ∇2u)dx

Kornprobst (INRIA) PDEs April 2014 28 / 48

Solution 3: Define an optimisation problem

Let us start from a model of formation of images, for example : u0 = R u + η where η is a white Gaussian noise and R is a linear operator.

Kornprobst (INRIA) PDEs April 2014 29 / 48

Solution 3: Define an optimisation problem

Let us start from a model of formation of images, for example : u0 = R u + η where η is a white Gaussian noise and R is a linear operator. Solve the least-square problem inf

u

• Ω

|u0 − Ru|2 dx → R∗u0 − R∗Ru = 0 is an ill-posed problem Why? Opertor R∗R is usually hard to invert (not bijective or low eingenvalues)

Kornprobst (INRIA) PDEs April 2014 29 / 48

Constrain possible solutions thanks to a regularity constraint

Tikhonov et Arsenin [1977] inf

u

• Ω

|u0 − Ru|2 dx + λ

• Ω

|∇u|2dx

Kornprobst (INRIA) PDEs April 2014 30 / 48

Constrain possible solutions thanks to a regularity constraint

Tikhonov et Arsenin [1977] inf

u

• Ω

|u0 − Ru|2 dx + λ

• Ω

|∇u|2dx The two main ingredients are here!

Kornprobst (INRIA) PDEs April 2014 30 / 48

Constrain possible solutions thanks to a regularity constraint

Tikhonov et Arsenin [1977] inf

u

• Ω

|u0 − Ru|2 dx + λ

• Ω

|∇u|2dx The two main ingredients are here! To minimise, compute the Euler-Lagrange equation, and in that case, we find again the Laplacian operator. EL(u) = λ△u − (R∗Ru − R∗u0) = 0, ∂u ∂t = EL(u).

Kornprobst (INRIA) PDEs April 2014 30 / 48

So, quadratic penalty gives linear diffusion, and othewise?

Consider a general formulation : inf

u

E(u) =

• Ω

(u0 − Ru)2 dx + λ

• Ω

φ(|∇u|) dx Diffusion operator : div φ′(|∇u|) |∇u| ∇u

• = φ′(|∇u|)

|∇u| uTT + φ′′(|∇u|) uNN

Kornprobst (INRIA) PDEs April 2014 31 / 48

So, quadratic penalty gives linear diffusion, and othewise?

Consider a general formulation : inf

u

E(u) =

• Ω

(u0 − Ru)2 dx + λ

• Ω

φ(|∇u|) dx Diffusion operator : div φ′(|∇u|) |∇u| ∇u

• = φ′(|∇u|)

|∇u| uTT + φ′′(|∇u|) uNN Choosing φ will have consequences on the solution regularity and as a consequence on the functional space to consider to study the variational formulation (not commented today, but lots of interesting maths here) φ(s) = 2

• 1 + s2 − 2

→ u ∈ BV (Ω) (preserve discontinuities)

Kornprobst (INRIA) PDEs April 2014 31 / 48

Take home messages

PDE appear as a natural way to smooth images. When it is linear, a PDE (or equivalently the convolution) do not preserve edges. Nonlinearity is needed to preserve discontinuities (seen in all formulations) PDE may or may not derive from an optimisation problem.

Kornprobst (INRIA) PDEs April 2014 32 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

2 Three solutions to go further

Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem

3 A success story: Levels-sets 4 Conclusion

Kornprobst (INRIA) PDEs April 2014 33 / 48

A curve evolves

Lagrangian formulation ∂c ∂t (t, q) = v(κ, ...) N c(0, q) = c0(q). Example with v = κ

Kornprobst (INRIA) PDEs April 2014 34 / 48

A curve evolves... but what could be the interest?

Kornprobst (INRIA) PDEs April 2014 35 / 48

A curve evolves... but what could be the interest? Example: Segmentation

Kornprobst (INRIA) PDEs April 2014 35 / 48

A curve evolves... but what could be the interest? Example: Segmentation

Kornprobst (INRIA) PDEs April 2014 35 / 48

A curve evolves... but what could be the interest? Example: Segmentation

... How to formalise the problem? By defining optimisation problems!

Kornprobst (INRIA) PDEs April 2014 35 / 48

*** IF TIME *** Active contours, Kass, Witkin etal [1987]

inf

c J(c) = b

• a
• c′(q)
• 2dq + β

b

• a
• c′′(q)
• 2dq
• internal energy

+ λ

b

• a

g2( |∇I(c(q))|) dq

• external energy

,

(g decreasing)

– : J(c) not intrinsic (depends on paramerisation) – : Because of regularity constraint, topology changes are impossible (restricted to a single convex object). – : Numerically, curve has to be initialised close to the object to segment

Kornprobst (INRIA) PDEs April 2014 36 / 48

*** IF TIME *** What about a model with curvature?

inf

c J1(c) = b

• a
• c′(q)
• 2 dq + λ

b

• a

g2( |∇I(c(q))| )dq – : Still not intrinsic. + : No high order terms in the Euler equation. + : It can be shown that curvature also decreases.

Kornprobst (INRIA) PDEs April 2014 37 / 48

*** IF TIME *** Idea!

Caselles, Kimmel, Kichenassamy [1995,..] : inf

c J2(c) = b

• a

g( |∇I(c(q))| )

• c′(q)
• dq.

+ : Model is intrinsic! + : Equivalent to infc J1, Aubert, Blanc-F´ erraud [1999] + : Euler equation is a curve evolution : ∂c ∂t = (κg − ∇g, N) N

Kornprobst (INRIA) PDEs April 2014 38 / 48

Ok, but numerically, evolving a curve is not trivial

Distribution of points won’t stay homogeneous Stability problems Topology changes...

Kornprobst (INRIA) PDEs April 2014 39 / 48

Idea: The Level-Sets method

Dervieux, Thomasset [1980], Osher, Sethian [1988]

A curve seen as an isophote of a function Evolve curve is equivalent to evolve function ∂c ∂t = v N, c(0, q) = c0(q). = ⇒ ∂u ∂t = v |∇u| u(0, x) = u0(x).

Kornprobst (INRIA) PDEs April 2014 40 / 48

Many advantages

Fixed system of coordinates Easy handling of topology changes Applicable in any dimension (think about surfaces in 3D!) Equation can be solved with suitable numerical schemes coming from hyperbolic equations.

Kornprobst (INRIA) PDEs April 2014 41 / 48

Exemple: Segmentation based on objects contours

Caselles, Kimmel, etal [1997], etc

∂u ∂t = g(|∇I|) |∇u| div ∇u |∇u|

• + αg(|∇I|) |∇u| + ∇g, ∇u

Kornprobst (INRIA) PDEs April 2014 42 / 48

Exemple: Segmentation based on regions

Chan, Vese [1999]

inf

i1,i2,c F(i1, i2, c) = µ|c| +

• inside(c)

|u0 − i1|2dx +

• utside(c)

|u0 − i2|2dx

Kornprobst (INRIA) PDEs April 2014 43 / 48

These are just two examples...

Many other applications have been considered (e..g, textures, 3D

• bjects in medical images, tracking of moving objects).

New methods to improve speed (e.g., fast marching). Use shape priors. Keep function as a distance function across iterations to avoid periodic re-initialisation steps. etc.

Kornprobst (INRIA) PDEs April 2014 44 / 48

1 Can we use PDEs to do some interesting image processing?

Definitions From Gaussian filtering to the heat equation

2 Three solutions to go further

Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem

3 A success story: Levels-sets 4 Conclusion

Kornprobst (INRIA) PDEs April 2014 45 / 48

Take home messages

PDE appear as a natural way to smooth images. When it is linear, a PDE (or equivalently the convolution) do not preserve edges. Nonlinearity is needed to preserve discontinuities (seen in all formulations) PDE may or may not derive from an optimisation problem. The notion of time evolution can be related to a notion of scale (in image restoration) but also to different aspects like a motion (in level-sets) (not shown here) Many theoretical results allow to prove if your problem is well defined or not. Giving formulations in a continuous setting offers high intuitions and discretisation aspects only come when simulations are needed. There has been a high activity in this area in [1990–2010] with lots of papers!

Kornprobst (INRIA) PDEs April 2014 46 / 48

If you want to learn more

1999 2001 2002 2002, 2006, 2014 Sethian Sapiro Osher, Fedwik Aubert, Kornprobst

Kornprobst (INRIA) PDEs April 2014 47 / 48

Thank you!

http://www-sop.inria.fr/members/Pierre.Kornprobst

Kornprobst (INRIA) PDEs April 2014 48 / 48