Image Processing using Variational Data Assimilation Methods - PowerPoint PPT Presentation

Image Processing using Variational Data Assimilation Methods Dominique B´ er´ eziat, Isabelle Herlin May 6, 2008

Introduction • Use a new methodology ... ◮ to solve ill-posed problems of Image Processing, ◮ to take into account temporal information, ◮ to efficiently manage missing data. • Well-posed problem [Hadamard, 1923]: 1 uniqueness of solution, 2 the solution depends continuously on the input data. • In the Image Processing field: ◮ low level: gradient estimation, contours detection. ◮ high level: segmentation, restoration, matching, motion estimation, shape from shading, ...

Solution • Constraining the solution’s space to obtain a well-posed [Tikhonov, 1963] problem. • Minimization of a cost function [Mumford et al , 1989]:   �  � F ( f , g ) � 2 + α �∇ f � 2  d x   Argmin f ∈ L 2 (Ω) Ω � �� � � �� � link to observation regularization R n Ω bounded domain in • Euler-Lagrange equations.

Temporal sequence • The previous method is easily extended to the temporal dimension:   � T �  � F ( f , g ) � 2 α Ψ( ∇ 3 f )  d x dt   Argmin + (1) f ∈ L 2 (Ω) Ω 0 � �� � � �� � R 3 same model regularization in � T � ∂ x , ∂ ∂ ∂ y , ∂ with ∇ 3 = ∂ t • Missing data are not taken into account: ◮ aberrant values are smoothed; ◮ aberrant values on large areas (common situation in remote sensing): smoothing is not sufficient. • No realistic model of image structures’ evolution. Only linear and regular dynamics are correctly handled.

Data Assimilation • Used in inverse modeling and simulation in the environmental context. • U ( x , t ) V ( x , t ) x ∈ Ω t ∈ [ 0 , T ] • System to be solved w.r.t. U : ∂ U  ∂ t ( x , t ) + M ( U )( x , t ) = ε m ( x , t )   U ( x , 0 ) = U 0 ( x ) + ε b ( x )   H ( U , V )( x , t ) = ε o ( x , t ) • The evolution equation provides the description of the temporal dynamics. • ε m , ε b , ε o quantify the model, initial condition and observation errors.

Variational method • Minimize the following cost function: ���� M ( U )) T Q − 1 ( U t + ( U t + M ( U )) d x dtd x ′ dt ′ + Argmin U � �� � � �� � x , t x ′ , t ′ �� ( U ( ., 0 ) − U 0 ) T B − 1 ( U ( ., 0 ) − U 0 ) d x d x ′ + � �� � � �� � x x ′ ���� H ( U , V ) T R − 1 H ( U , V ) d x dtd x ′ dt ′ � �� � � �� � x , t x ′ , t ′ • Q ( x , t , x ′ , t ′ ) , B ( x , x ′ ) , R ( x , t , x ′ , t ′ ) covariance matrices of ε m , ε b , ε o . • Euler-Lagrange equations.

Assimilation of images • How to apply the Data Assimilation framework to solve ill-posed Image Processing problems? F ( f , g ) = 0 link to observation �∇ f � bounded regularization • Our strategy: H and g is the F is the observation operator 1 The operator observation. 2 Regularization must be performed by the evolution equation. • Important points: 1 To choose a suitable evolution equation: strongly dependent on the problem. 2 To choose the matrices of covariance Q and R .

Covariance matrix Q • Inverse of covariance matrix: � C ( x , x ′′ ) C − 1 ( x ′′ , x ′ ) d x ′′ = δ ( x − x ′ ) • The Dirac covariance: C ( x , x ′ ) = δ ( x − x ′ ) . ◮ Inverse: C − 1 ( x , x ′ ) = δ ( x − x ′ ) � � U T ( x ) C − 1 U ( x ′ ) d x = � U � 2 d x ◮ → zero-order regularization. • The exponential covariance: C ( x , x ′ ) = exp ( | x − x ′ | ) . σ ◮ Inverse: C − 1 ( x , x ′ ) = 1 2 σ δ ( x − x ′ ) − σ 2 δ ′′ ( x − x ′ ) � � 1 � � 2 σ � U � 2 + σ U T ( x ) C − 1 U ( x ′ ) d x = 2 �∇ U � 2 d x ◮ → first-order regularization.

Covariance matrix R • R is weighting the contribution of the observation. • Missing data: the acquisition is partially missing or noisy. • Defining R is the natural way to manage missing data. ◮ If data is missing at time t 1 on a given region: the related covariance R has to be given to 0. ◮ Consequently, the observation equation does not act. ◮ The computation of the state vector is then only obtain by the evolution equation. • R has to be invertible (in convolution sense).

Application to optical flow • w = apparent motion (optical flow), ∂ x ∂ t = w . • Observation equation: the transport of image brightness. dI dt ( x , t ) = 0 ∇ I T w + ∂ I = 0 ∂ t • Standard method [Horn et al , 1981]: � � ( I T w + ∂ I � ∂ t ) 2 + α �∇ w � 2 d x Argmin w

Application to optical flow • Evolution equation: transport of velocity. dw dt ( x , t ) = 0 ∇ w T w + w t = 0 • Formulation in the Data Assimilation framework: ∂ w ∂ t + ∇ w T w = ε m ∂ I −∇ I T w + ε o = ∂ t w ( x , 0 ) w 0 ( x ) + ε b ( x ) = initial condition • Gradient values are assimilated into a model of velocity transport.

Matrices of covariance • Q : exponential covariance insuring a first order regularization of the velocity. • R : managing missing data using a function f characterizing the quality of the observation: ◮ Observation equation is verified even if the image spatio-temporal gradient are close to � 0. ◮ In this case, equation observation is then degenerated and observations should not be considered. ◮ Covariance R : R ( x , t , x ′ , t ) = r 0 ( 1 − f ( V ( x , t ))) + r 1 f ( V ( x , t )) δ ( x − x ) δ ( t − t ′ ) and f ( I ( x , t )) = �∇ 3 I ( x , t ) �

Some results Figure: Comparison Data Assimilation / Horn-Schunk – frame 3

Some results Figure: Comparison Data Assimilation / Horn-Schunk – frame 6

Some results Figure: Comparison Data Assimilation / Horn-Schunk – frame 9

... with missing data (a) Image gradient not available (b) Image gradient available Figure: Result without observation on frame 5 and with observation on adjacent frames

Some results Figure: Cuve Coriolis (courtesy of LEGI): comparison Data Assimilation / Horn-Schunk – frame 3

Some results Figure: Cuve Coriolis: comparison Data Assimilation / Horn-Schunk – frame 3

Perspectives • Experiment diffusion as evolution equation. • How to perform spatial regularization: covariance / evolution equation. • Using observation in the diffusion.