Optimal Interpolation Data 5 6 7 8 for Image Reconstructions 9 - PowerPoint PPT Presentation

Mathematical Image Analysis Group 1 2 3 4 Optimal Interpolation Data 5 6 7 8 for Image Reconstructions 9 10 11 12 13 14 PhD. Defense Talk 15 16 17 18 Laurent Arthur Hoeltgen 19 20 21 22 March 30th, 2015 23 24 25 26 This work is licensed under a Creative 27 28 Commons ‘Attribution-ShareAlike 4.0 In- 29 30 ternational’ license. 31 32 33 34

Motivation (1) 1 2 What Is Inpainting? 3 4 Recover missing data in given image through interpolation. 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Image with Reconstruction with 29 30 missing data Laplace interpolation 31 32 33 34

Motivation (2) 1 2 3 4 Standard setting: 5 6 7 8 impossible to choose missing data 9 10 requires optimal interpolation methods 11 12 13 14 We take a different approach: 15 16 17 18 fix interpolation method 19 20 select sparse and optimised data 21 22 23 24 Is it possible to get good results with only 5% of data? 25 26 27 28 Why do this? 29 30 31 32 allows image compression 33 34

Motivation (3) 1 2 How Important Is a Good Optimisation? 3 4 Position of interpolation data matters when sparse (say 5% ): 5 6 (Mainberger et al. 2012) 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Original image 27 28 29 30 31 32 Random positions Optimal positions 33 34

Motivation (4) 1 2 3 4 5 6 Pixel values at given positions equally important (Mainberger et al. 2012) 7 8 9 10 Reconstructions from same random mask with 5% density: 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Original Without grey value With grey value 27 28 image optimisation optimisation 29 30 31 32 How to find all this optimal data? 33 34

Motivation (5) 1 2 Related work 3 4 Previous findings on PDE-Based inpainting and compression: 5 6 7 8 Contour-based reconstructions (Carlsson 1988) 9 10 11 12 Wavelet decompositions with TV (Chan et al. 2001) 13 14 Toppoints in scale space (Kanter et al. 2005) 15 16 17 18 Spline based representations (Orzan et al. 2008) 19 20 Subdivision schemes (Galić et al. 2008, Schmaltz et al. 2009) 21 22 23 24 Variational approach (Belhachmi et al. 2009) 25 26 Stochastic optimisation (Mainberger et al. 2012) 27 28 29 30 Bilevel optimisation (Chen et al. 2014) 31 32 33 34

Outline 1 2 3 4 5 6 7 8 PDE-Based Image Inpainting 9 10 11 12 13 14 Finding Optimal Data Locations 15 16 17 18 19 20 Finding Optimal Data Values 21 22 23 24 An Image Compression Codec 25 26 27 28 29 30 31 32 33 34

PDE-Based Image Inpainting (1) 1 2 PDE-Based Image Inpainting 3 4 5 6 Which interpolation operator should be used? 7 8 9 10 Requirements: 11 12 13 14 1. simple to analyse 15 16 2. applicable for any domain and codomain 17 18 19 20 3. able to handle arbitrarily scattered data 21 22 4. fast to carry out 23 24 25 26 27 28 Laplace interpolation fulfils all these requirements. 29 30 31 32 33 34

PDE-Based Image Inpainting (2) 1 2 Laplace Interpolation for Image Inpainting 3 4 Consider the Laplace equation with mixed boundary conditions. 5 6 7 8 9 10 ∂Ω Ω 11 12  13 14 − ∆u = 0 , on Ω Ω K   15 16 u = g, on Ω K Ω K 17 18  ∂ n u = 0 , on ∂Ω  19 20 21 22 23 24 Ω K : represents known data 25 26 27 28 Ω \ Ω K : region to be inpainted (i.e. unknown data) 29 30 Image reconstructions are solutions u . 31 32 33 34

PDE-Based Image Inpainting (3) 1 2 3 4 5 6 ∂Ω Ω 7 8  − ∆u = 0 , on Ω Ω K  9 10  u = g, on Ω K Ω K 11 12  ∂ n u = 0 , on ∂Ω  13 14 15 16 Mixed boundary value problem can be rewritten as 17 18 19 20 c ( u − g ) + (1 − c ) ( − ∆ ) u = 0 , on Ω 21 22 ∂ n u = 0 , on ∂Ω 23 24 with c ≡ 1 on Ω K and c ≡ 0 on Ω \ Ω K . 25 26 Optimising binary c known as free knot problem for 1D signals. 27 28 29 30 Previous equation makes also sense if c maps to whole R . 31 32 can be seen as a regularisation 33 34 unclear if above equation remains solvable

PDE-Based Image Inpainting (4) 1 2 Benefits of a Non-Binary Mask c 3 4 5 6 Finding binary masks is a non-convex, combinatorial task. 7 8 Non-binary masks allow better fits to the data. 9 10 11 12 original 4 reconstruction 3 13 14 mask c 15 16 1 17 18 19 20 3 4 21 22 1 23 24 2 25 26 1 4 27 28 29 30 31 32 Reconstruction by piecewise linear inerpolation in 1D. 33 34

PDE-Based Image Inpainting (4) 1 2 Benefits of a Non-Binary Mask c 3 4 5 6 Finding binary masks is a non-convex, combinatorial task. 7 8 Non-binary masks allow better fits to the data. 9 10 11 12 original 4 reconstruction 3 13 14 mask c 15 16 1 17 18 19 20 3 4 21 22 1 23 24 2 25 26 1 4 27 28 29 30 31 32 L 2 Error: 0.45 33 34

PDE-Based Image Inpainting (4) 1 2 Benefits of a Non-Binary Mask c 3 4 5 6 Finding binary masks is a non-convex, combinatorial task. 7 8 Non-binary masks allow better fits to the data. 9 10 11 12 original 4 reconstruction 3 13 14 mask c 15 16 1 17 18 19 20 3 4 21 22 1 23 24 2 25 26 1 4 27 28 29 30 31 32 L 2 Error: 0.43 33 34

PDE-Based Image Inpainting (4) 1 2 Benefits of a Non-Binary Mask c 3 4 5 6 Finding binary masks is a non-convex, combinatorial task. 7 8 Non-binary masks allow better fits to the data. 9 10 11 12 original 4 reconstruction 3 13 14 mask c 15 16 1 17 18 19 20 3 4 21 22 1 23 24 2 25 26 1 4 27 28 29 30 31 32 L 2 Error: 0.41 33 34

PDE-Based Image Inpainting (4) 1 2 Benefits of a Non-Binary Mask c 3 4 5 6 Finding binary masks is a non-convex, combinatorial task. 7 8 Non-binary masks allow better fits to the data. 9 10 11 12 original 4 reconstruction 3 13 14 mask c 15 16 1 17 18 19 20 3 4 21 22 1 23 24 2 25 26 1 4 27 28 29 30 31 32 L 2 Error: 0.39 33 34

PDE-Based Image Inpainting (4) 1 2 Benefits of a Non-Binary Mask c 3 4 5 6 Finding binary masks is a non-convex, combinatorial task. 7 8 Non-binary masks allow better fits to the data. 9 10 11 12 original 4 reconstruction 3 13 14 mask c 15 16 1 17 18 19 20 3 4 21 22 1 23 24 2 25 26 1 4 27 28 29 30 31 32 L 2 Error: 0.37 33 34

PDE-Based Image Inpainting (4) 1 2 Benefits of a Non-Binary Mask c 3 4 5 6 Finding binary masks is a non-convex, combinatorial task. 7 8 Non-binary masks allow better fits to the data. 9 10 11 12 original 4 reconstruction 3 13 14 mask c 15 16 1 17 18 19 20 3 4 21 22 1 23 24 2 25 26 1 4 27 28 29 30 31 32 L 2 Error: 0.34 33 34

PDE-Based Image Inpainting (5) 1 2 New Findings 3 4 Standard finite difference schemes in 2D for 5 6 7 8 c ( u − g ) + (1 − c ) ( − ∆ ) u = 0 , on Ω 9 10 ∂ n u = 0 , on ∂Ω 11 12 13 14 yield a linear system of equations. 15 16 The system matrix has the following properties: 17 18 1. All eigenvalues are real if c is bounded by 1 . 19 20 21 22 0 , 8 � � 2. Matrix is invertible if c maps to . 7 23 24 3. Solutions obey max-min principle if c maps to [0 , 1] . 25 26 27 28 Generalisation of Mainberger et al. (2011) to the non-binary case. 29 30 31 32 33 34

Finding Optimal Data Locations (1) 1 2 A Novel Optimal Control Model (EMMCVPR 2013) 3 4 Optimal non-binary masks c for recovering g obtained from 5 6 7 8 � 2 ( u − g ) 2 + λ | c | + ε 1 2 | c | 2 d x } arg min { 9 10 u, c Ω 11 12 � c ( u − g ) + (1 − c )( − ∆ ) u = 0 , on Ω 13 14 subject to: ∂ n u = 0 , on ∂Ω 15 16 17 18 19 20 PDE enforces that we only get suitable solutions. 21 22 one solution u ( c ) for each valid choice of c 23 24 Cost function optimises reconstructions u and masks c . 25 26 2 ( u − g ) 2 favours accurate reconstructions. 1 27 28 29 30 λ | c | prefers sparse data sets. 31 32 33 34

Finding Optimal Data Locations (2) 1 2 Interpretation 3 4 5 6 7 8 � 2 ( u − g ) 2 + λ | c | + ε 1 2 | c | 2 d x } arg min { 9 10 u, c Ω 11 12 � c ( u − g ) + (1 − c )( − ∆ ) u = 0 , on Ω 13 14 subject to: ∂ n u = 0 , on ∂Ω 15 16 17 18 19 20 Energy reflects trade-off between accuracy and sparsity. 21 22 Objectives cannot be fulfilled simultaneously. 23 24 λ steers sparsity of the interpolation data. 25 26 Small, positive λ : many data points, good reconstruction 27 28 Large, positive λ : few data points, bad reconstruction 29 30 31 32 33 34

Finding Optimal Data Locations (3) 1 2 Properties 3 4 5 6 7 8 � 2 ( u − g ) 2 + λ | c | + ε 1 2 | c | 2 d x } arg min { 9 10 u, c Ω 11 12 � c ( u − g ) + (1 − c )( − ∆ ) u = 0 , on Ω 13 14 subject to: ∂ n u = 0 , on ∂Ω 15 16 17 18 19 20 Optimal control problem 21 22 Model has similarites to Belhachmi et al. (2009) 23 24 25 26 Large scale optimisation (often � 100 000 unknowns) 27 28 λ | c | is non-differentiable 29 30 31 32 Constraint is non-convex due to mixed products in c and u . 33 34