High Quality Image Compression Using PDE-Based Inpainting with - PowerPoint PPT Presentation

2016 SIAM Conference on Imaging Science Albuquerque, May 23 – May 26 High Quality Image Compression Using PDE-Based Inpainting with Optimal Data Laurent Hoeltgen hoeltgen@b-tu.de Chair for Applied Mathematics Brandenburg Technical University Cottbus - Senftenberg May 23, 2016 This work is licensed under a Creative Commons “Attribution-ShareAlike 4.0 International” license.

Motivation PDE-Based Image Compression � alternative to transform based approaches � has 3 important pillars � each requires advanced optimisation

Motivation PDE-Based Image Compression � alternative to transform based approaches � has 3 important pillars � each requires advanced optimisation Data Selection

Motivation PDE-Based Image Compression � alternative to transform based approaches � has 3 important pillars � each requires advanced optimisation Data Data Storage Selection

Motivation PDE-Based Image Compression � alternative to transform based approaches � has 3 important pillars � each requires advanced optimisation Data Data Data Storage Selection Reconstruction

Motivation PDE-Based Image Compression � alternative to transform based approaches � has 3 important pillars � each requires advanced optimisation Image Compression Data Data Data Storage Selection Reconstruction

Outlook Data Reconstruction � � PDE-Based Image Inpainting � Data Selection � Optimal Data Locations � Optimal Data Values Data Storage � Summary and Conclusion �

Data Reconstruction (1) Data Reconstruction We could use any diffusion type PDE. However, we want it to be: � simple to analyse � linear, parameter free, ... � applicable for any domain and codomain � no restriction on image type, size, quantisation, ... � fast to carry out Laplace interpolation fulfils all these requirements.

Data Reconstruction (2) Laplace Interpolation for Image Inpainting � consider Laplace equation with mixed boundary conditions: ∂ Ω Ω 8 − ∆ u = 0, on Ω Ω K > < u = g , on Ω K Ω K > : ∂ n u = 0, on ∂ Ω � Ω K : represents known data � Ω \ Ω K : region to be inpainted (i.e. missing data) � image reconstructions given as solutions u

Data Selection (1) Finding Optimal Interpolation Data � brute force search of optimal pixel locations is impossible � 10 5600 ways to choose 5% of data from 0.07 megapixel image � for comparison: 10 23 × =

Data Selection (2) Alternatives � search space reduction with binary trees (Galić et al. 2005, Schmaltz et al. 2009) � analytical derivation (Belhachmi et al. 2008) � probabilistic search methods (Mainberger et al. 2012) � stochastic optimisation on binary trees (Peter et al. 2015) � regularisation and optimal control models (Hoeltgen et al., 2013)

Data Selection (3) Search Space Reduction with Binary Trees � iteratively refine block-wise decomposition � a block is split if error is too large � interpret refinements as nodes of a binary tree Evaluation: � average reconstruction quality � run times of few seconds at most � storing data locations is extremely cheap

Data Selection (4) Binary Tree Representation 1 2 3 4 5 6 7

Data Selection (4) Binary Tree Representation 1 2 3 1 4 5 6 7

Data Selection (4) Binary Tree Representation 1 2 3 2 1 3 4 5 6 7

Data Selection (4) Binary Tree Representation 4 1 2 3 2 2 1 3 4 5 6 7 5

Data Selection (4) Binary Tree Representation 4 1 2 3 2 2 2 1 3 4 5 6 7 6 5 5 7

Data Selection (5) Example

Data Selection (6) Analytical Derivation � derive optimality conditions in Sobolev spaces density of optimal data proportional to Laplacian magnitude � apply dithering to get discrete distribution � works best with electrostatic halftoning (Schmaltz et al. 2010) � Floyd Steinberg is a faster alternative Evaluation: � average reconstruction quality � capable of real time processing (30 fps) � reconstruction data difficult to store

Data Selection (7) Example

Data Selection (7) Example

Data Selection (8) Probabilistic Search Methods � iteratively test importance of random subsets of pixels � discard unimportant pixels � shuffle remaining candidates � similarities to genetic algorithms and simulated annealing � works for any PDE, but only in discrete settings Evaluation: � very good reconstruction quality � extremely slow: run times of days to weeks � reconstruction data difficult to store

Data Selection (9) Example

Data Selection (10) Laplace Interpolation (Reminder) ∂ Ω Ω 8 − ∆ u = 0, on Ω \ Ω K Ω K > < u = g , on Ω K Ω K > : ∂ n u = 0, on ∂ Ω can be rewritten as ( c ( u − g ) + (1 − c ) ( − ∆ ) u = 0, on Ω ∂ n u = 0, on ∂ Ω with c ≡ 1 on Ω K and c ≡ 0 on Ω \ Ω K

Data Selection (11) Regularisation and Optimal Control Models � PDE makes also sense if c : Ω → R ( ≈ regularisation) � optimal non-binary and sparse masks c given by Z ff 1 2 ( u − g ) 2 + λ | c | + ε 2 | c | 2 d x arg min u , c Ω ( c ( u − g ) + (1 − c )( − ∆ ) u = 0, on Ω subject to: ∂ n u = 0, on ∂ Ω Evaluation: � highest quality of all presented methods � slow: run times of hours to days � reconstruction data difficult to store

Data Selection (12) Example

Data Selection (13) Tonal Optimisation � optimal pixel values necessary to maximise quality � quantisation to n ≪ 256 colours essential for compression � best number of colours hard to determine � often 30% less memory needed � optimisation criteria should take file size into account � hard to predict

Data Selection (14) Continuous Tonal Optimisation � optimise values in R � Mainberger et al. (2011): least squares model � Hoeltgen et al. (2015): equivalence to mask optimisation optimal non-binary masks c equivalent to optimal colours in R with binary masks! � maximises reconstruction quality � storage of colour values in R is very expensive

Data Selection (15) Discrete Tonal Optimisation � Schmaltz et al. (2009) iteratively optimise colour mapping � uniform distribution of colours necessary � very slow � number of colours fixed beforehand � excellent quality � Extension to take file size into account by Peter (2014) � k-means based alternative by Hoeltgen et al. (2016) � arbitrary distribution of colours possible � much faster � allows optimisation of quantisation levels � doesn’t consider file size

Data Selection (16) Discrete/Continuous Tonal Optimisation � apply K-means on colour values from mask locations 52 mean squared error K-means original data 50 48 46 10 15 20 25 30 35 number of clusters � K-means 30 colours outperforms original data (170 colours) � optimal quantisation found with silhouette statistics

Data Storage (1) Flexible Codec Design � depending on the application: 1. preserve quality at the expense of larger file size � arbitrary mask positions � tonal optimisation of data and binarisation of mask 2. sacrifice accuracy to maximise compression rate � use binary tree representation for mask � tonal optimisation

Data Storage (2) Image Encoding and Decoding Encoding: 1. computation of a binary mask (any strategy) 2. computation of optimal grey values (with quantisation) 3. encoding of all the data in a container file 4. application of a high performing entropy coder (PAQ) Decoding: � reverse order with additional final inpainting � can be done in real time (30 fps) (Baum 2013) independent from chosen optimisation strategies!

Data Storage (3) Comparison to JPEG and JPEG 2000 exact mask tree mask mean squared error JPEG 30 JPEG 2000 20 10 10 : 1 15 : 1 20 : 1 compression rate

Data Storage (3) Comparison to JPEG and JPEG 2000 exact mask tree mask mean squared error JPEG 30 JPEG 2000 20 10 10 : 1 15 : 1 20 : 1 compression rate

Data Storage (3) Comparison to JPEG and JPEG 2000 exact mask tree mask mean squared error JPEG 30 JPEG 2000 20 10 10 : 1 15 : 1 20 : 1 compression rate

Summary and Conclusion Take Home Message PDE-based compression is a viable alternative: � applicable to any data (images, videos, hyper-spectral, ...) � user friendly (single parameter for quality setting) � well suited if encoding time does not matter � mathematically well founded

Thank You Thank you very much for your attention! More information at: http://www-user.tu-cottbus.de/~hoeltgen/

References Laurent Hoeltgen and Michael Breuß. “Efficient Co-Domain Quantisation for PDE-based Image Compression”. Proceedings of the Conference Algoritmy . 2016. Pascal Peter et al. “Evaluating the True Potential of Diffusion-based Inpainting in a Compression Context”. Signal Processing: Image Communication (2016). Laurent Hoeltgen and Joachim Weickert. “Why Does Non-binary Mask Optimisation Work for Diffusion-based Image Compression?” Energy Minimization Methods in Computer Vision and Pattern Recognition . 2015. Kevin Baum. “GPGPU Supported Diffusion-based Naive Video Compression”. Master’s Thesis. Saarland University, 2013.