Outlier-aware Dictionary Learning for Sparse Representation Sajjad - PowerPoint PPT Presentation

Outlier-aware Dictionary Learning for Sparse Representation Sajjad Amini ∗ Mostafa Sadeghi ∗ Mohsen Joneidi ∗ Massoud Babaie-Zadeh ∗ Christian Jutten ∗∗ ∗ Department of Electrical Engineering, Sharif University of Technology, Tehran, Iran. ∗∗ GIPSA-Lab, Grenoble, and Institut Universitaire de France, France. September 2014

Sparse Representation Underdetermind Linear System of Equations y = Dx + e ✑ y ∈ R n , D ∈ R n × K , x ∈ R K , e ∈ R n ✑ D = [ d 1 d 2 . . . d K ] : dictionary, d i : atom ✑ The dictionary is usually overcomplete: K > n Sparse representation problem: x ∗ = argmin � x � 0 subject to � y − Dx � 2 ≤ ǫ x 2

Choosing the Dictionary Pre-defined and fixed dictionaries: Fourier, Gabor, DCT, wavelet, . . . ✦ Fast computations ✪ Unable to sparsely represent a given signal class Learned dictionaries ✦ More efficient for sparse representation ✦ Very promising results in many applications: image enhancement, pattern recognition, . . . ✪ High computational load 3

Dictionary Learning (DL) . . . Noisy Sample Y . . . . . . Given a noisy training data matrix, Y = [ y 1 , . . . , y L ] , the goal is to find an over-complete set of basis functions (atoms) over which each data can be sparsely represented Training Data Model y i = Dx i + n i i = 1 , . . . , L ✑ p ( x ) ∝ exp( − � x � 1 β 1 ) ✑ p ( n ) ∝ exp( − � n � 2 2 β 2 ) 4

Dictionary Learning (DL) MAP Estimation of Dictionary and Representations � � d i � 2 = 1 , i = 1 , . . . , K i � y i − Dx i � 2 � min D , { x i } L subject to 2 � x j � 0 ≤ T 0 , j = 1 , . . . , L i =1 � x � 0 � | supp ( x ) = { i : x i � = 0 } | Solution to the Dictionary Learning Problem Alternating Minimization Starting with an initial dictionary, the following two stages are repeated several times: 1 Sparse representation: X ( k +1) = argmin X ∈X � Y − D ( k ) X � 2 F subject to � x j � 0 ≤ T 0 , j = 1 , . . . , L = ⇒ OMP 2 Dictionary update: D ( k +1) = argmin � Y − DX ( k +1) � 2 F subject to � d i � 2 = 1 , i = 1 , . . . , K = ⇒ Differentiating D ∈D 5

Image Denoising Using Dictionary Learning Convert Image Patch to Vector 1 4 7 5 8 2 6 9 3 Image 1 2 Y Noisy Pixel 3 4 5 6 7 8 9 Image Denoising Training Image Dictionary ( D ⋆ ) Estimating Denoised Image Patch Representation ✑ ˆ x = OMP D ⋆ ( y ) Reconstructing Denoised Image Patch ✑ ˆ y = D ⋆ ˆ x 6

Robust Dictionary Learning . . . Noiseless Sample Y . . . Outlier . . . Problem Formulation Training Data Model: y i = Dx i + n i i = 1 , . . . , L ✑ p ( x ) ∝ exp( − � x � 1 β 1 ) ✑ p ( n ) ∝ exp( − � n � 1 β 2 ) MAP Estimation of Dictionary and Representations � i ( � y i − Dx i � 1 + λ � x i � 1 ) subject to � d i � 2 = 1 , i = 1 , . . . , K min D , { x i } L � w.r.t. D ⇒ Iteratively Reweighted Least Squares i =1 ✑ Strategy to Solve = ⇒ w.r.t. { x i } L i =1 ⇒ Iteratively Reweighted Least Squares 7

Robust Dictionary Learning by Error Source Decomposition . . . Noisy Sample Y . . . Outlier . . . Problem Formulation Training Data Model: y i = Dx i + n i + o i i = 1 , . . . , L ✑ p ( x ) ∝ exp( − � x � 1 β 1 ) ✑ p ( n ) ∝ exp( − � n � 2 β 2 ) 2 ✑ p ( o ) ∝ exp( − � o � 1 β 3 ) MAP Estimation of Dictionary and Representations � � d i � 2 = 1 , i = 1 , . . . , K i ( � y i − Dx i − o i � 2 � min D , { x i } L 2 + λ � o i � 1 ) subject to i =1 , { o i } L � x j � 0 ≤ T 0 , j = 1 , . . . , L i =1 � w.r.t. D ⇒ Differentiating ✑ Strategy to Solve = ⇒ w.r.t. { x i } L i =1 , { o i } L i =1 ⇒ Shrinkage 8

Outlier Aware Dictionary Learning (Proposed) . . . Noisy Sample Y . . . Outlier . . . Problem Formulation Training Data Model: y i = Dx i + n i + o i i = 1 , . . . , L ✑ p ( x ) ∝ exp( − � x � 1 β 1 ) ✑ p ( n ) ∝ exp( − � n � 2 β 2 ) 2 ✑ p ( o ) ∝ exp( − � o � 2 β 3 ) MAP Estimation of Dictionary and Representations � � d i � 2 = 1 , i = 1 , . . . , K i ( � y i − Dx i − o i � 2 � 2 + λ � o i � 2 ) subject to min D , { x i } L i =1 , { o i } L � x j � 0 ≤ T 0 , j = 1 , . . . , L i =1 � � d i � 2 = 1 , i = 1 , . . . , K min D , X , O � Y − DX i − O � 2 F + λ � O � 21 subject to � x j � 0 ≤ T 0 , j = 1 , . . . , L 9

Outlier Aware Dictionary Learning (Proposed) Solution Strategy Alternating Minimization: ✑ w.r.t. { x i } L i =1 ⇒ x i = OMP D ( y i − o i ) , i = 1 , . . . , L � (1 − λ if � r i � 2 > λ 2 � r i � 2 ) r i , ✑ w.r.t. { o i } L 2 i =1 ⇒ o i = r i = y i − Dx i , i = 1 , . . . , L otherwise 0 , ✑ w.r.t. D ⇒ D = ( Y − O ) X T ( XX T ) − 1 Initialization ✑ D ⇒ Overcomplete DCT ✑ { o i } L i =1 ⇒ o i = 0 , i = 1 , . . . , L At the beginning, all training signal are considered not to be an outlier. 10

Image Denoising Based on OADL Convert Image Patch to Vector 1 7 4 2 5 8 3 6 9 Image 1 2 Y Noisy Pixel 3 4 Outlier 5 6 7 8 9 Image Denoising Training Image Dictionary Using OADL( D ⋆ ) Estimating Denoised Image Patch Representation � min x � x � 0 subject to � ( y − o ) − Dx � 2 ≤ ǫ ✑ ( ˆ o ) = Alternating Minimization x , ˆ min o � ( y − Dx ) − o � 2 2 + � o � 2 Reconstructing Denoised Image Patch ✑ ˆ y = D ⋆ ˆ x 11

Simulation Results Synthetic Data Generate a random dictionary ( D ) Generate 2500 training signals and 500 test signals using 3 atoms of D Add Gaussian noise ( N ( 0 , 0 . 01 2 I ) ) to each of training signals Add Gaussian noise ( N ( 0 , 0 . 04 2 I ) ) to p % of randomly selected training signals (outlier) Train a dictionary using training signals ( D ⋆ ) Evaluate the ability of dictionary to code test signals using 3 atoms of D ⋆ Image Denoising Select 6 benchmark images ( 256 × 256 ) Add Gaussian noise ( N (0 , 10 2 ) ) to each of image pixels Add Gaussian noise ( N (0 , 20 2 ) ) to B blocks of pixels according to the following pattern Denoise the resultant image 1 2 3 4 6 5 7 8 9 Figure: Outlier block pattern 12

Simulation Results - Synthetic Data 0.2 0.2 0.2 Representation RMSE Representation RMSE Representation RMSE MOD MOD MOD OADL OADL OADL 0.15 0.15 0.15 0.1 0.1 0.1 0.05 0.05 0.05 0 0 0 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 Iteration Iteration Iteration (a) p = 2% (b) p = 6% (c) p = 12% -3 8x 10 6 ∆ RMSE 4 2 0 2 4 6 8 10 12 Percentage of corrupted samples (d) Final RMSE difference Figure: (a)-(c) Test data representation RMSE along iterations for p = 2, 4 and 6, respctively. (d) Final representation RMSE of test data versus p 13

Simulation Results - Synthetic Data 0.9 0.8 0.7 λ opt 0.6 0.5 0.4 0.3 2 4 6 8 10 12 Percentage of corrupted samples Figure: Best λ , which minimizes test data representation RMSE, versus percentage of outliers. � p directly related to β 3 λ inversely related to β 3 ⇒ p inversely related to λ p = 2% to 6% � Violation of outlier sparsity assumption OADL approaches regular Dictionary Learning ⇒ p directly related to λ p = 6% to 12% 14

Simulation Results - Image Denoising 34 OADL MOD 33 DCT PSNR (dB) 32 31 30 0 2 4 6 8 Number of outlier blocks Figure: Averaged PSNR over 6 different test images versus number of outlier blocks. 100 95 90 λ opt 85 80 75 70 0 2 4 6 8 Number of outlier blocks Figure: Best λ , which maximizes PSNR, versus percentage of outliers. 15

Conclusions A new and practical placement of outlier was considered. We introduce a new model for training signals based on separating noise and outlier source. We formulate DL problem using MAP estimation. We introduce a fast and efficient algorithm to solve the proposed robust dictionary learning problem. Simulation results showed that our new method leads to considerable improvements over traditional methods 16

References 1 K. Engan, S. O. Aase, and J. Hakon Husoy, “Method of optimal directions for frame design,” IEEE International Conference on Acoustics, Speech, and Signal Processing , vol. 5, pp. 2443–2446, 1999. 2 M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Trans. on Signal Processing , vol. 54, no. 11, pp. 4311–4322, 2006. 3 M. Elad and M. Aharon, “Image Denoising Via Sparse and Redundant representations over Learned Dictionaries,” IEEE Trans. on Image Processing , vol. 15, no. 12, pp. 3736-3745, December 2006. 4 C. Lu, J. Shi, and J. Jia, “Online robust dictionary learning,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 415–422. June 2013. 5 Z. Chen, and Y. Wu, “Robust Dictionary Learning by Error Source Decomposition,” IEEE International Conference on Computer Vision (ICCV) , pp. 2216–2223, December 2013. 17