Dictionary Learning for Graph Signals 236862 Introduction to Sparse - PowerPoint PPT Presentation

Dictionary Learning for Graph Signals 236862 – Introduction to Sparse and Redundant Representations joint work with Yael Yankelevsky 22.12.2019 Prof. Michael Elad

The Sparseland Model Dictionary Learning: = Model assumption: All data vectors are linear combinations of FEW ( 𝑈 ≪ 𝑂 ) atoms from 𝐄 Dictionary Learning for Graph Signals 2 By: Yael Yankelevsky

K-SVD Algorithm Overview [Aharon et al. ’ 06] ≈ … … X Y D . . Dictionary Initialize Sparse Coding Update Dictionary Using OMP Atom-by-atom + coeffs. For the j-th atom: Dictionary Learning for Graph Signals 3 By: Yael Yankelevsky

Data is often structured… Biological Networks Energy Networks Meshes & Point Clouds Transportation Networks Social Networks Dictionary Learning for Graph Signals 4 By: Yael Yankelevsky

What happens for non-conventionally structured signals? Can dictionary learning work well for such signals as well? The general idea: Model the underlying structure as a graph and incorporate it in the dictionary learning algorithm Dictionary Learning for Graph Signals 5 By: Yael Yankelevsky

Basic Notations 𝑤 2 We are given a graph: 𝑤 13 𝑤 1 The 𝑗 𝑢ℎ node is characterized  by a feature vector 𝑤 𝑗 𝑤 9 𝑤 10  The edge between the 𝑗 𝑢ℎ and 𝑤 8 𝑘 𝑢ℎ nodes carries a weight 𝑤 4 𝑤 11 −1 𝑥 𝑗𝑘 ∝ 𝑒 𝑤 𝑗 , 𝑤 𝑘 𝑤 3  The degree matrix: 𝑤 12 𝑤 7  Graph Laplacian: 𝑤 6 𝑤 5 Dictionary Learning for Graph Signals 6 By: Yael Yankelevsky

Basic Notations 𝑤 2 2 𝑔 𝑤 13 The 𝑗 𝑢ℎ node has a value 𝑔  𝑗 𝑤 1 13 𝑔 f = graph signal 1 𝑔 𝑤 9  The combinatorial Laplacian is a 𝑤 10 9 𝑔 𝑤 8 differential operator: 10 𝑔 8 𝑔 𝑤 4 𝑤 11 4 𝑔 11 𝑔 12 𝑤 3 𝑔  3 Defines global regularity on the 𝑔 𝑤 12 7 𝑤 7 𝑔 graph (Dirichlet energy): 𝑤 6 𝑤 5 6 𝑔 5 𝑔 Dictionary Learning for Graph Signals 7 By: Yael Yankelevsky

Related Work: Dictionaries for Graph Signals  Ignore structure (MOD, K-SVD) [Engan et al. ’ 99],[Aharon et al. ’ 06]  Analytic transforms  Graph Fourier transform [Sandryhaila & Moura ’ 13]  Windowed Graph Fourier transform [Shuman et al. ’ 12]  Graph Wavelets [Coifman & Maggioni ’ 06],[Gavish et al. ’ 10],[Hammond et al. ’ 11],[Ram et al. ’ 12 ],[Shuman et al. ’ 16 ],…  Structured learned dictionaries [Zhang et al. ’ 12],[Thanou et al. ’ 14] Our solution: Graph Regularized Dictionary Learning Dictionary Learning for Graph Signals 8 By: Yael Yankelevsky

The Basic Concept Construct 2 graphs capturing the feature dependencies and the data manifold structure Y L c L Dictionary Learning for Graph Signals 9 By: Yael Yankelevsky

Example: Traffic Dataset Y Dictionary Learning for Graph Signals 10 By: Yael Yankelevsky

Dual Graph Regularized Dictionary Learning Introduce graph regularization terms that preserve these structures Imposed smoothness (graph Dirichlet energy): Dictionary Learning for Graph Signals 11 By: Yael Yankelevsky

The Importance of the Underlying Graph [Shuman et al. ’ 13] A good estimation of L is crucial! We can learn L and adapt it to promote the desired smoothness [Hu et al. ’ 13] , [Dong et al. ‘ 15], [Kalofolias ’ 16], [Segarra et al. ‘ 17 ],… Dictionary Learning for Graph Signals 12 By: Yael Yankelevsky

Dual Graph Regularized Dictionary Learning Key idea: Dictionary atoms preserve feature similarities Similar signals have similar sparse representations The graph is adapted to promote the desired smoothness Dictionary Learning for Graph Signals 13 By: Yael Yankelevsky

The DGRDL Algorithm Graph Update Sparse Coding Dictionary Update Initialize Using ADMM Atom-by-atom + coeffs. Dictionary pursuit (modified update rule) Dictionary Learning for Graph Signals 14 By: Yael Yankelevsky

The DGRDL Algorithm For the j-th atom: Graph Update Sparse Coding Dictionary Update Initialize ? Using ADMM Atom-by-atom + coeffs. Dictionary pursuit (modified update rule) Dictionary Learning for Graph Signals 15 By: Yael Yankelevsky

Graph Regularized Pursuit Dictionary Learning for Graph Signals 16 By: Yael Yankelevsky

Graph Regularized Pursuit ADMM: [Boyd et al. ’ 11] Dictionary Learning for Graph Signals 17 By: Yael Yankelevsky

Graph Regularized Pursuit graph SC [Zheng et al. ’ 11] ADMM: [Boyd et al. ’ 11] Dictionary Learning for Graph Signals 18 By: Yael Yankelevsky

Theoretical Guarantees Classical sparse theory: Theorem: If the true representation 𝐲 satisfies 𝐲 0 = s < 1 1 2 1 + μ 𝐄 ϵ ) must be close to it then a solution 𝐲 for (𝐐 0 4ϵ 2 4ϵ 2 2 ≤ 𝐲 − 𝐲 2 ≤ 1 − δ 2s 1 − 2s − 1 μ 𝐄 Dictionary Learning for Graph Signals 19 By: Yael Yankelevsky

Theoretical Guarantees Graph sparse coding: Theorem: If the true representation 𝐘 satisfies 𝐘 0,∞ = s < 1 2 1 + 1 + f β, 𝐌 𝐝 μ 𝐄 for (𝐐 𝟏,∞ ϵ then a solution 𝐘 ) must be close to it 4ϵ 2 4ϵ 2 2 ≤ − 𝐘 F 𝐘 ≤ 1 − δ 2s 1 − 2s − 1 μ 𝐄 + f β, 𝐌 𝐝 ≥ 0 Dictionary Learning for Graph Signals 20 By: Yael Yankelevsky

Back to DGRDL… dictionary atoms are similar signals have smooth graph signals similar sparse codes graph is adapted to promote smoothness Graph Update Sparse Code Dictionary Update Initialize Using ADMM Atom-by-atom + coeffs. Dictionary pursuit (modified update rule) Dictionary Learning for Graph Signals 21 By: Yael Yankelevsky

Results: Network Data Recovery Dictionary Learning for Graph Signals 22 By: Yael Yankelevsky

Traffic Dataset Settings:  N=578 sensors  M=2892 signals  1500 for training  1392 for testing  Graph signal = daily avg. bottleneck (min.) measured at each station Dictionary Learning for Graph Signals 23 By: Yael Yankelevsky

Representation Sparse Coding . . . . . . Original Reconstructed signals signals Dictionary Learning for Graph Signals 24 By: Yael Yankelevsky

Representation K-SVD GRDL (fixed L) GRDL (learned L) DGRDL Dictionary Learning for Graph Signals 25 By: Yael Yankelevsky

Denoising AWGN + Sparse Coding . . . . . . . . . Original Noisy Reconstructed signals signals signals Dictionary Learning for Graph Signals 26 By: Yael Yankelevsky

Denoising K-SVD GRDL (fixed L) GRDL (learned L) DGRDL Dictionary Learning for Graph Signals 27 By: Yael Yankelevsky

Inpainting Discard p% of the Sparse Coding samples randomly . . . . . . . . . Original Incomplete Reconstructed signals signals signals Dictionary Learning for Graph Signals 28 By: Yael Yankelevsky

Inpainting K-SVD GRDL (fixed L) GRDL (learned L) DGRDL Dictionary Learning for Graph Signals 29 By: Yael Yankelevsky

Results: Image Denoising Revisited Dictionary Learning for Graph Signals 30 By: Yael Yankelevsky

A Glimpse at Image Processing 𝑁 𝑧 𝑘 𝑜 𝑜 2 𝑜  𝐌 is an 𝑜 × 𝑜 grid (patch structure) 𝑧 𝑗 𝐄 is learned from only 1000 patches  Dictionary Learning for Graph Signals 31 By: Yael Yankelevsky

Image Denoising ( σ =25) Original Noisy (20.18dB) K-SVD (28.35dB) DGRDL (28.50dB) Original Noisy (20.18dB) K-SVD (30.56dB) DGRDL (30.71dB) Dictionary Learning for Graph Signals 32 By: Yael Yankelevsky

Structure Inference Learn the underlying patch structure (pixel dependencies) from the data input image learned L Dictionary Learning for Graph Signals 33 By: Yael Yankelevsky

Time to Conclude… Processing data is enabled We have shown how by an appropriate sparsity-based models modeling that exposes its become applicable also for inner structure graph structured data Extensions include We developed an supervised efficient algorithm for dictionary learning joint learning of the and supporting dictionary and the graph We demonstrated how high dimensions various applications can benefit from the new model Dictionary Learning for Graph Signals 34 By: Yael Yankelevsky

Thank You Dictionary Learning for Graph Signals 35 By: Yael Yankelevsky