Bag of Pursuits and Neural Gas for Improved Sparse Coding Manifold - PowerPoint PPT Presentation

Bag of Pursuits and Neural Gas for Improved Sparse Coding Manifold Learning with Sparse Coding Thomas Martinetz Institute for Neuro- and Bioinformatics University of L¨ ubeck 26.8.2010 1 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Natural signals and images Natural signals usually occupy only a small fraction within the signal space. Example: natural images lie on a submanifold within the high-dimensional image space. Knowledge about this submanifold is helpful in many respects. 2 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Example: image reconstruction 90% of the pixels are missing. Reconstruction by projection onto the submanifold. image Image dimension 600 x 400 = 240 . 000 Submanifold dim. ≈ 10 . 000 3 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Submanifold representation Submanifold representation by Vector Quantization. Each point on the submanifold is represented by its closest reference vector w i ∈ R N . w i The w i can be learned by k -means, Neural Gas or many others. Image reconstruction through the w i closest to the image. Submanifold representation by linear subspaces of zero dimension. 4 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Submanifold representation Submanifold representation by linear subspaces. Each linear subspace of dimension K is defined by W i ∈ R N × ( K +1) . Each point on the submanifold is represented by its closest W i linear subspace W i . The W i can be learned similar to k -means or Neural Gas. Image reconstruction through the closest point on the closest subspace. 5 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Compact description To describe L linear subspaces of dimension K with individual W i we need L × N × ( K + 1) parameters. However, this description can be highly redundant. For example, N subspaces of dimension N − 1 can be described by O ( N 2 ) instead of N 3 parameters. A ” K out of M ”structure can be much more compact. 6 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Compact description N = 3, subspace dimension K = 2, number of subspaces L = 3 W 1 = ( w 0 (1) , w 1 (1) , w 2 (1) ) W 2 = ( w 0 (2) , w 1 (2) , w 2 (2) ) W 3 = ( w 0 (3) , w 1 (3) , w 2 (3) ) C = ( c 1 , c 2 , c 3 , c 4 ) 7 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Compact description by sparse coding Forming K dimensional subspaces by choosing K vectors out of a set (dictionary) C of M vectors allows to realize � M � L = K subspaces. Finding the closest subspace to a given x requires to solve the optimization problem � x − C a � 2 min subject to � a � 0 = K 2 a Problem 1 : NP-hard combinatorial optimization problem Problem 2 : How to choose C for a given K ? 8 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Important Message The manifold learning problem can be cast into the sparse coding and compressive sensing framework. 9 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

(Approximately) solving the NP-hard problem Greedy Optimization Directly tackle the problem by a pursuit method Matching Pursuit Orthogonal Matching Pursuit Optimized Orthogonal Matching Pursuit If x has a sparse enough ( K << N ) representation, and C fulfills certain properties, the solution provided by the pursuit methods will be the optimal solution (Donoho 2003). 10 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

How to choose C (and K )? Given data x 1 , . . . , x p , x i ∈ R N (like natural images) which are supposed to lie on an unknown submanifold. The goal is to find a C which provides a small average reconstruction error for a K which is as small as possible. Find C = ( c 1 , . . . , c M ) with c j ∈ R N and a i ∈ R M minimizing p E = 1 � � x i − C a i � 2 2 L i =1 Constraints a i : � a i � 0 = K C : � c j � = 1 (without loss of generality) 11 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Predefined dictionaries for image data How to chose C ? Overcomplete 8 × 8 Overcomplete 8 × 8 DCT-Dictionary HAAR-Dictionary 12 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Learning dictionaries 13 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Learning dictionaries The problem: find � � � a � x i − C a � 2 min min subject to � a � 0 = K 2 C i Current state-of-the-art solver: MOD (Engan et al 1999) K-SVD (Aharon et al 2006) Our new approach: Neural-Gas-like soft-competitive stochastic gradient descent. Generalization of the Neural Gas to linear subspaces within the sparse coding framework. 14 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

What was Neural Gas? With a randomly chosen data point x reference vectors for Vector Quantization w i are updated according to ∆ w j l = α t e − l λ t ( x − w j l ) 0 = 1 , ..., L − 1 w j 0 is the reference vector closest to x w j 1 is the reference vector second closest to x etc. The update step decreases with the distance rank (reconstruction error) of the reference vectors to the data point x . Neural Gas performs soft-competitive stochastic gradient descent on the Vector Quantization error function. Neural Gas provides very good and robust solutions to the Vector Quantization problem. 15 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Generalization to linear subspaces With a randomly chosen data point x the linear subspaces W i are updated according to ∆ W j l = α t e − l λ t ( x − W j l a j l ) a T l = 0 , ..., L − 1 j l with � x − W j l a � 2 a j l = arg min 2 a W j 0 is the linear subspace closest to x W j 1 is the linear subspace second closest to x etc. The update step decreases with the distance rank (reconstruction error) of the linear subspace to the data point x . 16 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Within the sparse coding framework For a randomly chosen sample x determine � x − C a � 2 a j 0 = arg min subject to � a � 0 = K 2 a and a bag of further good solutions. Sort the solutions according to the obtained reconstruction error: � x − C a j 0 � ≤ � x − C a j 1 � ≤ · · · ≤ � x − C a j l � ≤ · · · ≤ � x − C a j L − 1 � Update the dictionary by soft-competitive stochastic gradient descent: L e − l � λ t ( x − C a j l ) a T ∆ C = α t j l l =0 17 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Bag of Pursuits (BOP) For finding a bag of good solutions we developed the so-called ” bag of pursuits (BOP)”which is derived from Optimized Orthogonal Matching Pursuit provides a set of good choices for a with � a � 0 = K instead of a single solution expands the set of solutions in a tree-like fashion and can be directly combined with the Neural-Gas-like stochastic gradient descent for learning dictionaries. 18 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Bag of Pursuits (BOP) Dictionary R = ( r 1 , . . . , r 5 ) = D , � r i � = 1 sort according to ( r T i ǫ ) 2 Residual ǫ = y 5 1 4 2 3 5 ǫ ) ǫ ǫ = ǫ − ( r T r i = r i − ( r T 5 r i ) r i ǫ = ǫ − ( r T r i = r i − ( r T 1 ǫ ) ǫ 1 r i ) r i sort according to ( r T i ǫ ) 2 sort according to ( r T i ǫ ) 2 2 4 3 1 2 4 3 5 ǫ = ǫ − ( r T r i = r i − ( r T r i = r i − ( r T 2 r i ) r i ǫ ) ǫ r i ǫ = ǫ − ( r T T ) 2 ǫ ) ǫ 2 r r i ( T − 2 � ǫ � ≤ δ ⇒ STOP r ( 4 ǫ ) ǫ ǫ − 4 r i ) r i = i r ǫ = r i sort according to ( r T i ǫ ) 2 sort according to ( r T i ǫ ) 2 3 4 1 2 3 1 r i = r i − ( r T r i = r i − ( r T 3 ) r i 2 r i ) r i ǫ = ǫ − ( r T ǫ = ǫ − ( r T 3 ǫ ) ǫ 2 ǫ ) ǫ � ǫ � ≤ δ ⇒ STOP � ǫ � ≤ δ ⇒ STOP 19 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Synthetical experiments Do we really find the“correct”dictionary? Generate synthetical dictionaries C true ∈ R 20 × 50 and data x 1 , . . . , x 1500 ∈ R 20 that are linear combinations of C true : x i = C true b i . Each b i has k non-zero entries. The positions of the non-zero entries are chosen according to three different scenarios. 20 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding

Synthetical experiments Scenarios Random dictionary elements Chose uniformly k different dictionary elements Independent subspaces Define ⌊ 50 / k ⌋ disjoint groups of k dictionary elements Uniformly chose one of the groups Dependent subspaces Uniformly select k − 1 dictionary elements. Use 50 − k + 1 groups of dictionary elements where each group consists of the k − 1 selected dictionary elements plus one further dictionary element. 21 / 27 Thomas Martinetz Bag of Pursuits and Neural Gas for Improved Sparse Coding