bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C A message-passing approach to low-rank matrix reconstruction and application to clustering Toshiyuki TANAKA tt@i.kyoto-u.ac.jp Graduate School of Informatics, Kyoto University 1 September, 2014 1 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Table of contents Low-rank matrix reconstruction via message passing 1 Application to clustering 2 Application to multivariate Poisson clustering 3 Conclusions 4 2 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Collaborators Ryosuke Matsushita (NTT DATA Mathematical Systems Inc., Japan) Kei Sano (Graduate School of Informatics, Kyoto University, Japan) 3 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Brief biography 1993: Graduated from Department of Electronic Engineering, Graduate School of Engineering, the University of Tokyo. 1993–2005: Department of Electronics and Information Engineering, Tokyo Metropolitan University. 2005–: Graduate School of Informatics, Kyoto University. 4 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Analysis and extensions of compressed sensing and low-rank matrix reconstruction Low-rank matrix reconstruction via message passing 1 Problem formulation Approach Application to clustering 2 Application to multivariate Poisson clustering 3 Conclusions 4 5 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Low-rank matrix reconstruction via message passing Reference: R. Matsushita and T. Tanaka, “Low-rank matrix reconstruction and clustering via approximate message passing,” in C. J. C. Burges et al. (eds.), Advances in Neural Information Processing Systems , volume 26, pages 917–925, 2013. 6 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Problem formulation Analysis and extensions of compressed sensing and low-rank matrix reconstruction Low-rank matrix reconstruction via message passing 1 Problem formulation Approach Application to clustering 2 Application to multivariate Poisson clustering 3 Conclusions 4 7 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Problem formulation Low-rank matrix reconstruction Problem formulation A 0 ∈ R M × N , rank A 0 = r ≪ min { M , N } (Low-rank). Observation noise: W = ( W ij ) ∈ R M × N , W ij ∼ N (0 , M σ 2 ). Observe (part of) A = A 0 + W → Estimate the low-rank mtx. A 0 Consider SVD A = U Σ V T of A , and let ˆ A = U Σ r V T using Σ r constructed by leaving the largest r singular values of Σ. “Nuclear-norm” minimization: ˆ � X � 1 + λ � X − A � 2 � � A = arg min F X ∈ R M × N . . . 9 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Problem formulation Shatten norms Shatten’s p -norm � 1 / p �� σ p � A � p := p ≥ 1 . , i i Rank = Number of non-zero singular values = “0-norm”. Nuclear norm = Shatten’s 1-norm. Regarded as a convex relaxation of Rank. 10 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Problem formulation Low-rank matrix reconstruction Formulation via probability model A = UV T + W , U ∈ R M × r , V T ∈ R r × N ⇒ e − ( A ij − u T i v j ) 2 / (2 M σ 2 ) � p ( A | U , V ) ∝ i , j M N � e − c ( u i ) , � e − c ( v j ) p ( U ) ∝ p ( V ) ∝ i =1 j =1 ⇒ p ( U , V | A ) ∝ p ( A | U , V ) p ( U ) p ( V ) � M � N e − ( A ij − u T i v j ) 2 / (2 M σ 2 ) � � e − c ( u i ) � e − c ( v j ) = i , j i =1 j =1 11 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Problem formulation Low-rank matrix reconstruction Formulation via probability model p ( U , V | A ) ∝ p ( A | U , V ) p ( U ) p ( V ) � M � N i v j ) 2 / (2 M σ 2 ) � e − ( A ij − u T � � e − c ( u i ) e − c ( v j ) = i , j i =1 j =1 Allows straightforward incorporation of prior knowledge on U and V . Non-negativeness (Paatero-Tapper, 1994) c u ( u i ) = 0 ( u i ≥ 0 ) , ∞ (else) Sparseness (Olshausen-Field, 1996) c u ( u i ) = � u i � 0 or � u i � 1 Generally a non-convex optimization. Hard to solve. 12 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Problem formulation Low-rank matrix reconstruction Two alternative objectives: Posterior-mean (PM) estimation: Optimal in terms of � UV T − ˆ U ˆ V T � F . ( ˆ U PM , ˆ V PM ) = E p ( U , V | A ) ( U , V ) Maximum-A-Posteriori (MAP) estimation: ( ˆ U MAP , ˆ V MAP ) = arg max U , V p ( U , V | A ) 13 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Problem formulation Low-rank matrix reconstruction PM estimation: ( ˆ U PM , ˆ V PM ) = E p ( U , V | A ) ( U , V ) MAP estimation: ( ˆ U MAP , ˆ V MAP ) = arg max U , V p ( U , V | A ) One-parameter extension p ( U , V | A ; β ) ∝ [ p ( U , V | A )] β Extended PM estimation: ( ˆ U β , ˆ V β ) = E p ( U , V | A ; β ) ( U , V ) PM estimation: ( ˆ U PM , ˆ V PM ) = ( ˆ U β , ˆ V β ) | β =1 MAP estimation: ( ˆ U MAP , ˆ V MAP ) = ( ˆ U β , ˆ V β ) | β →∞ 14 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Approach Analysis and extensions of compressed sensing and low-rank matrix reconstruction Low-rank matrix reconstruction via message passing 1 Problem formulation Approach Application to clustering 2 Application to multivariate Poisson clustering 3 Conclusions 4 15 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Approach Our approach � M � N i v j ) 2 / (2 M σ 2 ) � e − β ( A ij − u T � � e − β c ( u i ) e − β c ( v j ) p ( U , V | A ; β ) ∝ i , j i =1 j =1 Belief-propagation (BP) / Approximate message passing (AMP): Factor-graph representation. Apply BP ⇒ Message-passing algorithm. Msgs are densities. Take large-system limit ⇒ AMP algorithm. Msgs are parameters. 16 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Approach v 1 v 2 v 3 v 4 v 5 11 12 13 u 1 14 15 21 22 23 u 2 24 25 31 32 33 34 u 3 35 41 42 43 44 u 4 45 � M � N e − β ( A ij − u T i v j ) 2 / (2 M σ 2 ) � � e − β c ( u i ) � e − β c ( v j ) p ( U , V | A ; β ) ∝ i , j i =1 j =1 17 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Approach v 1 v 2 v 3 v 4 v 5 11 12 13 u 1 14 15 21 22 23 u 2 24 25 31 32 33 u 3 34 35 41 42 43 u 4 44 45 � µ i → ( i , j ) ( u i ) ∝ p ( u i ) λ ( i , l ) → i ( u i ) l � = j � i v j ) 2 / (2 M σ 2 ) µ j → ( i , j ) ( v j ) d v j e − ( A ij − u T λ ( i , j ) → i ( u i ) ∝ Msgs have functional degree of freedom. Hard to implement. 19 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Approach v 1 v 2 v 3 v 4 v 5 11 12 13 u 1 14 15 21 22 23 u 2 24 25 31 32 33 u 3 34 35 41 42 43 u 4 44 45 � µ i → ( i , j ) ( u i ) ∝ p ( u i ) λ ( i , l ) → i ( u i ) l � = j � i v j ) 2 / (2 M σ 2 ) µ j → ( i , j ) ( v j ) d v j e − ( A ij − u T λ ( i , j ) → i ( u i ) ∝ Msgs have functional degree of freedom. Hard to implement. 19 / 57
bg=white Low-rank matrix reconstruction via message passing Application to clustering Application to multivariate Poisson clustering C Approach v 1 v 2 v 3 v 4 v 5 11 12 13 u 1 14 15 21 22 23 u 2 24 25 31 32 33 u 3 34 35 41 42 43 u 4 44 45 � µ i → ( i , j ) ( u i ) ∝ p ( u i ) λ ( i , l ) → i ( u i ) l � = j � l � = j ( A il − u T i v l ) 2 / (2 M σ 2 ) � e − � � � ∝ p ( u i ) µ l → ( i , l ) ( v l ) d v l l � = j � i v j ) 2 / 2 σ 2 µ j → ( i , j ) ( v j ) d v j e − ( A ij − u T λ ( i , j ) → i ( u i ) ∝ AMP: Apply CLT and represent msgs in terms of means and covariances. 20 / 57
Recommend
More recommend
