Latent Factor Analysis of Gaussian Distributions under Graphical - PowerPoint PPT Presentation

Latent Factor Analysis of Gaussian Distributions under Graphical Constraints Md Mahmudul Hasan, Shuangqing Wei, Ali Moharrer School of Electrical Engineering and Computer Science, Louisiana State University Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 1 / 23

Outline Motivation 1 Introduction 2 Main results 3 Building a Gaussian Tree 4 Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 2 / 23

Motivation Outline Motivation 1 Introduction 2 Main results 3 Building a Gaussian Tree 4 Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 3 / 23

Motivation Motivation • Here the primary aim is to do dimension reduction using rank of a matrix as the object function, which is hard to achieve. Hence we use trace as the object function, which is effectively almost as good as rank minimization. • Unlike traditional ways to do factor analysis i.e. numerically, providing algorithms or providing convergence proof, we are trying to do factor analysis under graphical constrainsts. Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 4 / 23

Introduction Outline Motivation 1 Introduction 2 Main results 3 Building a Gaussian Tree 4 Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 5 / 23

Introduction Definitions The well-known factor analytic decomposition of an nxn population covariance matrix Σ , Σ = (Σ − D ) + D (1) • MTFA- matrix Σ − D is Gramian and matrix D is diagonal • CMTFA- matrix Σ − D and matrix D is both are Gramian. Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 6 / 23

Introduction A brief on CMTFA CMTFA seeks a minimum trace Σ t that solves the following decomposition problem. Σ x = Σ t + D (2) such that, Σ t is low rank (rank < n ), D is diagonal, both Σ t and D are Gramian matrices. It was shown in [1] that, the above decomposition problem is equivalent to solving the following convex optimization problem. min − tr ( D ) D s.t. − λ min ( D ) ≤ 0 and − D i,i ≤ 0 , i = 1 , . . . , n (3) where, λ min ( D ) is the minimum eigenvalue of the matrix D . 1 Giacomo Della Riccia and Alexander Shapiro, ”Minimum Rank and Minimum Trace of Covariance Matrices”, Psychometrika, vol. 47, No. 4, December, 1982. Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 7 / 23

Introduction A brief on CMTFA (cont’d) The following Theorem given in the same paper, sets the ground rules for a matrix D ∗ to be the solution for the optimization problem given by (3). Theorem The matrix D ∗ is a solution of the CMTFA problem if and only if D ∗ i,i ≥ 0 , 1 ≤ i ≤ n , λ min (Σ x − D ∗ ) = 0 , and there exists n × r matrix T such that � t ∗ ,i ∈ N (Σ x − D ∗ ) , i = 1 , ...., r and the following holds, r � � t 2 � µ j � � 1 = ∗ ,i − ξ j (4) i =1 j ∈ I ( D ∗ ) where r ≤ n indicating the number of columns of the matrix T , I ( D ∗ ) = { i : D ∗ j ∈ I ( D ∗ ) } are non-negative numbers and i,i = 0 , 1 ≤ i ≤ n } , { µ j , ξ j , j ∈ I ( D ∗ ) } are column vectors in R n with all the components equal to 0 except for { � the j th component which is equal to 1 . Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 8 / 23

Introduction Problem Statement Our goal is to characterize the solution space of CMTFA, when CMTFA is applied to a specially generated Σ x i.e. generated from the following model.  x 1   α 1   z 1  . . . .  = . � � + . (5) y       . . .      x n α n z n where, y ∼ N (0 , 1) , 0 ≤ | α j | ≤ 1 , j = 1 , 2 , . . . , n . and { z i } are independent Gausian random varables with z i ∼ N (0 , 1 − α 2 i ) The above model makes the following star topology with y being the latent variable. Figure: A star topology generative model. Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 9 / 23

Introduction State of the Art In [2], a necessary and sufficient condition was found on the subspace of Σ x for MTFA solution of Σ x to recover a star structure, when Σ x is equipped with a latent star graphical constraint. The main differences between their work and ours are, • We found the same condtion for CMTFA as they did for MTFA. • Even more importantly, we also characterized the solution for the case when CMTFA fails to recover a star structure. • We characterized the solution for all possible situations. 2 J. SAUNDERSON, V. CHANDRASEKARAN, P. A. PARRILO , AND A. S. WILLSKY, ”DIAGONAL AND LOW-RANK MATRIX DECOMPOSITIONS, CORRELATION MATRICES, AND ELLIPSOID FITTING” SIAM J. MATRIX ANAL. APPL., vol. 33, no. 4, pp. 1395-1416, 2015. Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 10 / 23

Introduction Our contributions α = [ α 1 , α 2 , α 3 , ....., α n ] ′ , and without the loss of generality we have We define, vector � assumed | α 1 | ≥ | α 2 | ≥ · · · ≥ | α n | . We apply CMTFA to the following Σ x , aiming to characterize the solution space. 1  . . . α 1 α n  . . ... . . Σ x = (6)   . .   1 α n α 1 . . . Our contributions can be summarized by the following two theorems. Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 11 / 23

Main results Outline Motivation 1 Introduction 2 Main results 3 Building a Gaussian Tree 4 Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 12 / 23

Main results Theorem α is non-dominant, i.e. | α 1 | ≤ � n i =2 | α i | . CMTFA solution of Σ x is Σ t,ND if and only if, � α 2  α 1 α 2 . . . α 1 α n  1 α 2 α 2 α 1 . . . α 2 α n 2   Σ t,ND = (7)  . . .  ... . . .   . . .   α 2 α n α 1 α n α 2 . . . n Theorem α is dominant, i.e. | α 1 | > � n CMTFA solution of Σ x is Σ t,DM if and only if, � i =2 | α i | . | α 1 | ( � n i =2 | α i | )  α 1 α 2 . . . α 1 α n  | α 2 | ( | α 1 | − � i � =1 , 2 | α i | ) α 2 α 1 . . . α 2 α n   Σ t,DM =  . . .  ... . . .   . . .   α n α 1 α 1 α 2 . . . | α n | ( | α 1 | − � i � =1 ,n | α i | ) (8) Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 13 / 23

Main results Proof of the non-dominant case The following Lemma will help us prove our first theorem. Lemma α given by (9) is a necessary condition for the existence of such Non-dominance of vector � t j, ∗ || 2 = 1 , n × r matrix T that � 1 ≤ i ≤ r and || � t ∗ ,i ∈ N (Σ t,ND ) , 1 ≤ j ≤ n . n � | α 1 | ≤ | α i | (9) i =2 Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 14 / 23

Main results Proof of the non-dominant case (cont’d) Referring to the necessary and sufficient condtion for CMTFA solution. • Σ t,ND is rank 1 , its minimum eigenvalue is 0 . • We only need to show the existance of such n × r matrix T that t j, ∗ || 2 = 1 , � 1 ≤ i ≤ r and || � t ∗ ,i ∈ N (Σ t,ND ) , 1 ≤ j ≤ n where 1 ≤ r ≤ n . • the Lemma has already stated that, for the existence of such T non-dominance of � α is a necessary condition. • We next show that for the existence of such a T , non-dominance of � α is also a sufficient condition. Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 15 / 23

Main results Proof of the non-dominant case (cont’d) It is straightforward to find the following basis vectors for the null space of Σ t,ND , − α 2 − α 3 − α n       α 1 α 1 α 1 1 0 0        0   1   0  v 1 = v 2 = v n − 1 = (10) � ,� , . . . , �        .   .   .  . . .       . . .       0 0 1 v n − 1 , � n − 1 v i ] , where c i ∈ { 1 , − 1 } , We define, V = [ � v 1 , . . .� i =1 c i +1 � i = 2 , . . . , n . Let, T n × n = V n × n · B n × n , where B n × n is a diagonal matrix. We have, TT T = V BB T V T = V βV T (11) The diagonal matrix β = BB T can have only non-negative entries. Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 16 / 23

Main results Proof of non-dominant case (cont’d) t j, ∗ || 2 = 1 , Because of || � 1 ≤ j ≤ n condition on T , we get the following n equations, � n � 2 n − 1 α 2 c j α j � i +1 � β ii + β nn = 1 (12) α 2 α 1 1 i =1 j =2 β ii + c 2 i = 1 , . . . , n − 1 i +1 β nn = 1 , (13) If we select c i α i = | α i | , i = 2 , . . . , n and solve the above equations under such selections, we get β ii ≥ 0 , 1 ≤ i ≤ n . That completes the proof of our first Theorem . Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 17 / 23

Main results Proof of the dominant case The following two Lemmas play pivotal role in proving the second theorem we proposed. Lemma Σ t,DM is a rank n − 1 matrix. Lemma There exists a column vector Φ = [Φ 1 , Φ 2 , ...., Φ n ] ′ such that Σ t,DM Φ = 0 , where Φ i ∈ {− 1 , 1 } , 1 ≤ i ≤ n . Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 18 / 23