Tensors Lek-Heng Lim Statistics Department Retreat October 27, - PowerPoint PPT Presentation

Tensors Lek-Heng Lim Statistics Department Retreat October 27, 2012 Thanks: NSF DMS 1209136 and DMS 1057064 L.-H. Lim (Stat Retreat) Tensors October 27, 2012 1 / 20

tensors on one foot a tensor is a multilinear functional f : V 1 × · · · × V d → C if we give f coordinates, get hypermatrix A = ( a j 1 ··· j d ) ∈ C n 1 ×···× n d where n 1 = dim V 1 , . . . , n d = dim V d d -dimensional hypermatrix represents d -tensor the same way matrix represents 2-tensor (i.e. linear operators, bilinear forms, bivectors) for more info: ◮ P. McCullagh, Tensor Methods in Statistics , Chapman and Hall, London, 1987. ◮ plug: L.-H. Lim, “Tensors,” in L. Hogben (Ed.), Handbook of Linear Algebra , 2nd Ed., CRC Press, Boca Raton, FL, 2013. L.-H. Lim (Stat Retreat) Tensors October 27, 2012 2 / 20

where do we find tensors? higher-order derivatives ∇ f ( x ) ∈ R n , ∇ 2 f ( x ) ∈ R n × n , f ( x ) ∈ R , ∇ 3 f ( x ) ∈ R n × n × n , ∇ 4 f ( x ) ∈ R n × n × n × n , . . . multivariate moments and cumulants [Fisher-Wishart, 1932]: m i | α | κ α ( x ) t α � log E(exp( i � t , x � )) ≈ α ! . | α | =1 coefficients are symmetric tensors: ( κ α ( x )) | α | =1 ∈ C p , ( κ α ( x )) | α | =2 ∈ C p × p , ( κ α ( x )) | α | =3 ∈ C p × p × p , ( κ α ( x )) | α | =4 ∈ C p × p × p × p , . . . L.-H. Lim (Stat Retreat) Tensors October 27, 2012 3 / 20

where do we find tensors? quantum mechanics ◮ H 1 , . . . , H k state spaces, state space of unified system is H ⊆ H 1 ⊗ · · · ⊗ H k ◮ H contains factorizable states ψ 1 ⊗ · · · ⊗ ψ k but also mixed states αψ 1 ⊗ · · · ⊗ ψ k + · · · + βϕ 1 ⊗ · · · ⊗ ϕ k ◮ H j : H j → H j Hamiltonian of j th system and I identity operator H 1 ⊗ I ⊗ · · · ⊗ I + I ⊗ H 2 ⊗ · · · ⊗ I + · · · + I ⊗ · · · ⊗ I ⊗ H k Hamiltonian of unified system provided systems do not interact self-concordance in convex optimization ∇ 3 f ( x ) ⊗ ∇ 3 f ( x ) � 4 ∇ 2 f ( x ) ⊗ ∇ 2 f ( x ) ⊗ ∇ 2 f ( x ) L.-H. Lim (Stat Retreat) Tensors October 27, 2012 4 / 20

what can we do with a single tensor? rank hyperdeterminant various decompositions system of multilinear equations multilinear programming multilinear least squares eigenvalues and eigenvectors singular values and singular vectors Gaussian elimination and QR factorization nonnegative tensors and Perron-Frobenius theory spectral, operator, H¨ older, Schatten, Ky Fan norms symmetric positive definite tensors and Cholesky decomposition linear preservers of rank, hyperdeterminant, singular, and eigenvalues L.-H. Lim (Stat Retreat) Tensors October 27, 2012 5 / 20

why study tensors? a rich source of new problems ◮ hypermatrix analogues of matrix notions ◮ problems trivial for matrices become non-trivial a rich source of tools for known applications ◮ quantum systems ◮ holographic algorithms ◮ algebraic complexity of matrix multiplication and inversion a rich source of tools for new applications ◮ causal inference ◮ phylogenetics inference ◮ higher order optimization theory ◮ principal components of higher order moments and cumulants ◮ spectral hypergraph theory ◮ encoding NP-hard and #P-hard problems ◮ multiarray signal processing ◮ diffusion MRI imaging caveat: there will be obstacles L.-H. Lim (Stat Retreat) Tensors October 27, 2012 6 / 20

tensor rank rank of A ∈ C l × m × n [Hitchcock, 1927] is � r � A = � � � rank( A ) := min i =1 σ i u i ⊗ v i ⊗ w i r computational complexity: Strassen matrix multiplication/inversion � rank ⊗ �� n = O ( n ω ) � � � � inf ω i , j , k =1 ϕ ik ⊗ ϕ kj ⊗ E ij = 2? quantum computing: algebraic measure of entanglement | GHZ � = | 0 � ⊗ | 0 � ⊗ | 0 � + | 1 � ⊗ | 1 � ⊗ | 1 � ∈ C 2 × 2 × 2 machine learning: na¨ ıve Bayes model � Pr( x , y , z ) = h Pr( h ) Pr( x | h ) Pr( y | h ) Pr( z | h ) H ◦ • • • X Y Z L.-H. Lim (Stat Retreat) Tensors October 27, 2012 7 / 20

example: phylogenetic invariants Markov model for evolution of 3-taxon tree [Allman-Rhodes, 2006] probability distribution given by 4 × 4 × 4 table with model P = π A ρ A ⊗ σ A ⊗ θ A + π C ρ C ⊗ σ C ⊗ θ C + π G ρ G ⊗ σ G ⊗ θ G + π T ρ T ⊗ σ T ⊗ θ T for i , j , k ∈ { A , C , G , T } , p ijk = π A ρ Ai σ Aj θ Ak + π C ρ Ci σ Cj θ Ck + π G ρ Gi σ Gj θ Gk + π T ρ Ti σ Tj θ Tk L.-H. Lim (Stat Retreat) Tensors October 27, 2012 8 / 20

multilinear systems and hyperdeterminants hyperdeterminant of A = ( a ijk ) ∈ R 2 × 2 × 2 [Cayley, 1845] is Det 2 , 2 , 2 ( A ) = 1 � �� a 000 � � a 100 �� a 010 a 110 det + 4 a 001 a 011 a 101 a 111 ��� 2 �� a 000 � � a 100 a 010 a 110 − det − a 001 a 011 a 101 a 111 � a 000 � � a 100 � a 010 a 110 − 4 det det a 001 a 011 a 101 a 111 a result that parallels the matrix case: system of bilinear equations a 000 x 0 y 0 + a 010 x 0 y 1 + a 100 x 1 y 0 + a 110 x 1 y 1 = 0 , a 001 x 0 y 0 + a 011 x 0 y 1 + a 101 x 1 y 0 + a 111 x 1 y 1 = 0 , a 000 x 0 z 0 + a 001 x 0 z 1 + a 100 x 1 z 0 + a 101 x 1 z 1 = 0 , a 010 x 0 z 0 + a 011 x 0 z 1 + a 110 x 1 z 0 + a 111 x 1 z 1 = 0 , a 000 y 0 z 0 + a 001 y 0 z 1 + a 010 y 1 z 0 + a 011 y 1 z 1 = 0 , a 100 y 0 z 0 + a 101 y 0 z 1 + a 110 y 1 z 0 + a 111 y 1 z 1 = 0 , has non-trivial solution iff Det 2 , 2 , 2 ( A ) = 0 L.-H. Lim (Stat Retreat) Tensors October 27, 2012 9 / 20

eigenvalues and singular values of tensors eigenvalues and singular values are Lagrange multipliers eigenvalues/vectors of S = ( s ijk ) ∈ S 3 ( C n ): cubic Rayleigh quotient [LHL, 2005; Qi, 2005] � n S ( x , x , x ) = i , j , k =1 s ijk x i x j x k constrained to unit ℓ 3 -sphere � x � 3 = 1 singular values/vectors of A = ( a ijk ) ∈ C l × m × n : trilinear Rayleigh quotient [LHL, 2005] � l , m , n A ( x , y , z ) = i , j , k =1 a ijk x i y j z k constrained to product of unit ℓ 3 -spheres � x � 3 = � y � 3 = � z � 3 = 1 Perron-Frobenius theorem for nonnegative tensors [LHL, 2005], [Chang-Pearson-Zhang, 2010], [Friedland-Gaubert-Han, 2012] L.-H. Lim (Stat Retreat) Tensors October 27, 2012 10 / 20

tensor norms operator norm of A ∈ C l × m × n | A ( x , y , z ) | � A � 2 , 2 , 2 = max � x �� y �� z � = σ max ( A ) x � =0 , y � =0 , z � =0 i.e. equals largest singular value of A Schatten and Ky Fan norms [LHL-Comon, 2012] i =1 | λ i | p � 1 / p � ��� r � r � � A � ∗ , p := inf � A = i =1 λ i u i ⊗ v i ⊗ w i , � � � u i � = � v i � = � w i � = 1 , r ∈ N one interesting property [LHL-Comon, 2012] � A � ∗ , 1 ≤ rank( A ) � A � ∗ , ∞ analogue of � v � 1 ≤ � v � 0 � v � ∞ and � M � ∗ ≤ rank( M ) � M � 2 for v ∈ C n and M ∈ C m × n L.-H. Lim (Stat Retreat) Tensors October 27, 2012 11 / 20

most tensor problems are NP-hard NP-Hard some have no FPTAS some are NP-hard even to Tensor Problems approximate some are #P-hard NP-Complete some are undecidable C.J. Hillar and L.-H. Lim, “Most tensor problems are NP NP hard,” J. Assoc. Comput. Mach. , to appear. P Matrix Problems L.-H. Lim (Stat Retreat) Tensors October 27, 2012 12 / 20

3-coloring encoded as tensor problem 1 2 1 2 3-colorings of left graph can be encoded as nonzero real solutions to following square set of n = 35 4 3 4 3 quadratic polynomials in 35 real unknowns a i , b i , c i , d i ( i = 1 , . . . , 4), u , w i ( i = 1 , . . . , 18): a 1 c 1 − b 1 d 1 − u 2 , b 1 c 1 + a 1 d 1 , c 1 u − a 2 1 + b 2 1 , d 1 u − 2 a 1 b 1 , a 1 u − c 2 1 + d 2 1 , b 1 u − 2 d 1 c 1 , a 2 c 2 − b 2 d 2 − u 2 , b 2 c 2 + a 2 d 2 , c 2 u − a 2 2 + b 2 2 , d 2 u − 2 a 2 b 2 , a 2 u − c 2 2 + d 2 2 , b 2 u − 2 d 2 c 2 , a 3 c 3 − b 3 d 3 − u 2 , b 3 c 3 + a 3 d 3 , c 3 u − a 2 3 + b 2 3 , d 3 u − 2 a 3 b 3 , a 3 u − c 2 3 + d 2 3 , b 3 u − 2 d 3 c 3 , a 4 c 4 − b 4 d 4 − u 2 , b 4 c 4 + a 4 d 4 , c 4 u − a 2 4 + b 2 4 , d 4 u − 2 a 4 b 4 , a 4 u − c 2 4 + d 2 4 , b 4 u − 2 d 4 c 4 , a 2 1 − b 2 1 + a 1 a 3 − b 1 b 3 + a 2 3 − b 2 3 , a 2 1 − b 2 1 + a 1 a 4 − b 1 b 4 + a 2 4 − b 2 4 , a 2 1 − b 2 1 + a 1 a 2 − b 1 b 2 + a 2 2 − b 2 2 , a 2 2 − b 2 2 + a 2 a 3 − b 2 b 3 + a 2 3 − b 2 3 , a 2 3 − b 2 3 + a 3 a 4 − b 3 b 4 + a 2 4 − b 2 4 , 2 a 1 b 1 + a 1 b 2 + a 2 b 1 + 2 a 2 b 2 , 2 a 2 b 2 + a 2 b 3 + a 3 b 2 + 2 a 3 b 3 , 2 a 1 b 1 + a 1 b 3 + a 2 b 1 + 2 a 3 b 3 , 2 a 1 b 1 + a 1 b 4 + a 4 b 1 + 2 a 4 b 4 , 2 a 3 b 3 + a 3 b 4 + a 4 b 3 + 2 a 4 b 4 , w 2 1 + w 2 2 + · · · + w 2 17 + w 2 18 equivalent to checking if bilinear system has non-trivial solution: y ⊤ A k z = 0 , x ⊤ B k z = 0 , x ⊤ C k y = 0 , k = 1 , . . . , n L.-H. Lim (Stat Retreat) Tensors October 27, 2012 13 / 20

spectral hypergraph theory G = ( V , E ) is 3- hypergraph , V vertices, E hyperedges � 1 [ i , j , k ] ∈ E adjacency hypermatrix A ∈ C n × n × n , a ijk = 0 otherwise Lemma (L, 2007) G m-regular 3 -hypergraph and A adjacency hypermatrix. Then 1 m is an eigenvalue of A 2 if λ is an eigenvalue of A, then | λ | ≤ m 3 λ has multiplicity 1 if and only if G is connected Lemma (L, 2007) G connected m-regular k-partite k-hypergraph on n vertices. Then 1 k ≡ 1 mod 4 , eigenvalue of A occurs with multiplicity a multiple of k 2 k ≡ 3 mod 4 , spectrum of A symmetric, ie. λ is eigenvalue iff − λ is L.-H. Lim (Stat Retreat) Tensors October 27, 2012 14 / 20