Computational Problems in Tensors Shmuel Friedland Univ. Illinois - PowerPoint PPT Presentation

Computational Problems in Tensors Shmuel Friedland Univ. Illinois at Chicago ICERM Topical workshop: Computational Nonlinear Algebra ICERM June 5, 2014 ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Overview Uniqueness of best approximation Primer on tensors Best rank one approximation of tensors Number of critical points Numerical methods for best rank one approximation Compressive sensing of sparse matrices and tensors ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

The approximation problem ν : R n → [ 0 , ∞ ) a norm on R n C ⊂ R n a closed subset, Problem: approximate a given vector x ∈ R n by a point y ∈ C : dist ν ( x , C ) := min { ν ( x − y ) , y ∈ C } y ⋆ ∈ C is called a best ν -( C )approximation of x if ν ( x − y ⋆ ) = dist ν ( x , C ) � · � the Euclidean norm on R n , dist ( x , C ) = dist �·� ( x , C ) . We call a best � · � -approximation briefly a best ( C )-approximation Main Theoretical Result: In most of applicable cases a best approximation is unique outside a corresponding variety Example: ν (( x 1 , x 2 )) = | x 1 | + | x 2 | , C := { ( t , t ) , t ∈ R } . For each ( x 1 , x 2 ) �∈ C a best approximation is not unique ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Uniqueness of ν -approxim. in semi-algebraic setting Thm F-Stawiska: Let C ⊂ R n semi-algebraic, ν semi-algebraic strictly convex norm Then the set of all points x ∈ R n \ C , denoted by S ( C ) , where ν -approximation to x in C is not unique is a semi-algebraic set which does not contain an open set. In particular S ( C ) is contained in some hypersurface H ⊂ R n . Def: S ⊂ R n is semi-algebraic if it is a finite union of basic semi-algebraic sets : p i ( x ) = 0 , i ∈ { 1 , ..., λ } , q j ( x ) > 0 , j ∈ { 1 , ..., λ ′ } f : R n → R semi-algebraic if G ( f ) = { ( x , f ( x )) : x ∈ R n } semi-algebraic ℓ p norms are semi-algebraic if p ≥ 1 is rational ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Numerical challenges Most numerical methods for finding best approximation are local Usually they will converge to a critical point or at best to a local minimum In many cases the number of critical points is exponential in n How far our minimal numerical solution is from a best approximation? Give a lower bound for best approximation Give a fast approximation for big scale problems We will address these problems for tensors ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Primer on tensors: I d -mode tensor T = [ t i 1 ,..., i d ] ∈ F n 1 × ... × n d , i j ∈ [ n j ] := { 1 , . . . , n j } , j ∈ [ d ] d = 1 vector: x ; d = 2 matrix A = [ a ij ] rank one tensor T = [ x i 1 , 1 x i 2 , 2 · · · x i d , d ] = x 1 ⊗ x 2 · · · ⊗ x d = ⊗ d j = 1 x j � = 0 T = � r rank of tensor rank T := min { r : k = 1 ⊗ d j = 1 x j , k } It is an NP-hard problem to determine rank T for d ≥ 3. border rank brank T the minimal r s.t. T is limit of tensors of rank r brank T < rank T for some d ≥ 3 mode tensors (Nongeneric case) n k × N Unfolding tensor in mode k : T k ( T ) ∈ F nk , N = n 1 · · · n d grouping indexes ( i 1 , . . . , i d ) into two groups i k and the rest rank T k ( T ) ≤ brank T ≤ rank T for each k ∈ [ d ] R ( r 1 , . . . , r d ) ⊂ F n 1 × ... × n d variety of all tensors rank T k ( T ) ≤ r k , k ∈ [ d ] j = 1 F n j - Segre variety (variety of rank one tensors) R ( 1 , . . . , 1 ) = ⊗ d ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Primer on tensors: II Contraction of tensors T = [ t i 1 ,..., i d ] , X = [ x i k 1 ,..., i kl ] , { k 1 , . . . , k l } ⊂ [ d ] T × X := � i k 1 ∈ [ n k 1 ,..., i kl ∈ [ n kl ] t i 1 ,..., i l x i k 1 ,..., i kl Symmetric d -mode tensor S ∈ S ( F n , d ) : n 1 = · · · = n d = n , entries s i 1 ,..., i d are symmetric in all indexes rank one symmetric tensor ⊗ d x := x ⊗ · · · ⊗ x � = 0 S = � r symmetric rank (Waring rank) srank S := min { r , k = 1 ⊗ d x k } Conjecture (P . Comon 2009) srank S = rank S for S ∈ S ( C n , d ) Some cases proven by Comon-Golub-Lim-Mourrain 2008 For finite fields ∃S s.t. srank S not defined F-Stawiska ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Examples of approximation problems R N := R n 1 × ... × n d - and C : 1. Tensors of border rank k -at most, denoted as C k 2. C ( r ) := R ( r 1 , . . . , r d ) ν ( · ) = � · � - Hilbert-Schmidt norm (other norms sometime) r 1 = · · · = r d = r and S ∈ S ( R n , d ) n 1 = · · · = n d = n , Problem: Can a best approximation can be chosen symmetric? For matrices: yes For k = 1: yes - Banach’s theorem 1938 For some range of k : yes for some open semi-algebraic set of S ∈ S ( R n , d ) - F - Stawiska ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Best rank one approximation of 3-tensors R m × n × l IPS: �A , B� = � m , n , l � i = j = k a i , j , k b i , j , k , �T � = �T , T � � x ⊗ y ⊗ z , u ⊗ v ⊗ w � = ( u ⊤ x )( v ⊤ y )( w ⊤ z ) X subspace of R m × n × l , X 1 , . . . , X d an orthonormal basis of X � P X ( T ) � 2 = � d P X ( T ) = � d i = 1 �T , X i � 2 i = 1 �T , X i �X i , �T � 2 = � P X ( T ) � 2 + �T − P X ( T ) � 2 Best rank one approximation of T : min x , y , z �T − x ⊗ y ⊗ z � = min � x � = � y � = � z � = 1 , a �T − a x ⊗ y ⊗ z � � m , n , l Equivalent: �T � ∞ := max � x � = � y � = � z � = 1 i = j = k t i , j , k x i y j z k Hillar-Lim 2013: computation of �T � ∞ NP-hard Lagrange multipliers: T × y ⊗ z := � j = k = 1 t i , j , k y j z k = λ x T × x ⊗ z = λ y , T × x ⊗ y = λ z λ singular value, x , y , z singular vectors Lim 2005 ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Number of singular values of 3-tensor: I c ( m , n , l ) - # distinct singular values for a generic T ∈ C m × n × l in pol. (( t 2 + t 3 ) m − t m (( t 1 + t 3 ) n − t n is coefficient of t m − 1 t n − 1 t l − 1 1 ) 2 ) (( t 1 + t 2 ) l − t 3 ) 1 2 3 ( t 2 + t 3 − t 1 ) ( t 1 + t 3 − t 2 ) ( t 1 + t 2 − t 3 ) Recall x m − y m = x m − 1 + x m − 2 y + · · · + xy m − 2 + y m − 1 x − y d 1 , d 2 , d 3 c ( d 1 , d 2 , d 3 ) 2 , 2 , 2 6 2 , 2 , n 8 n ≥ 3 2 , 3 , 3 15 2 , 3 , n 18 n ≥ 4 2 , 4 , 4 28 2 , 4 , n 32 n ≥ 5 2 , 5 , 5 45 2 , 5 , n 50 n ≥ 6 2 m 2 2 , m , m + 1 Table : Values of c ( d 1 , d 2 , d 3 ) ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Number of singular values of 3-tensor: II d 1 , d 2 , d 3 c ( d 1 , d 2 , d 3 ) 3 , 3 , 3 37 3 , 3 , 4 55 3 , 3 , n 61 n ≥ 5 3 , 4 , 4 104 3 , 4 , 5 138 3 , 4 , n 148 n ≥ 6 3 , 5 , 5 225 3 , 5 , 6 280 3 , 5 , n 295 n ≥ 7 3 m 3 − 2 m 2 + 7 8 3 , m , m + 2 3 m Table : Values of c ( d 1 , d 2 , d 3 ) ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Number of singular values of 3-tensor: III d 1 , d 2 , d 3 c ( d 1 , d 2 , d 3 ) 4 , 4 , 4 240 4 , 4 , 5 380 4 , 4 , 6 460 4 , 4 , n 480 n ≥ 7 4 , 5 , 5 725 4 , 5 , 6 1030 4 , 5 , 7 1185 4 , 4 , 4 240 4 , 4 , 5 380 4 , 4 , 6 460 4 , 4 , n 480 n ≥ 7 Table : Values of c ( d 1 , d 2 , d 3 ) ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Number of singular values of 3-tensor: IV d 1 , d 2 , d 3 c ( d 1 , d 2 , d 3 ) 4 , 5 , 5 725 4 , 5 , 6 1030 4 , 5 , 7 1185 4 , 4 , 4 240 4 , 4 , 5 380 4 , 4 , 6 460 4 , 4 , n 480 n ≥ 7 4 , 5 , 5 725 4 , 5 , 6 1030 4 , 5 , 7 1185 4 , 5 , 7 1185 4 , 5 , n 1220 n ≥ 8 Table : Values of c ( d 1 , d 2 , d 3 ) ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53

Number of singular values of 3-tensor: V d 1 , d 2 , d 3 c ( d 1 , d 2 , d 3 ) 5 , 5 , 5 1621 5 , 5 , 6 2671 5 , 5 , 7 3461 5 , 5 , 8 3811 5 , 5 , n 3881 n ≥ 9 Table : Values of c ( d 1 , d 2 , d 3 ) Friedland-Ottaviani 2014 ICERM Topical workshop: Computational Nonlinear Shmuel Friedland Univ. Illinois at Chicago Computational Problems in Tensors / 53