Singular Values of Tensors Harm Derksen University of Michigan - PowerPoint PPT Presentation

Singular Values of Tensors Harm Derksen University of Michigan CUNY/NYU February 15, 2019 Harm Derksen Singular Values of Tensors

Tensor Rank F a field, F = R or F = C V ( i ) ∼ = F n i for i = 1 , 2 , . . . , d V = V (1) ⊗ V (2) ⊗ · · · ⊗ V ( d ) ∼ = F n 1 ×···× n d tensor product space Definition A simple tensor is a tensor of the form v (1) ⊗ v (2) ⊗ · · · ⊗ v ( d ) ( v ( i ) ∈ V ( i ) ). Harm Derksen Singular Values of Tensors

Tensor Rank F a field, F = R or F = C V ( i ) ∼ = F n i for i = 1 , 2 , . . . , d V = V (1) ⊗ V (2) ⊗ · · · ⊗ V ( d ) ∼ = F n 1 ×···× n d tensor product space Definition A simple tensor is a tensor of the form v (1) ⊗ v (2) ⊗ · · · ⊗ v ( d ) ( v ( i ) ∈ V ( i ) ). Definition (tensor rank) The rank of T is the smallest positive integer r such that T is the sum of r simple tensors. Harm Derksen Singular Values of Tensors

Matrix Multiplication Tensor d d d � � � e i , j ⊗ e j , k ⊗ e k , i ∈ F n × n ⊗ F n × n ⊗ F n × n M d = i =1 j =1 k =1 clearly rank( M d ) ≤ d 3 Harm Derksen Singular Values of Tensors

Matrix Multiplication Tensor d d d � � � e i , j ⊗ e j , k ⊗ e k , i ∈ F n × n ⊗ F n × n ⊗ F n × n M d = i =1 j =1 k =1 clearly rank( M d ) ≤ d 3 Theorem (Strassen) if rank( M d ) = s then complexity of n × n matrix multiplication is O ( n log d ( s ) ) (the standard algorithm is O ( n 3 ) ) Harm Derksen Singular Values of Tensors

Matrix Multiplication Tensor d d d � � � e i , j ⊗ e j , k ⊗ e k , i ∈ F n × n ⊗ F n × n ⊗ F n × n M d = i =1 j =1 k =1 clearly rank( M d ) ≤ d 3 Theorem (Strassen) if rank( M d ) = s then complexity of n × n matrix multiplication is O ( n log d ( s ) ) (the standard algorithm is O ( n 3 ) ) Theorem (Strassen) rank( M 2 ) ≤ 7 , so complexity of n × n matrix multiplication is O ( n log 2 (7) ) = O ( n 2 . 81 ) Current record: O ( n 2 . 373 ) (Le Gall) Harm Derksen Singular Values of Tensors

The Canonical Polyadic (CP) Model aka PARAFAC, CANDECOMP Problem Given a tensor T ∈ V = V (1) ⊗ · · · ⊗ V ( d ) , write T = v 1 + v 2 + · · · + v r where v 1 , v 2 , . . . , v r are simple tensors and r is minimal. Harm Derksen Singular Values of Tensors

The Canonical Polyadic (CP) Model aka PARAFAC, CANDECOMP Problem Given a tensor T ∈ V = V (1) ⊗ · · · ⊗ V ( d ) , write T = v 1 + v 2 + · · · + v r where v 1 , v 2 , . . . , v r are simple tensors and r is minimal. ◮ the CP-decomposition is sometimes unique, not always ◮ not numerically stable: there is no upper bound for max {� v 1 � , . . . , � v r �} as a function of � T � . ◮ difficult to compute: for many tensors of interest the rank is unknown Many numerical applications in psychometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, graph analysis, neuroscience, chemometrics, etc. Harm Derksen Singular Values of Tensors

The Canonical Polyadic (CP) Model To make it more numerically stable, we could allow an error term. Problem Given a tensor T ∈ V and fixed r, write T = v 1 + v 2 + · · · + v r + E where v 1 , v 2 , . . . , v r are simple tensors and � E � is minimal. Harm Derksen Singular Values of Tensors

The Canonical Polyadic (CP) Model To make it more numerically stable, we could allow an error term. Problem Given a tensor T ∈ V and fixed r, write T = v 1 + v 2 + · · · + v r + E where v 1 , v 2 , . . . , v r are simple tensors and � E � is minimal. This optimization problem might be ill-posed. Define T = e 2 ⊗ e 1 ⊗ e 1 + e 1 ⊗ e 2 ⊗ e 1 + e 1 ⊗ e 1 ⊗ e 2 , v 1 ( t ) = t − 1 ( e 1 + te 2 ) ⊗ ( e 1 + te 2 ) ⊗ ( e 1 + te 2 ) , v 2 ( t ) = − t − 1 e 1 ⊗ e 1 ⊗ e 1 Then T = v 1 ( t ) + v 2 ( t ) + E ( t ) with � E ( t ) � = t √ 3 + t . So we can write T = v 1 + v 2 + E with � E � = ε for every ε > 0, but we cannot write T = v 1 + v 2 + E with � E � = 0. Harm Derksen Singular Values of Tensors

Motivation: Compressed Sensing and Convex Relaxation For x ∈ R n , its sparsity is measured by � x � 0 = # { i | x i � = 0 } . Problem Given A ∈ R m × n , b ∈ R m , find a solution x ∈ R n for Ax = b with � x � 0 minimal (a sparsest solution). But, � · � 0 is not convex and this optimization problem is difficult, Harm Derksen Singular Values of Tensors

Motivation: Compressed Sensing and Convex Relaxation For x ∈ R n , its sparsity is measured by � x � 0 = # { i | x i � = 0 } . Problem Given A ∈ R m × n , b ∈ R m , find a solution x ∈ R n for Ax = b with � x � 0 minimal (a sparsest solution). But, � · � 0 is not convex and this optimization problem is difficult, so instead we consider: Problem (Basis Pursuit) Given A ∈ R m × n , b ∈ R m , find a solution x ∈ R n for Ax = b with � x � 1 minimal. Basis Pursuit can be solved by linear programming and is generally fast. Under reasonable assumptions, Basis Pursuit also gives the sparsest solutions (Cand` es-Tao, Donoho). Harm Derksen Singular Values of Tensors

The Nuclear and Spectral Norms Low rank tensors are sparse in some sense, and by convex relaxation: Definition The nuclear norm � T � ⋆ is the smallest value of � r i =1 � v i � where T = � r i =1 v i and v 1 , . . . , v r are simple tensors. (well-defined) Harm Derksen Singular Values of Tensors

The Nuclear and Spectral Norms Low rank tensors are sparse in some sense, and by convex relaxation: Definition The nuclear norm � T � ⋆ is the smallest value of � r i =1 � v i � where T = � r i =1 v i and v 1 , . . . , v r are simple tensors. (well-defined) V (1) , . . . , V ( d ) , V a spaces with a positive definite bilinear/hermitian form �· , ·� Harm Derksen Singular Values of Tensors

The Nuclear and Spectral Norms Low rank tensors are sparse in some sense, and by convex relaxation: Definition The nuclear norm � T � ⋆ is the smallest value of � r i =1 � v i � where T = � r i =1 v i and v 1 , . . . , v r are simple tensors. (well-defined) V (1) , . . . , V ( d ) , V a spaces with a positive definite bilinear/hermitian form �· , ·� Definition The spectral norm is defined by � T � σ = max {|� T , v �| | v simple tensor with � v � = 1 } . The spectral norm is dual to the nuclear norm, in particular |� T , S �| ≤ � T � ⋆ � S � σ for all tensors S , T . Harm Derksen Singular Values of Tensors

The Nuclear and Spectral Norms if A ∈ R n × m = R n ⊗ R m is an n × m matrix then the tensor rank of A coincides with the matrix rank of A Harm Derksen Singular Values of Tensors

The Nuclear and Spectral Norms if A ∈ R n × m = R n ⊗ R m is an n × m matrix then the tensor rank of A coincides with the matrix rank of A If λ 1 ≥ λ 2 ≥ · · · ≥ λ r are the singular values of A , then � A � ⋆ = λ 1 + · · · + λ r , � A � σ = λ 1 (spectral/operator norm) and � λ 2 1 + · · · + λ 2 � A � = � A � 2 = � A � F = r (Euclidean/Frobenius norm) Harm Derksen Singular Values of Tensors

Example: Determinant Tensor D n = � σ ∈ S n sgn( σ ) e σ (1) ⊗ e σ (2) ⊗ · · · ⊗ e σ ( n ) ∈ C n ⊗ · · · ⊗ C n . clearly � D n � ⋆ ≤ n ! and rank( D n ) ≤ n ! Harm Derksen Singular Values of Tensors

Example: Determinant Tensor D n = � σ ∈ S n sgn( σ ) e σ (1) ⊗ e σ (2) ⊗ · · · ⊗ e σ ( n ) ∈ C n ⊗ · · · ⊗ C n . clearly � D n � ⋆ ≤ n ! and rank( D n ) ≤ n ! � D n � σ = max {| det( v 1 v 2 · · · v n ) | | � v 1 � = · · · = � v n � = 1 } = 1 � D n � ⋆ = � D n � ⋆ � D n � σ ≥ � D n , D n � = n ! . so � D n � ⋆ ≥ n ! Theorem (D.) � D n � ⋆ = n ! Harm Derksen Singular Values of Tensors

Example: Permanent Tensor P n = � σ ∈ S n e σ (1) ⊗ e σ (2) ⊗ · · · ⊗ e σ ( n ) ∈ C n ⊗ · · · ⊗ C n . � P n � σ = max {| perm( v 1 v 2 · · · v n ) | | � v 1 � = · · · = � v n � = 1 } Harm Derksen Singular Values of Tensors

Example: Permanent Tensor P n = � σ ∈ S n e σ (1) ⊗ e σ (2) ⊗ · · · ⊗ e σ ( n ) ∈ C n ⊗ · · · ⊗ C n . � P n � σ = max {| perm( v 1 v 2 · · · v n ) | | � v 1 � = · · · = � v n � = 1 } Theorem (Carlen, Lieb and Moss, 2006) max { perm( v 1 v 2 · · · v n ) | � v 1 � = · · · = � v n � = 1 } = n ! / n n / 2 n ! n n / 2 � P n � ⋆ = � P n � ⋆ � P n � σ ≥ � P n , P n � = n ! . so � P n � ⋆ ≥ n n / 2 Harm Derksen Singular Values of Tensors

Example: Permanent Tensor Theorem (Glynn 2010) 1 � � n � � ( � n i =1 δ i e i ) ⊗ · · · ⊗ ( � n P n = i =1 δ i i =1 δ i e i ) 2 n − 1 δ where δ runs over { 1 } × {− 1 , 1 } n − 1 . Harm Derksen Singular Values of Tensors

Example: Permanent Tensor Theorem (Glynn 2010) 1 � � n � � ( � n i =1 δ i e i ) ⊗ · · · ⊗ ( � n P n = i =1 δ i i =1 δ i e i ) 2 n − 1 δ where δ runs over { 1 } × {− 1 , 1 } n − 1 . � � ≤ rank( P n ) ≤ 2 n − 1 and � P n � ⋆ ≤ n n / 2 , so n In particular, ⌊ n / 2 ⌋ Theorem (D.) � P n � ⋆ = n n / 2 Harm Derksen Singular Values of Tensors