NEW TENSOR DECOMPOSITIONS IN NUMERICAL ANALYSIS AND DATA PROCESSING - PowerPoint PPT Presentation

NEW TENSOR DECOMPOSITIONS IN NUMERICAL ANALYSIS AND DATA PROCESSING Eugene Tyrtyshnikov Institute of Numerical Mathematics of Russian Academy of Sciences eugene.tyrtyshnikov@gmail.com 11 October 2012 Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

COLLABORATION MOSCOW: I.Oseledets, D.Savostyanov S.Dolgov, V.Kazeev, O.Lebedeva, A.Setukha, S.Stavtsev, D.Zheltkov S.Goreinov, N.Zamarashkin LEIPZIG: W.Hackbusch, B.Khoromskij, R.Schneider H.-J.Flad, V.Khoromskaia, M.Espig, L.Grasedyck Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

TENSORS IN 20TH CENTURY used chiefly as desriptive tools: ◮ physics ◮ differential geometry ◮ multiplication tables in algebras ◮ applied data management ◮ chemometrics ◮ sociometrics ◮ signal/image processing ◮ many others Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

WHAT IS TENSOR Tensor = d -linear form = d -dimensional array: A = [ a i 1 i 2 ... i d ] Tensor A possesses: ◮ dimensionality (order) d = number of indices (dimensions, modes, axes, directions, ways) ◮ size n 1 × ... × n d (number of points at each dimension) Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

EXAMPLES OF PROMINENT THEORIES FOR TENSORS IN 20th CENTURY ◮ Kruskal’s theorem (1977) on essential uniqueness of canonical tensor decomposition introduced by Hitchcock (1927); ◮ canonical tensor decompositions as a base for Strassen’s method of matrix multiplication of complexity less than n 3 (1969); ◮ interrelations between tensors (especially symmetric) and polynomials as a topic in algebraic geometry. Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

BEGIN WITH 2 × 2 MATRICES The column-by-row rule for 2 × 2 matrices yields 8 mults: � a 11 a 12 � � b 11 b 12 � = a 21 a 22 b 21 b 22 � a 11 b 11 + a 12 b 21 a 11 b 12 + a 12 b 22 � a 21 b 11 + a 22 b 21 a 21 b 12 + a 22 b 22 Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

DISCOVERY BY STRASSEN Only 7 mults is enough! IMPORTANT: for block 2 × 2 matrices these are 7 mults of blocks: α 1 = ( a 11 + a 22 )( b 11 + b 22 ) α 2 = ( a 21 + a 22 ) b 11 c 11 = α 1 + α 4 − α 5 + α 7 α 3 = a 11 ( b 12 − b 22 ) c 12 = α 3 + α 5 α 4 = a 22 ( b 21 − b 11 ) c 21 = α 2 + α 4 α 5 = ( a 11 + a 12 ) b 22 c 22 = α 1 + α 3 − α 2 + α 6 α 6 = ( a 21 − a 11 )( b 11 + b 12 ) α 7 = ( a 12 − a 22 )( b 21 + b 22 ) Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

HOW A TENSOR ARISES AND HELPS n 2 n 2 � c 1 c 2 � � a 1 a 2 � � b 1 b 2 � � � = c k = h ijk a i b j c 3 c 4 a 3 a 4 b 3 b 4 i = 1 j = 1 R � h ijk = u i α v j α w k α α = 1     n 2 n 2 R � � � ⇒ c k = w k α u i α a i v j α b j     α = 1 i = 1 j = 1 Now only R mults of blocks! If n = 2 then R = 7 (Strassen, 1969). Recursion ⇒ O ( n log 2 7 ) scalar mults for any n . Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

GENERAL CASE BY RECURSION Two matrices of order n = 2 d can be multiplied with 7 d = n log 2 7 scalar multiplications and 7 n log 2 7 scalar additions/subtrations. n = 2 d n / 2 n / 2 n / 2 n / 2 n / 2 n / 2 n / 2 Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

TENSORS IN 21ST CENTURY: NUMERICAL METHODS WITH TENSORIZATION OF DATA We consider typical problems of numerical analysis (matrix computations, interpolation, optimization) under the assumption that the input, output and all intermediate data are represented by tensors with many dimensions (tens, hundreds, even thousands). Of course, it assumes a very special structure of data. But we have it in really many problems! Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

THE CURSE OF DIMENSIONALITY The main problem is that using arrays as means to introduce tensors in many dimensions is infeasible : ◮ if d = 300 and n = 2, then such an array contains 2 300 ≫ 10 83 entries Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

NEW REPRESENTATION FORMATS Canonical polyadic and Tucker decompositions are of limited use for our purposes (by different reasons). New decompositions: ◮ TT (Tensor Train) ◮ HT (Hierarchical Tucker) Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

REDUCTION OF DIMENSIONALITY i 1 i 2 i 3 i 4 i 5 i 6 i 1 i 2 i 3 i 4 i 5 i 6 i 1 i 2 i 3 i 4 i 5 i 6 i 3 i 4 i 5 i 6 Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

SCHEME FOR TT i 1 i 2 i 3 i 4 i 5 i 6 i 1 i 2 α i 3 i 4 i 5 i 6 α i 1 β i 2 αβ i 3 i 4 γ i 5 i 6 αγ i 3 δ i 4 γδ i 5 αη i 6 γη Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

SCHEME FOR HT i 1 i 2 i 3 i 4 i 5 i 6 i 1 i 2 α i 3 i 4 i 5 i 6 α i 1 β i 2 αβ i 3 i 4 γ i 5 i 6 αγ i 2 φ αβφ i 4 γδ i 5 i 6 ξ γηξ i 3 δ i 4 ψ γδψ i 5 ζ i 6 ξζ i 6 ν ξζν Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

THE BLESSING OF DIMENSIONALITY TT and HT provide new representation formats for d -tensors + algorithms with complexity linear in d . Let the amount of data be N . In numerical analysis, complexity O ( N ) is usually considered as a dream. With ultimate tensorization we go beyond the dream : since d ∼ log N , we may obtain complexity O ( log N ) . Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

BASIC TT ALGORITHMS ◮ TT rounding . Like the rounding of machine numbers. COMLEXITY = O ( dnr 3 ) . √ d − 1 · BEST ERROR . ERROR � ◮ TT interpolation . A tensor train is constructed from sufficiently few elements of the tensor, the number of them is O ( dnr 2 ) . ◮ TT quantization and wavelets . Low-dimensional → high-dimensional ⇒ algebraic wavelet tranbsforms (WTT). In matrix problems the complexity may drop from O ( N ) down to O ( log N ) . Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

SUMMATION AGREEMENT Omit the symbol of summation. Assume summation if the index in a product of quantities with indices is repeated at least twice. Equations hold for all values of other indices. Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

SKELETON DECOMPOSITION   u 1 α r A = UV ⊤ = �  � � . . . v 1 α . . . v n α  u m α α = 1 According to the summation agreement, a ( i , j ) = u ( i , α ) v ( j , α ) Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

CANONICAL AND TUCKER CANONICAL DECOMPOSITION a ( i 1 . . . i d ) = u 1 ( i 1 α ) . . . u d ( i d α ) TUCKER DECOMPOSITION a ( i 1 . . . i d ) = g ( α 1 . . . α d ) u 1 ( i 1 α 1 ) . . . u d ( i d α d ) Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

TENSOR TRAIN (TT) IN THREE DIMENSIONS a ( i 1 ; i 2 i 3 ) = g 1 ( i 1 ; α 1 ) a 1 ( α 1 ; i 2 i 3 ) a 1 ( α 1 i 2 ; i 3 ) = g 2 ( α 1 i 2 ; α 2 ) g 3 ( α 2 ; i 3 ) TENSOR TRAIN (TT) a ( i 1 i 2 i 3 ) = g 1 ( i 1 α 1 ) g 2 ( α 1 i 2 α 2 ) g 3 ( α 2 i 3 ) Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

TENSOR TRAIN (TT) IN d DIMENSIONS a ( i 1 . . . i d ) = g 1 ( i 1 α 1 ) g 2 ( α 1 i 2 α 2 ) . . . g d − 1 ( α d − 2 i d − 1 α d − 1 ) g d ( α d − 1 i d ) d � a ( i 1 . . . i d ) = g k ( α k − 1 i k α k ) k = 1 Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

KRONECKER REPRESENTATION OF TENSOR TRAINS A = G 1 α 1 ⊗ G 2 α 1 α 2 ⊗ . . . ⊗ G d − 1 α d − 2 α d − 1 ⊗ G d α d − 1 A is of size ( m 1 . . . m d ) × ( n 1 . . . n d ) . G k α k − 1 α k is of size m k × n k . Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

ADVANTAGES OF TENSOR-TRAIN REPRESENTATION The tensor is determined through d tensor carriages g k ( α k − 1 i k α k ) , each of size r k − 1 × n k × r k . If the maximal size is r × n × r , then the number of representation parameters does not exceed dnr 2 ≪ n d . Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

TENSOR TRAIN PROVIDES STRUCTURED SKELETON DECOMPOSITIONS OF UNFOLDING MATRICES A k = a ( i 1 . . . i k ; i k + 1 . . . i d ) = u k ( i 1 . . . i k ; α k ) v k ( α k ; i k + 1 . . . i d ) = U k V ⊤ k u k ( i 1 . . . i k α k ) = g 1 ( i 1 α 1 ) . . . g k ( α k − 1 i k α k ) v k ( α k i k + 1 . . . i d ) = g k + 1 ( α k i k + 1 α k + 1 ) . . . g d ( α k − 1 i d ) Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

TT RANKS ARE BOUNDED BY THE RANKS OF UNFOLDING MATRICES r k � rank A k , A k = [ a ( i 1 . . . i k ; i k + 1 . . . i d )] Equalities are always possible. Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

ORTHOGONAL TENSOR CARRIAGES A tensor carriage g ( α i β ) is called row orthogonal if its first unfolding matrix g ( α ; i β ) has orthonormal rows. A tensor carriage g ( α i β ) is called column orthogonal if its second unfolding matrix g ( α i ; β ) has orthonormal columns. Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

ORTHOGONALIZATION OF TENSOR CARRIAGES ∀ tensor carriage g ( α i β ) ∃ decomposition g ( α i β ) = h ( αα ′ ) q ( α ′ i β ) with q ( α ′ i β ) being row orthogonal. ∀ tensor carriage g ( α i β ) ∃ decomposition g ( α i β ) = q ( α i β ′ ) h ( β ′ β ) with q ( α i β ′ ) being column orthogonal. Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA

PRODUCTS OF ORTHOGONAL TENSOR CARRIAGES A product of row (column) orthogonal tensor carriages t � p ( α s , i s . . . i t , α t ) = g k ( α k − 1 i k α k ) k = s + 1 is also row (column) orthogonal. Eugene Tyrtyshnikov NUMERICAL METHODS WITH TENSOR DATA