Numerical tensor methods and their applications I.V. Oseledets 8 - PowerPoint PPT Presentation

Numerical tensor methods and their applications I.V. Oseledets 8 May 2013 I.V. Oseledets Numerical tensor methods and their applications

All lectures 4 lectures, 2 May, 08:00 - 10:00: Introduction: ideas, matrix results, history. 7 May, 08:00 - 10:00: Novel tensor formats (TT, HT, QTT). 8 May, 08:00 - 10:00: Advanced tensor methods (eigenproblems, linear systems). 14 May, 08:00 - 10:00: Advanced topics, recent results and open problems. I.V. Oseledets Numerical tensor methods and their applications

Brief recap of Lecture 2 Tensor Train format Arithmetics Rounding QTT-format (idea of tensorization) Cross approximation I.V. Oseledets Numerical tensor methods and their applications

Plan of lecture 2 QTT-format: explicit representations of functions QTT-format: explicit representation of operators Classification theory QTT-Fourier transform QTT-convolution Linear systems Eigenvalue problems I.V. Oseledets Numerical tensor methods and their applications

QTT-format (publications-first) [1] S. V. Dolgov, B. N. Khoromskij, and D. V. Savostyanov. Superfast Fourier transform using QTT approximation. J. Fourier Anal. Appl. , 18(5):915–953, 2012. [2] V. Kazeev, B. N. Khoromskij, and E. E. Tyrtyshnikov. Multilevel Toeplitz matrices generated by tensor-structured vectors and convolution with logarithmic complexity. Technical Report 36, MPI MIS, Leipzig, 2011. [3] V. A. Kazeev and B. N. Khoromskij. Low-rank explicit QTT representation of the Laplace operator and its inverse. SIAM J. Matrix Anal. Appl. , 33(3):742–758, 2012. [4] B. N. Khoromskij. O ( d log n ) –Quantics approximation of N – d tensors in high-dimensional numerical modeling. Constr. Appr. , 34(2):257–280, 2011. [5] I. V. Oseledets. Approximation of 2 d × 2 d matrices using tensor decomposition. SIAM J. Matrix Anal. Appl. , 31(4):2130–2145, 2010. I.V. Oseledets Numerical tensor methods and their applications

QTT-format We have a vector v of values of a function f on a uniform grid with 2 d points: v ( i ) = f ( x i ) , x i = a + ih , h = ( b − a ) / ( n − 1 ) . I.V. Oseledets Numerical tensor methods and their applications

QTT-format We have a vector v of values of a function f on a uniform grid with 2 d points: v ( i ) = f ( x i ) , x i = a + ih , h = ( b − a ) / ( n − 1 ) . Reshaping into a tensor: i → ( i 0 , i 1 , . . . , i d − 1 ) . i = i 1 + 2 i 2 + 4 i 3 + . . . + 2 d − 1 i d V ( i 1 , . . . , i d ) = v ( i ) . I.V. Oseledets Numerical tensor methods and their applications

QTT-format Finally, we have: t k = a d + 2 k i k h V ( i 1 , . . . , i d ) = f ( t 1 + . . . + t d ) , I.V. Oseledets Numerical tensor methods and their applications

Exponential function f ( x ) = exp λ x , Then f ( t 1 + . . . + t d ) = exp ( λ t 1 ) . . . exp ( λ t d ) , it has rank 1! I.V. Oseledets Numerical tensor methods and their applications

Linear function f ( x ) = x f ( t 1 + . . . + t d ) = t 1 + . . . + t d I.V. Oseledets Numerical tensor methods and their applications

Linear function(full steps) t 1 + t 2 + t 3 + t 4 = I.V. Oseledets Numerical tensor methods and their applications

Linear function(full steps) � t 1 1 � � � 1 t 1 + t 2 + t 3 + t 4 = t 2 + t 3 + t 4 I.V. Oseledets Numerical tensor methods and their applications

Linear function(full steps) � t 1 1 � � � 1 t 1 + t 2 + t 3 + t 4 = = t 2 + t 3 + t 4 � t 1 1 � � � � � 1 0 1 = t 2 1 t 3 + t 4 I.V. Oseledets Numerical tensor methods and their applications

Linear function(full steps) � t 1 1 � � � 1 t 1 + t 2 + t 3 + t 4 = = t 2 + t 3 + t 4 � t 1 1 � � � � � 1 0 1 = = t 2 1 t 3 + t 4 � t 1 1 � � � � � � � 1 0 1 0 1 = t 2 1 t 3 1 t 4 I.V. Oseledets Numerical tensor methods and their applications

Sine function Similar representation can be obtained for a sine function: f ( x ) = sin λ x f ( t 1 + . . . + t d ) = sin ( t 1 + . . . + t d ) = � � sin t 2 � � sin t d − 1 � � cos x d � � − cos t 2 − cos t d − 1 = sin t 1 cos t 1 . . . cos t 2 sin t 2 cos t d − 1 sin t d − 1 sin x d The rank is still 2! I.V. Oseledets Numerical tensor methods and their applications

General result Theorem Let f be such that (O. Const. Approx., 2013) r � f ( x + y ) = u α ( x ) v α ( y ) α = 1 then the QTT-ranks are bounded by r Interesting example: rational functions I.V. Oseledets Numerical tensor methods and their applications

TT-format for matrices What about matrices? I.V. Oseledets Numerical tensor methods and their applications

TT-format for matrices What about matrices? Solution - a vector x associated with a d -tensor X ( i 1 , . . . , i d ) Linear operators, acting on such tensors, can be indexed as A ( i 1 , . . . , i d ; j 1 , . . . , j d ) . Terminology: d -level matrix I.V. Oseledets Numerical tensor methods and their applications

Two-level matrix: illustration Even in 2d it is interesting: A , B are n × n , Kronecker product   a 11 B a 12 B a 13 B a 14 B   a 21 B a 22 B a 23 B a 24 B   C = A ⊗ B =   a 31 B a 32 B a 33 B a 34 B a 41 B a 42 B a 43 B a 44 B I.V. Oseledets Numerical tensor methods and their applications

Two-level matrix: illustration In the index form: C = A ⊗ B C ( i 1 , i 2 ; j 1 j 2 ) = A ( i 1 , j 1 ) B ( i 2 , j 2 ) I.V. Oseledets Numerical tensor methods and their applications

Two-level matrix: illustration In the index form: C = A ⊗ B C ( i 1 , i 2 ; j 1 j 2 ) = A ( i 1 , j 1 ) B ( i 2 , j 2 ) Exactly rank-1 decomposition under permutation I.V. Oseledets Numerical tensor methods and their applications

Back to d -dimensions Let A be a d-level matrix: A ( i 1 , . . . , i d ; j 1 , . . . , j d ) (say, d -dimensional Laplace operator) I.V. Oseledets Numerical tensor methods and their applications

Back to d -dimensions Let A be a d-level matrix: A ( i 1 , . . . , i d ; j 1 , . . . , j d ) (say, d -dimensional Laplace operator) No low TT-ranks if considered as a 2 d -array! I.V. Oseledets Numerical tensor methods and their applications

Back to d -dimensions Right way: permute indices B ( i 1 j 1 ; i 2 j 2 ; . . . i d j d ) = A ( i 1 , . . . , i d ; j 1 , . . . , j d ) A ( i 1 , . . . , i d ; j 1 , . . . , j d ) = A 1 ( i 1 , j 1 ) A 2 ( i 2 , j 2 ) . . . A d ( i d , j d ) I.V. Oseledets Numerical tensor methods and their applications

Matrix-by-vector product A ( i 1 , . . . , i d ; j 1 , . . . j d ) = A 1 ( i 1 , j 1 ) . . . A d ( i d , j d ) X ( j 1 , . . . , j d ) = X 1 ( j 1 ) . . . X d ( j d ) Y ( I ) = � J A ( I , J ) X ( J ) Exercise: Find a formula I.V. Oseledets Numerical tensor methods and their applications

QTT-matrix Matrices in the QTT-format: 1 i , j = 1 , . . . , 2 d . a ij = i − j + 0 . 5 , Let us see what are the ranks. (Demo) I.V. Oseledets Numerical tensor methods and their applications

Laplace operator Consider an operator − d 2 dx 2 with Dirichlet boundary conditions, discretized using the simplest finite-difference scheme: (Illustration for n = 4)   2 − 1 0 0   − 1 2 − 1 0   ∆ =   0 − 1 2 − 1 0 0 − 1 2 (Demo) I.V. Oseledets Numerical tensor methods and their applications

Laplace operator V. Kazeev: QTT-ranks of the Laplace operator are bounded by 3 I.V. Oseledets Numerical tensor methods and their applications

Laplace operator: general results We will need a special matrix-by-matrix product: � � � � A 11 A 12 B 11 B 12 = ✶ A 21 A 22 B 21 B 22 � � A 11 ⊗ B 11 + A 12 ⊗ B 21 A 11 ⊗ B 12 + A 12 ⊗ B 22 = A 21 ⊗ B 11 + A 22 ⊗ B 21 A 21 ⊗ B 12 + A 22 ⊗ B 22 Doing Kronecker product for the blocks! I.V. Oseledets Numerical tensor methods and their applications

Basic QTT-blocks � � � � � � 1 0 0 1 0 1 I = J = P = , , , 0 1 0 0 1 0 � � � � 0 0 1 0 I 2 = , I 1 = , 0 1 0 0 � � � � − 1 1 1 1 E = , F = , − 1 1 1 1 � � � � − 1 − 1 0 0 K = , L = , 0 1 1 0 I.V. Oseledets Numerical tensor methods and their applications

QTT representation of 1D-Laplace     ✶ ( d − 2 ) I J ′ J 2 I − J − J ′ � �     ∆ ( d ) = I J ′ J J − J ✶   ✶   DD J ′ − J ′ I.V. Oseledets Numerical tensor methods and their applications

QTT representation in the Toolbox In the TT-Toolbox, it is defined via the function tt_qlaplace_dd I.V. Oseledets Numerical tensor methods and their applications

Wavelets and tensor trains QTT can be applied to matrices: A ( i , j ) → A ( i 1 , . . . , i d , j 1 , . . . , j d ) → A ( i 1 , j 1 , i 2 , j 2 , . . . , i d , j d ) I.V. Oseledets Numerical tensor methods and their applications

Wavelets and tensor trains Smooth function and/or special function: 512 × 512 “Hilbert image”: a ij = 1 . 0 / ( i − j + 0 . 5 ) 540 bytes (QTT) vs 8 KB (JPEG) I.V. Oseledets Numerical tensor methods and their applications

Wavelets and tensor trains QTT compression of simple images Good: Triangle BAD: Circle 70 KB (QTT) vs 8 KB 50 bytes (QTT) vs 8 KB (JPEG) (JPEG) I.V. Oseledets Numerical tensor methods and their applications