tensor matrix products with a compressed sparse tensor
play

Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith - PowerPoint PPT Presentation

Feb 22, 2024 •601 likes •1.09k views

Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Tensor-Matrix Products with a Compressed Sparse Tensor 1 / 47

  1. Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Tensor-Matrix Products with a Compressed Sparse Tensor 1 / 47 http://cs.umn.edu/~splatt/

  2. Tensor Introduction Tensors are the generalization of matrices to ≥ 3 D Tensors have m dimensions (or modes ) and are I 1 × . . . × I m . users items words Tensor-Matrix Products with a Compressed Sparse Tensor 2 / 47 http://cs.umn.edu/~splatt/

  3. Canonical Polyadic Decomposition (CPD) The CPD is an extension of the SVD to tensors We compute matrices A 1 , . . . , A m , each with F columns and λ , a vector of weights λ 1 λ F + · · · + ≈ Usually computed via alternating least squares (ALS) Tensor-Matrix Products with a Compressed Sparse Tensor 3 / 47 http://cs.umn.edu/~splatt/

  4. MTTKRP Matricized Tensor Times Khatri-Rao Product (MTTKRP) MTTKRP is the core computation of each iteration A 1 = X (1) ( A m � · · · � A 2 ) I 1 I 2 F X (1) I 1 I 1 I 2 ( A 3 � A 2 ) Tensor-Matrix Products with a Compressed Sparse Tensor 4 / 47 http://cs.umn.edu/~splatt/

  5. Related Work Tensor-Matrix Products with a Compressed Sparse Tensor 5 / 47 http://cs.umn.edu/~splatt/

  6. Uncompressed Tensors Stored as a list of coordinates ( i , j , k ) = v represents one nonzero ← A 1 ( i , :) ← A 1 ( i , :) + X ( i , j , k ) [ A 2 ( j , :) ∗ A 3 ( k , :)] Tensor-Matrix Products with a Compressed Sparse Tensor 6 / 47 http://cs.umn.edu/~splatt/

  7. Compressed Tensors SPLATT SPLATT uses a hierarchical storage scheme for 3D tensors This allows for operation reduction and coarse-grained parallelism ← Tensor-Matrix Products with a Compressed Sparse Tensor 7 / 47 http://cs.umn.edu/~splatt/

  8. Contributions Tensor-Matrix Products with a Compressed Sparse Tensor 8 / 47 http://cs.umn.edu/~splatt/

  9. Compressed Sparse Fiber (CSF) 1 2  i j k l  i 1 1 1 2     1 1 1 3 j 1 2 2     1 2 1 3   →   1 2 2 1 k 1 1 2 1 2     2 2 1 1     2 2 1 3 2 3 3 1 1 3 2   l 2 2 2 2 Tensor-Matrix Products with a Compressed Sparse Tensor 9 / 47 http://cs.umn.edu/~splatt/

  10. MTTKRP with a CSF Tensor Objective We want to perform MTTKRP on each tensor mode with only one CSF representation There are three types of nodes in a tree: root , internal , and leaf ◮ Each will have a tailored algorithm Tensor-Matrix Products with a Compressed Sparse Tensor 10 / 47 http://cs.umn.edu/~splatt/

  11. CSF-ROOT We do a depth-first traversal on the CSF structure Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 11 / 47 http://cs.umn.edu/~splatt/

  12. CSF-ROOT Inner products are accumulated in a buffer Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 12 / 47 http://cs.umn.edu/~splatt/

  13. CSF-ROOT Inner products are accumulated in a buffer Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 13 / 47 http://cs.umn.edu/~splatt/

  14. CSF-ROOT Hadamard products are then propagated up the CSF tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 14 / 47 http://cs.umn.edu/~splatt/

  15. CSF-ROOT Hadamard products are then propagated up the CSF tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 15 / 47 http://cs.umn.edu/~splatt/

  16. CSF-ROOT Results are accumulated when we reach the top Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 16 / 47 http://cs.umn.edu/~splatt/

  17. CSF-ROOT The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 17 / 47 http://cs.umn.edu/~splatt/

  18. CSF-ROOT The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 18 / 47 http://cs.umn.edu/~splatt/

  19. CSF-ROOT Partial results are kept in buffer Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 19 / 47 http://cs.umn.edu/~splatt/

  20. CSF-ROOT Inner products are accumulated in a buffer Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 20 / 47 http://cs.umn.edu/~splatt/

  21. CSF-ROOT Inner products are accumulated in a buffer Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 21 / 47 http://cs.umn.edu/~splatt/

  22. CSF-LEAF This time, Hadamard products are pushed down the tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 22 / 47 http://cs.umn.edu/~splatt/

  23. CSF-LEAF This time, Hadamard products are pushed down the tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 23 / 47 http://cs.umn.edu/~splatt/

  24. CSF-LEAF This time, Hadamard products are pushed down the tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 24 / 47 http://cs.umn.edu/~splatt/

  25. CSF-LEAF This time, Hadamard products are pushed down the tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 25 / 47 http://cs.umn.edu/~splatt/

  26. CSF-LEAF Leaves designate write locations Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 26 / 47 http://cs.umn.edu/~splatt/

  27. CSF-LEAF Leaves designate write locations Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 27 / 47 http://cs.umn.edu/~splatt/

  28. CSF-LEAF The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 28 / 47 http://cs.umn.edu/~splatt/

  29. CSF-LEAF The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 29 / 47 http://cs.umn.edu/~splatt/

  30. CSF-LEAF The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 30 / 47 http://cs.umn.edu/~splatt/

  31. CSF-LEAF The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 31 / 47 http://cs.umn.edu/~splatt/

  32. CSF-LEAF The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 32 / 47 http://cs.umn.edu/~splatt/

  33. CSF-INTERNAL Internal nodes use a combination of CSF-ROOT and CSF-LEAF Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 33 / 47 http://cs.umn.edu/~splatt/

  34. CSF-INTERNAL Hadamard products are pushed down to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 34 / 47 http://cs.umn.edu/~splatt/

  35. CSF-INTERNAL CSF-ROOT next pulls up to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 35 / 47 http://cs.umn.edu/~splatt/

  36. CSF-INTERNAL CSF-ROOT next pulls up to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 36 / 47 http://cs.umn.edu/~splatt/

  37. CSF-INTERNAL CSF-ROOT next pulls up to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 37 / 47 http://cs.umn.edu/~splatt/

  38. CSF-INTERNAL CSF-ROOT next pulls up to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 38 / 47 http://cs.umn.edu/~splatt/

  39. CSF-INTERNAL CSF-ROOT next pulls up to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 39 / 47 http://cs.umn.edu/~splatt/

  40. Parallelism – Tiling A 1 A 3 A 2 Tensor-Matrix Products with a Compressed Sparse Tensor 40 / 47 http://cs.umn.edu/~splatt/

  41. Datasets Dataset I 1 I 2 I 3 nnz NELL-2 12K 9K 28K 77M Beer 33K 66K 960K 94M Netflix 480K 18K 2K 100M Delicious 532K 17M 3M 140M NELL-1 3M 2M 25M 143M Amazon 5M 18M 2M 1.7B Tensor-Matrix Products with a Compressed Sparse Tensor 41 / 47 http://cs.umn.edu/~splatt/

  42. Storage Comparison SPLATT COORD CSF-M CSF-T 99.3 10 2 51.9 47.5 36.4 Tensor storage (GB) 10 1 8.8 8.2 5.0 4.5 4.2 4.3 4.0 3.7 3.6 3.0 2.8 2.6 2.4 2.3 2.1 1.8 1.6 1.4 1.2 1.1 10 0 NELL-2 Beer Netflix Delicious NELL-1 Amazon Dataset Tensor-Matrix Products with a Compressed Sparse Tensor 42 / 47 http://cs.umn.edu/~splatt/

  43. CSF-ROOT 90 SPLATT CSF-M CSF-T 80 Speedup over COORD with 16 threads 70 60 50 40 30 20 10 0 NELL-2 Beer Netflix Delicious NELL-1 Amazon Dataset Tensor-Matrix Products with a Compressed Sparse Tensor 43 / 47 http://cs.umn.edu/~splatt/

Recommend

Memory-Efficient Parallel Computation of Tensor and Matrix Products for Big Tensor Decomposition

Memory-Efficient Parallel Computation of Tensor and Matrix Products for Big Tensor Decomposition

Memory-Efficient Parallel Computation of Tensor and Matrix Products for Big Tensor Decomposition N. Ravindran , N.D. Sidiropoulos , S. Smith , and G. Karypis Dept. of ECE & DTC, Dept. of CSci & DTC University of

553 views • 19 slides
8. Tensor Field Visualization Tensor: extension of concept of scalar and vector Tensor data

8. Tensor Field Visualization Tensor: extension of concept of scalar and vector Tensor data

8. Tensor Field Visualization Tensor: extension of concept of scalar and vector Tensor data for a tensor of level k is given by t i1,i2,,ik (x 1 ,,x n ) Second-order tensor often represented by matrix Examples:

292 views • 15 slides
A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory

A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory

A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory January 25, 2019 @ MIT Figure sources: A brief survey of tensors by Berton Earnshaw and NVIDIA Tensor Cores HiCOO: Hierarchical Storage of Sparse

272 views • 23 slides
and You Tensor network methods Matrix product states (MPS) Projected Entangled Pair States

and You Tensor network methods Matrix product states (MPS) Projected Entangled Pair States

Tensor Networks and You Nikko Pomata How do you network tensors? Tensors: a review The tensor-network notation T ENSOR N ETWORKS Tensor network examples and You Tensor network methods Matrix product states (MPS) Projected Entangled Pair

642 views • 24 slides
Tensor For F , F Sh( X ) we have F F Sh( X ). Since tensor commutes with

Tensor For F , F Sh( X ) we have F F Sh( X ). Since tensor commutes with

Tensor For F , F Sh( X ) we have F F Sh( X ). Since tensor commutes with taking stalk, the non-exactness comes from that F x might not torsion-free, and this can be taken care by a simple resolution 0 P P F

1.85k views • 7 slides
Renormalization of Tensor Network States II. RG of Tensor Network States Tao Xiang Institute of

Renormalization of Tensor Network States II. RG of Tensor Network States Tao Xiang Institute of

Renormalization of Tensor Network States II. RG of Tensor Network States Tao Xiang Institute of Physics Chinese Academy of Sciences txiang@iphy.ac.cn Tensor-Network Ansatz of the ground state wavefunction 1D: Matrix Product State (MPS) : m

603 views • 57 slides
Tensor Products of A -algebras with Homotopy Inner Products (Joint work with Thomas Tradler,

Tensor Products of A -algebras with Homotopy Inner Products (Joint work with Thomas Tradler,

Tensor Products of A -algebras with Homotopy Inner Products (Joint work with Thomas Tradler, CUNY) Ron Umble Millersville University Lehigh Geometry and Topology Conference May 25, 2013 Tensor Products of A -infinity Algebras Let K =

1.17k views • 105 slides
TENSOR ALGEBRA Continuum Mechanics Course (MMC) - ETSECCPB - UPC Introduction to Tensors Tensor

TENSOR ALGEBRA Continuum Mechanics Course (MMC) - ETSECCPB - UPC Introduction to Tensors Tensor

TENSOR ALGEBRA Continuum Mechanics Course (MMC) - ETSECCPB - UPC Introduction to Tensors Tensor Algebra 2 Introduction SCALAR , , ... v VECTOR v f , , ... , , ... MATRIX ? , ... C 3 Concept of Tensor A TENSOR

885 views • 88 slides
Tensor Field Visualization 9-1 Ronald Peikert SciVis 2007 - Tensor Fields Tensors

Tensor Field Visualization 9-1 Ronald Peikert SciVis 2007 - Tensor Fields Tensors

Tensor Field Visualization 9-1 Ronald Peikert SciVis 2007 - Tensor Fields Tensors "Tensors are the language of mechanics" T Tensor of order (rank) f d ( k) 0: scalar 1: vector 1: vector 2: matrix (example: stress tensor)

773 views • 28 slides
Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith &

Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith &

Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith & George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Sparse Tensor Factorization: Algorithms,

844 views • 80 slides
(Some) Challenges in (Some) Challenges in Tensor Mining Tensor Mining Evrim Acar Sandia

(Some) Challenges in (Some) Challenges in Tensor Mining Tensor Mining Evrim Acar Sandia

(Some) Challenges in (Some) Challenges in Tensor Mining Tensor Mining Evrim Acar Sandia National Labs., Livermore, CA Tensor Mining Tensor Mining Parafac Parafac unsupervised unsupervised = + + X X E X X = + dense or sparse

340 views • 17 slides
SPLATT Efficient and Parallel Sparse Tensor-Matrix Multiplication Shaden Smith 1 Niranjay

SPLATT Efficient and Parallel Sparse Tensor-Matrix Multiplication Shaden Smith 1 Niranjay

SPLATT Efficient and Parallel Sparse Tensor-Matrix Multiplication Shaden Smith 1 Niranjay Ravindran Nicholas D. Sidiropoulos George Karypis University of Minnesota 1 shaden@cs.umn.edu Shaden Smith, shaden@cs.umn.edu (U. Minnesota) SPLATT 1

3.48k views • 24 slides
Tensor products of finitely cococomplete and abelian categories 1 Ignacio Lpez Franco

Tensor products of finitely cococomplete and abelian categories 1 Ignacio Lpez Franco

Delignes tensor product Questions we answer Existence of Delignes tensor Counterexample to the existence Tensor products of finitely cococomplete and abelian categories 1 Ignacio Lpez Franco University of Cambridge Gonville and Caius

554 views • 33 slides
Parallel Sparse Matrix-Vector and Matrix- Transpose-Vector Multiplication using Compressed Sparse

Parallel Sparse Matrix-Vector and Matrix- Transpose-Vector Multiplication using Compressed Sparse

Parallel Sparse Matrix-Vector and Matrix- Transpose-Vector Multiplication using Compressed Sparse Blocks Aydn Bulu, UCSB Jeremy T. Fineman (MIT) Matteo Frigo (Cilk Arts) John R. Gilbert (UCSB) Charles E. Leiserson (MIT & Cilk Arts) 1

523 views • 21 slides
A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Medium-Grained Sparse Tensor Factorization 1 / 24

424 views • 25 slides
Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares

Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares

Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares Edgar Solomonik Department of Computer Science University of Illinois at Urbana-Champaign Invited Workshop on Compiler Techniques for Sparse Tensor

222 views • 9 slides
Accelerating the Tucker Decomposition with Compressed Sparse Tensors Shaden Smith and George

Accelerating the Tucker Decomposition with Compressed Sparse Tensors Shaden Smith and George

Accelerating the Tucker Decomposition with Compressed Sparse Tensors Shaden Smith and George Karypis Department of Computer Science & Engineering, University of Minnesota { shaden, karypis } @cs.umn.edu Euro-Par 2017 1 / 40 Outline Tensor

921 views • 57 slides
Invited Workshop on Compiler Techniques for Sparse Tensor Algebra Cambridge MA, January 26th

Invited Workshop on Compiler Techniques for Sparse Tensor Algebra Cambridge MA, January 26th

Invited Workshop on Compiler Techniques for Sparse Tensor Algebra Cambridge MA, January 26th Sparse Tensors are Everywhere Data Analytics Machine Learning Science and Engineering Movie ratings Convolutional Networks Product Reviews QCD

427 views • 29 slides
BOOLEAN MATRIX AND TENSOR DECOMPOSITIONS Pauli Miettinen TML 2013 27 September 2013 BOOLEAN

BOOLEAN MATRIX AND TENSOR DECOMPOSITIONS Pauli Miettinen TML 2013 27 September 2013 BOOLEAN

BOOLEAN MATRIX AND TENSOR DECOMPOSITIONS Pauli Miettinen TML 2013 27 September 2013 BOOLEAN MATRIX AND TENSOR DECOMPOSITIONS Boolean decompositions are like normal decompositions, except that Input is a binary matrix or tensor

964 views • 36 slides
TENSOR CORE PROGRAMMABILITY AND PROFILING FOR AI AND HPC APPLICATIONS Griffin Lacey Max Katz

TENSOR CORE PROGRAMMABILITY AND PROFILING FOR AI AND HPC APPLICATIONS Griffin Lacey Max Katz

TENSOR CORE PROGRAMMABILITY AND PROFILING FOR AI AND HPC APPLICATIONS Griffin Lacey Max Katz TOO LONG; DIDNT LISTEN Tensor Cores enable fast mixed precision matrix multiplications Growing number of AI/HPC examples accelerated up to

810 views • 47 slides
When tensor decomposition meets compressed sensing Pierre Comon I3S, CNRS, University of Nice -

When tensor decomposition meets compressed sensing Pierre Comon I3S, CNRS, University of Nice -

Intro General Exact CP Approx CP NonNeg Coherence AAP END When tensor decomposition meets compressed sensing Pierre Comon I3S, CNRS, University of Nice - Sophia Antipolis, France Collaborator: Lek-Heng Lim Sept. 27-30, 2010 Pierre

1.46k views • 95 slides
Quantum channel entanglement producing and distance of the channel matrix from the tensor

Quantum channel entanglement producing and distance of the channel matrix from the tensor

Quantum channel entanglement producing and distance of the channel matrix from the tensor product Igor Popov, Maria Faleeva ITMO University St. Petersburg, Russia Report plan 1. Introduction. Quantum computing motivation. 2. Matrix problem.

461 views • 31 slides
Tensor learning approach to sparse QMC sampling of two-particle Greens function in DMFT

Tensor learning approach to sparse QMC sampling of two-particle Greens function in DMFT

Tensor learning approach to sparse QMC sampling of two-particle Greens function in DMFT Hiroshi SHINAOKA Collaborators N. Chikano, J. Otsuki, M. Ohzeki, K. Yoshimi, K. Haule, M. Wallerberger, J. Li, E. Gull, D. Geffroy, J. Kune Matsubara

337 views • 17 slides
Tensor MUSIC in Multidimensional Sparse Arrays Chun-Lin Liu 1 and P . Vaidyanathan 2 . P Dept. of

Tensor MUSIC in Multidimensional Sparse Arrays Chun-Lin Liu 1 and P . Vaidyanathan 2 . P Dept. of

Tensor MUSIC in Multidimensional Sparse Arrays Chun-Lin Liu 1 and P . Vaidyanathan 2 . P Dept. of Electrical Engineering, MC 136-93 California Institute of Technology, Pasadena, CA 91125, USA cl.liu@caltech.edu 1 , ppvnath@systems.caltech.edu 2

523 views • 24 slides

More recommend