sparse tensor factorization algorithms data structures
play

Sparse Tensor Factorization: Algorithms, Data Structures, and - PowerPoint PPT Presentation

Jul 20, 2023 •44 likes •844 views

Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith & George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Sparse Tensor Factorization: Algorithms,

  1. Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith & George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  2. Talk Outline Introduction 1 Compressed Sparse Fiber 2 Cache-Friendly Reordering & Tiling 3 Distributed-Memory MTTKRP 4 Conclusions 5 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  3. Table of Contents Introduction 1 Compressed Sparse Fiber 2 Cache-Friendly Reordering & Tiling 3 Distributed-Memory MTTKRP 4 Conclusions 5 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  4. Tensor Introduction Tensors are the generalization of matrices to ≥ 3 D Tensors have m dimensions (or modes ) and are I 1 × . . . × I m ◮ We’ll usually stick to I × J × K in this talk users contexts items Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  5. Applications Dataset I J K nnz NELL-2 12K 9K 28K 77M Beer 33K 66K 960K 94M Netflix 480K 18K 2K 100M Delicious 532K 17M 3M 140M NELL-1 3M 2M 25M 143M Amazon 5M 18M 2M 1.7B Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  6. Canonical Polyadic Decomposition (CPD) We compute matrices A , B , C , each with F columns ◮ We will use A (1) , . . . , A ( m ) when ≥ 3 modes ≈ + · · · + Usually computed via alternating least squares (ALS) Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  7. Matricized Tensor Times Khatri-Rao Product MTTKRP MTTKRP is the core computation of each iteration A = X (1) ( C � B ) J · K F X (1) I J · K ( C � B ) Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  8. Alternating Least Squares 1: while not converged do A ⊺ = ( C ⊺ C ∗ B ⊺ B ) − 1 � � ⊺ X (1) ( C � B ) 2: B ⊺ = ( C ⊺ C ∗ A ⊺ A ) − 1 � � ⊺ X (2) ( C � A ) 3: C ⊺ = ( B ⊺ B ∗ A ⊺ A ) − 1 � � ⊺ X (3) ( B � A ) 4: 5: end while Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  9. Tensor Storage – Coordinate Form   i j k l v 1 1 1 2 1 .     1 1 1 3 1 .     1 2 1 3 3 .     1 2 2 1 8 .     2 2 1 1 1 .     2 2 1 3 3 .   2 2 2 2 8 . Why don’t we unfold? We need a representation of X for each mode NELL has dimensions 3 M × 2 M × 25 M ◮ Add a fourth mode and we exceed 2 64 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  10. MTTKRP A C i ← k B j A ( i , :) ← A ( i , :) + X ( i , j , k ) [ B ( j , :) ∗ C ( k , :)] Limitations Memory bandwidth Parallelism Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  11. Table of Contents Introduction 1 Compressed Sparse Fiber 2 Cache-Friendly Reordering & Tiling 3 Distributed-Memory MTTKRP 4 Conclusions 5 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  12. Can we do better? Consider three nonzeros in the fiber X ( i , j , :) (a vector) A ( i , :) ← A ( i , :) + X ( i , j , k 1 ) [ B ( j , :) ∗ C ( k 1 , :)] A ( i , :) ← A ( i , :) + X ( i , j , k 2 ) [ B ( j , :) ∗ C ( k 2 , :)] A ( i , :) ← A ( i , :) + X ( i , j , k 3 ) [ B ( j , :) ∗ C ( k 3 , :)] Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  13. Can we do better? Consider three nonzeros in the fiber X ( i , j , :) (a vector) A ( i , :) ← A ( i , :) + X ( i , j , k 1 ) [ B ( j , :) ∗ C ( k 1 , :)] A ( i , :) ← A ( i , :) + X ( i , j , k 2 ) [ B ( j , :) ∗ C ( k 2 , :)] A ( i , :) ← A ( i , :) + X ( i , j , k 3 ) [ B ( j , :) ∗ C ( k 3 , :)] A little factoring... � � 3 � A ( i , :) ← A ( i , :) + B ( j , :) ∗ X ( i , j , k x ) C ( k x , :) x =1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  14. SPLATT: The S urprisingly P aralle L sp A rse T ensor T oolkit ← [Smith, Ravindran, Sidiropoulos, and Karypis 2015] Fibers are sparse vectors Slice X ( i , : , :) is almost a CSR matrix... But, we need m representations of X Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  15. Compressed Sparse Fiber (CSF)   1 2 i j k l i 1 1 1 2     1 1 1 3 j 1 2 2     1 2 1 3     → 1 2 2 1   1 1 2 1 2 k   2 2 1 1     2 2 1 3   2 3 3 1 1 3 2 l 2 2 2 2 [Smith and Karypis 2015] Modes are recursively compressed Values are stored in the leaves (not shown) Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  16. MTTKRP with a CSF Tensor Objective We want to perform MTTKRP on each tensor mode with only one CSF representation There are three types of nodes in a tree: root , internal , and leaf ◮ Each will have a tailored algorithm ◮ root and leaf are special cases of internal Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  17. CSF-LEAF The leaf nodes determine the output location Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  18. CSF-LEAF Hadamard products are pushed down the tree Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  19. CSF-LEAF Hadamard products are pushed down the tree Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  20. CSF-LEAF Hadamard products are pushed down the tree Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  21. CSF-LEAF Leaves designate write locations Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  22. CSF-LEAF Leaves designate write locations Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  23. CSF-LEAF The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  24. CSF-LEAF The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  25. CSF-LEAF The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  26. CSF-LEAF The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  27. CSF-LEAF The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  28. CSF-ROOT Inner products are accumulated in a buffer Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  29. CSF-ROOT Inner products are accumulated in a buffer Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  30. CSF-ROOT Hadamard products are then propagated up the CSF tree Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  31. CSF-ROOT Hadamard products are then propagated up the CSF tree Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  32. CSF-ROOT Results are written to A (1) Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  33. CSF-ROOT The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  34. CSF-ROOT The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  35. CSF-ROOT Partial results are kept in buffer Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  36. CSF-ROOT Inner products are accumulated in a buffer Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

Recommend

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Medium-Grained Sparse Tensor Factorization 1 / 24

424 views • 25 slides
Sparse Tensor Factorization on Many-Core Processors with High-Bandwidth Memory Shaden Smith 1

Sparse Tensor Factorization on Many-Core Processors with High-Bandwidth Memory Shaden Smith 1

Sparse Tensor Factorization on Many-Core Processors with High-Bandwidth Memory Shaden Smith 1 , Jongsoo Park 2 , and George Karypis 1 1 Department of Computer Science & Engineering, University of Minnesota 2 Parallel Computing Lab, Intel

802 views • 37 slides
Parallel Numerical Algorithms Chapter 4 Sparse Linear Systems Section 4.1 Direct Methods

Parallel Numerical Algorithms Chapter 4 Sparse Linear Systems Section 4.1 Direct Methods

Sparse Matrices Sparse Triangular Solve Cholesky Factorization Sparse Cholesky Factorization Parallel Numerical Algorithms Chapter 4 Sparse Linear Systems Section 4.1 Direct Methods Michael T. Heath and Edgar Solomonik Department of

906 views • 63 slides
Algorithms and Data Structures: Overview Algorithms and data structures Data Abstraction,

Algorithms and Data Structures: Overview Algorithms and data structures Data Abstraction,

Algorithms and Data Structures: Overview Algorithms and data structures Data Abstraction, Ch. 3 Linked lists, Ch. 4 Recursion, Ch. 5 Stacks, Ch. 6 Queues, Ch. 7 Algorithm Efficiency and Sorting, Ch. 9 Trees, Ch. 10

729 views • 44 slides
Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares

Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares

Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares Edgar Solomonik Department of Computer Science University of Illinois at Urbana-Champaign Invited Workshop on Compiler Techniques for Sparse Tensor

222 views • 9 slides
Basker : A Threaded Sparse LU factorization utilizing Hierarchical Parallelism and Data Layouts

Basker : A Threaded Sparse LU factorization utilizing Hierarchical Parallelism and Data Layouts

Basker : A Threaded Sparse LU factorization utilizing Hierarchical Parallelism and Data Layouts Siva Rajamanickam Joshua Booth, Heidi Thornquist Sixth International Workshop on Accelerators and Hybrid Exascale Systems (IPDPS 2016) Sandia

424 views • 15 slides
Introduction to Algorithms and Data Structures CSC 1051 Algorithms and Data Structures I Dr.

Introduction to Algorithms and Data Structures CSC 1051 Algorithms and Data Structures I Dr.

Introduction to Algorithms and Data Structures CSC 1051 Algorithms and Data Structures I Dr. Mary-Angela Papalaskari Department of Computing Sciences Villanova University Course website: www.csc.villanova.edu/~map/1051/s18 Some slides in

975 views • 64 slides
Invited Workshop on Compiler Techniques for Sparse Tensor Algebra Cambridge MA, January 26th

Invited Workshop on Compiler Techniques for Sparse Tensor Algebra Cambridge MA, January 26th

Invited Workshop on Compiler Techniques for Sparse Tensor Algebra Cambridge MA, January 26th Sparse Tensors are Everywhere Data Analytics Machine Learning Science and Engineering Movie ratings Convolutional Networks Product Reviews QCD

427 views • 29 slides
Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith George Karypis University

Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith George Karypis University

Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Tensor-Matrix Products with a Compressed Sparse Tensor 1 / 47

1.09k views • 47 slides
A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory

A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory

A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory January 25, 2019 @ MIT Figure sources: A brief survey of tensors by Berton Earnshaw and NVIDIA Tensor Cores HiCOO: Hierarchical Storage of Sparse

272 views • 23 slides
ACCELERATING SPARSE CHOLESKY FACTORIZATION ON THE GPU STEVE RENNICH, SR. ENGINEER, NVIDIA

ACCELERATING SPARSE CHOLESKY FACTORIZATION ON THE GPU STEVE RENNICH, SR. ENGINEER, NVIDIA

ACCELERATING SPARSE CHOLESKY FACTORIZATION ON THE GPU STEVE RENNICH, SR. ENGINEER, NVIDIA DEVELOPER TECHNOLOGY DARKO STOSIC, PHD CANDIDATE, UNIV. FEDERAL DE PERNAMBUCO TIM DAVIS, PROFESSOR, CSE, TEXAS A&M UNIVERSITY SPARSE MATRIX

425 views • 28 slides
Data Structures and What is a data structure? Algorithms Way of storing data in computer

Data Structures and What is a data structure? Algorithms Way of storing data in computer

Introduction: Data Structures Data Structures and What is a data structure? Algorithms Way of storing data in computer so can be used efficiently Set of operations that access the data in prescribed ways Prof. Nadeem Abdul Hamid

686 views • 10 slides
Data Structures and Java Collections Framework 1 Algorithms and Data Structures

Data Structures and Java Collections Framework 1 Algorithms and Data Structures

IS311 Data Structures and Java Collections Framework 1 Algorithms and Data Structures Algorithm Sequence of steps used to solve a problem Operates on collection of data Each element of collection - > data structure

794 views • 54 slides
Tensor Factorization via Matrix Factorization Volodymyr Kuleshov Arun Tejasvi Chaganty Percy

Tensor Factorization via Matrix Factorization Volodymyr Kuleshov Arun Tejasvi Chaganty Percy

Tensor Factorization via Matrix Factorization Volodymyr Kuleshov Arun Tejasvi Chaganty Percy Liang Stanford University May 11, 2015 Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 1 / 28 Introduction:

1.1k views • 79 slides
Structured sparse methods for matrix factorization Francis Bach Sierra team, INRIA - Ecole

Structured sparse methods for matrix factorization Francis Bach Sierra team, INRIA - Ecole

Structured sparse methods for matrix factorization Francis Bach Sierra team, INRIA - Ecole Normale Sup erieure March 2011 Joint work with R. Jenatton, J. Mairal, G. Obozinski Structured sparse methods for matrix factorization Outline

1.07k views • 73 slides
Parallel Numerical Algorithms Chapter 7 Differential Equations Section 7.5 Tensor

Parallel Numerical Algorithms Chapter 7 Differential Equations Section 7.5 Tensor

Tensor Algebra Tensor Decompositions Fast Algorithms Parallel Numerical Algorithms Chapter 7 Differential Equations Section 7.5 Tensor Analysis Edgar Solomonik Department of Computer Science University of Illinois at Urbana-Champaign

722 views • 26 slides
Review Informatik 2 What did we learn about Algorithms and Data Structures? The Big Picture

Review Informatik 2 What did we learn about Algorithms and Data Structures? The Big Picture

Review Informatik 2 What did we learn about Algorithms and Data Structures? The Big Picture Data Structures Algorithms Linked Lists Theory : - Single and Double Links - Recursion - Anchors - Complexity Trees - Tree Properties Practice

1.07k views • 72 slides
Exams Gerth Stlting Brodal Algorithms and Data Structures Retreat, Sandbjerg, Denmark, March 3,

Exams Gerth Stlting Brodal Algorithms and Data Structures Retreat, Sandbjerg, Denmark, March 3,

Autogenerating Algorithms and Data Structures Exams Gerth Stlting Brodal Algorithms and Data Structures Retreat, Sandbjerg, Denmark, March 3, 2020 Algorithms and Data Structures Cource ADS 10 ECTS, 2 nd semester 1991 - 2003 Written exam ADS

668 views • 29 slides
Streaming Tensor Factorization for Infinite Data Sources Shaden Smith - Intel Parallel Computing

Streaming Tensor Factorization for Infinite Data Sources Shaden Smith - Intel Parallel Computing

Streaming Tensor Factorization for Infinite Data Sources Shaden Smith - Intel Parallel Computing Lab Kejun Huang - University of Minnesota Nicholas D. Sidiropoulos - University of Virginia George Karypis - University of Minnesota

332 views • 14 slides
Analysis of Algorithms Data Structures and Algorithms for CL III, WS 2019-2020 Corina Dima

Analysis of Algorithms Data Structures and Algorithms for CL III, WS 2019-2020 Corina Dima

Department of General and Computational Linguistics Analysis of Algorithms Data Structures and Algorithms for CL III, WS 2019-2020 Corina Dima corina.dima@uni-tuebingen.de M ICHAEL G OODRICH Data Structures & Algorithms in Python R OBERTO T

1.36k views • 79 slides
Infinite Positive Semidefinite Tensor Factorization for Source Separation of Mixture Signals

Infinite Positive Semidefinite Tensor Factorization for Source Separation of Mixture Signals

Infinite Positive Semidefinite Tensor Factorization by K. Yoshii, R. Tomioka, D. Mochihashi, and M. Goto Infinite Positive Semidefinite Tensor Factorization for Source Separation of Mixture Signals Kazuyoshi Yoshii Ryota Tomioka Daichi

461 views • 8 slides
Format Abstraction for Sparse Tensor Algebra Compilers Stephen Chou , Fredrik Kjolstad, and Saman

Format Abstraction for Sparse Tensor Algebra Compilers Stephen Chou , Fredrik Kjolstad, and Saman

Format Abstraction for Sparse Tensor Algebra Compilers Stephen Chou , Fredrik Kjolstad, and Saman Amarasinghe Sparse tensors are a natural way of representing real-world data 2 Sparse tensors are a natural way of representing real-world data 2

1.56k views • 134 slides
CS 310 - Advanced Data Structures and Algorithms Basic Data Structures May 31, 2018 Mohammad

CS 310 - Advanced Data Structures and Algorithms Basic Data Structures May 31, 2018 Mohammad

CS 310 - Advanced Data Structures and Algorithms Basic Data Structures May 31, 2018 Mohammad Hadian Advanced Data Structures and Algorithms May 31, 2018 1 / 21 Basic Data Structures Array Dynamic Array (amortized analysis) LinkedList

469 views • 21 slides
More Graph Algorithms Data Structures and Algorithms CSE 373 SP 18 - KASEY CHAMPION 1

More Graph Algorithms Data Structures and Algorithms CSE 373 SP 18 - KASEY CHAMPION 1

More Graph Algorithms Data Structures and Algorithms CSE 373 SP 18 - KASEY CHAMPION 1 Announcements Calendar on the webpage has updated office hours for this week. Last Time: Dijkstras Algorithm to find shortest paths. Today: Two more

509 views • 29 slides

More recommend