planc parallel low rank approximations with non
play

PLANC: Parallel Low Rank Approximations with Non-negativity - PowerPoint PPT Presentation

Jan 17, 2023 •252 likes •484 views

PLANC: Parallel Low Rank Approximations with Non-negativity Constraints Ramakrishnan Kannan, Michael Matheson, Grey Ballard, Srinivas Eswar , Koby Hayashi, Haesun Park Ph.D student, School of CSE Georgia Institute of Technology Advisors: Rich

  1. PLANC: Parallel Low Rank Approximations with Non-negativity Constraints Ramakrishnan Kannan, Michael Matheson, Grey Ballard, Srinivas Eswar , Koby Hayashi, Haesun Park Ph.D student, School of CSE Georgia Institute of Technology Advisors: Rich Vuduc and Haesun Park January 26, 2019 Workshop on Compiler Techniques for Sparse Tensor Algebra Acknowledgement: This work was partly sponsored by NSF, Sandia and ORNL Srinivas Eswar (GT) PLANC January 26, 2019 1 / 12

  2. Summary PLANC is an open source, scalable and flexible software package to compute Non-negative Tensor Factorisation. Srinivas Eswar (GT) PLANC January 26, 2019 2 / 12

  3. Summary PLANC is an open source, scalable and flexible software package to compute Non-negative Tensor Factorisation. Implements state of the art communication avoiding algorithm for matricised-tensor times Khatri-Rao product (MTTKRP). Srinivas Eswar (GT) PLANC January 26, 2019 2 / 12

  4. Summary PLANC is an open source, scalable and flexible software package to compute Non-negative Tensor Factorisation. Implements state of the art communication avoiding algorithm for matricised-tensor times Khatri-Rao product (MTTKRP). Popular optimisation methods like Block Principal Pivoting, Alternating Direction Method of Multipliers, first order Nesterov methods etc. for Non-negative Least Squares are included. Srinivas Eswar (GT) PLANC January 26, 2019 2 / 12

  5. Summary PLANC is an open source, scalable and flexible software package to compute Non-negative Tensor Factorisation. Implements state of the art communication avoiding algorithm for matricised-tensor times Khatri-Rao product (MTTKRP). Popular optimisation methods like Block Principal Pivoting, Alternating Direction Method of Multipliers, first order Nesterov methods etc. for Non-negative Least Squares are included. NTF is an important contributor towards explainable AI with a wide range of applications like spectral unmixing, scientific visualization, healthcare analytics, topic modelling etc. Srinivas Eswar (GT) PLANC January 26, 2019 2 / 12

  6. CP Decomposition R � u r ( σ r v T Matrix: M ≈ r ) r =1 R � Tensor: X ≈ ( λ r u r ) ◦ v r ◦ w r r =1 This is known as the CANDECOMP or PARAFAC or canonical polyadic or CP decomposition . It approximate tensors as sum of outer products or rank-1 tensors. NNCP imposes non-negativity constraints on the factor matrices to aid interpretability. Srinivas Eswar (GT) PLANC January 26, 2019 3 / 12

  7. Computational Bottlenecks The MTTKRP is the major bottleneck for NNCP. M (1) = X (1) ( W ⊙ V ) J K � � m ir = x ijk v jr w kr j =1 k =1 Standard approach is to explicitly matricise the tensor and form the Khatri-Rao product before calling DGEMM. Can we do better? Avoid matricisation of the tensor and full Khatri-Rao products ... Srinivas Eswar (GT) PLANC January 26, 2019 4 / 12

  8. Communication Lower Bounds Following the nested arrays lower bounds [BKR18]. Theorem Any parallel MTTKRP algorithm involving a tensor with I k = I 1 / N for all k and that evenly distributes one copy of the input and output performs at least � I N �� NIR � 1 / N � � 2 N − 1 Ω + NR P P sends and receives. (Either term can dominate.) Key Assumptions: algorithm is not allowed to pre-compute and re-use temporary values. � I � � 1 / N � Ω NR is the most frequently occurring case for relatively P small P or R . Srinivas Eswar (GT) PLANC January 26, 2019 5 / 12

  9. Shared Memory Optimisation - Dimension Trees Reuse computations across MTTKRPs. M (1) = X (1) ( U (3) ⊙ U (2) ) M (2) = X (2) ( U (3) ⊙ U (1) ) Utilise a “dimension tree” to store and reuse partial products [PTC13, LKL + 17, HBJT18]. { 1 , 2 , 3 } PM PM { 1 , 2 } M (3) mTTV mTTV M (1) M (2) PM = partial MTTKRP mTTV = multi-Tensor-Times-Vector Srinivas Eswar (GT) PLANC January 26, 2019 6 / 12

  10. Distributed Memory Optimisation - Communication Avoiding Each processor Starts with one subtensor and 1 subset of rows of each input factor matrix U (1) U (3) M (2) Srinivas Eswar (GT) PLANC January 26, 2019 7 / 12

  11. Distributed Memory Optimisation - Communication Avoiding Each processor Starts with one subtensor and 1 subset of rows of each input factor matrix All-Gathers all the rows needed 2 from U (1) U (1) U (3) M (2) Srinivas Eswar (GT) PLANC January 26, 2019 7 / 12

  12. Distributed Memory Optimisation - Communication Avoiding Each processor Starts with one subtensor and 1 subset of rows of each input factor matrix All-Gathers all the rows needed 2 from U (1) U (1) All-Gathers all the rows needed 3 from U (3) U (3) M (2) Srinivas Eswar (GT) PLANC January 26, 2019 7 / 12

  13. Distributed Memory Optimisation - Communication Avoiding Each processor Starts with one subtensor and 1 subset of rows of each input factor matrix All-Gathers all the rows needed 2 from U (1) U (1) All-Gathers all the rows needed 3 from U (3) Computes its contribution to rows 4 of M (2) (local MTTKRP) U (3) M (2) Srinivas Eswar (GT) PLANC January 26, 2019 7 / 12

  14. Distributed Memory Optimisation - Communication Avoiding Each processor Starts with one subtensor and 1 subset of rows of each input factor matrix All-Gathers all the rows needed 2 from U (1) U (1) All-Gathers all the rows needed 3 from U (3) Computes its contribution to rows 4 of M (2) (local MTTKRP) U (3) M (2) Srinivas Eswar (GT) PLANC January 26, 2019 7 / 12

  15. Distributed Memory Optimisation - Communication Avoiding Each processor Starts with one subtensor and 1 subset of rows of each input factor matrix All-Gathers all the rows needed 2 from U (1) U (1) All-Gathers all the rows needed 3 from U (3) Computes its contribution to rows 4 of M (2) (local MTTKRP) Reduce-Scatters to compute and 5 distribute M (2) evenly U (3) M (2) Srinivas Eswar (GT) PLANC January 26, 2019 7 / 12

  16. Distributed Memory Optimisation - Communication Avoiding Each processor Starts with one subtensor and 1 subset of rows of each input factor matrix All-Gathers all the rows needed 2 from U (1) U (1) All-Gathers all the rows needed 3 from U (3) Computes its contribution to rows 4 of M (2) (local MTTKRP) Reduce-Scatters to compute and 5 distribute M (2) evenly U (3) Solve local NLS problem using 6 M (2) M (2) and U (2) Srinivas Eswar (GT) PLANC January 26, 2019 7 / 12

  17. Performance Plots - Strong Scaling 4D Strong Scaling 50 40 gram Running Time (in Secs) nnls 30 mttkrp multittv reducescatter 20 allgather allreduce 10 0 0-128 0-16 0-32 0-64 1-128 1-16 1-32 1-64 2-128 2-16 2-32 2-64 4-128 4-16 4-32 4-64 5-128 5-16 5-32 5-64 alg-p Figure: Strong Scaling on synthetic tensor of dimensions 256 × 256 × 256 × 256 on 8, 16, 32, 64 and 128 nodes of Titan Can achieve nearly linear scaling since NNCP is compute bound. Srinivas Eswar (GT) PLANC January 26, 2019 8 / 12

  18. Performance Plots - CPU vs GPU MU 400 60.0 HALS ANLS/BPP 57.5 350 AO-ADMM Total Time(in Secs) Total Time(in Secs) Nestrov MU 55.0 300 CP/ALS HALS ANLS/BPP 52.5 250 AO-ADMM Nestrov 50.0 200 CP/ALS 47.5 150 45.0 100 42.5 20 30 40 50 60 70 80 90 100 20 30 40 50 60 70 80 90 100 Low Rank (k) Low Rank (k) Figure: CPU Figure: GPU 4D synthetic tensor of dimensions 384 × 384 × 384 × 384 on 81 Titan nodes as a 3 × 3 × 3 × 3 grid with varying low rank. Offloading DGEMM calls to GPU can provide 7X speedup . Srinivas Eswar (GT) PLANC January 26, 2019 9 / 12

  19. Compiler Challenges and Extensions to the Sparse Setting 1 Dimension Tree ordering. Combinatorial explosion for sparse case (contrasted with the single split choice for the dense case). Sparse case involves growth in intermediate values. 2 Communication Pattern establishment and load balancing. Automatic communicator setup given a processor grid and tensor operation. Automatic data distribution using the communication-avoiding loop optimisation [Kni15, DR16]. 3 Block Parallelism in Least Squares Solvers. Active Set orderings can be grouped in an embarrassingly parallel call. Sparse case with masking matrix has a similar RHS pattern. 4 Binary Bloat Separate binaries for GPU/CPU and Sparse/Dense. Srinivas Eswar (GT) PLANC January 26, 2019 10 / 12

  20. Summary PLANC is an open source, scalable and flexible software package to compute Non-negative Tensor Factorisation. Implements state of the art communication avoiding algorithm for matricised-tensor times Khatri-Rao product (MTTKRP). Popular optimisation methods like Block Principal Pivoting, Alternating Direction Method of Multipliers, first order Nesterov methods etc. for Non-negative Least Squares are included. NTF is an important contributor towards explainable AI with a wide range of applications like spectral unmixing, scientific visualization, healthcare analytics, topic modelling etc. Coming soon as a miniapp on OLCF machines. https://github.com/ramkikannan/planc. Srinivas Eswar (GT) PLANC January 26, 2019 11 / 12

Recommend

Parallel Numerical Algorithms Chapter 6 Matrix Models Section 6.2 Low Rank Approximation

Parallel Numerical Algorithms Chapter 6 Matrix Models Section 6.2 Low Rank Approximation

Low Rank Approximation by SVD Computing Low Rank Approximations Randomness and Approximation Hierarchical Low-Rank Structure Parallel Numerical Algorithms Chapter 6 Matrix Models Section 6.2 Low Rank Approximation Edgar Solomonik

584 views • 37 slides
1 Low-rank approximations to a matrix using SVD First point: we can write the SVD as a sum of

1 Low-rank approximations to a matrix using SVD First point: we can write the SVD as a sum of

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 4 notes: PCA Thurs, 2.15 1 Low-rank approximations to a matrix using SVD First point: we can write the SVD as a sum of rank-1

363 views • 3 slides
Learning with Low Rank Approximations or how to use near separability to extract content from

Learning with Low Rank Approximations or how to use near separability to extract content from

Learning with Low Rank Approximations or how to use near separability to extract content from structured data Jeremy E. Cohen IRISA, INRIA, CNRS, University of Rennes, France 30 April 2019 1/34 1 Introduction: separability and matrix/tensor

609 views • 34 slides
Conditional Gradient Algorithms for Rank-One Matrix Approximations with a Sparsity Constraint

Conditional Gradient Algorithms for Rank-One Matrix Approximations with a Sparsity Constraint

Conditional Gradient Algorithms for Rank-One Matrix Approximations with a Sparsity Constraint Marc Teboulle School of Mathematical Sciences Tel Aviv University Joint work with Ronny Luss Optimization and Statistical Learning OSL 2013

823 views • 48 slides
LR-GLM: High-Dimensional Bayesian Inference Using Low-Rank Data Approximations Brian Trippe ,

LR-GLM: High-Dimensional Bayesian Inference Using Low-Rank Data Approximations Brian Trippe ,

LR-GLM: High-Dimensional Bayesian Inference Using Low-Rank Data Approximations Brian Trippe , Jonathan Huggins, Raj Agrawal, and Tamara Broderick LR-GLM: High-Dimensional Bayesian Inference Using Low-Rank Data Approximations Brian Trippe ,

502 views • 23 slides
Polynomial, sparse and low-rank approximations Anthony Nouy Centrale Nantes Laboratoire de

Polynomial, sparse and low-rank approximations Anthony Nouy Centrale Nantes Laboratoire de

Polynomial, sparse and low-rank approximations Anthony Nouy Centrale Nantes Laboratoire de Mathmatiques Jean Leray RICAM Special Semester on Multivariate Algorithms and their Foundations in Number Theory, Linz, December 14, 2018

1.07k views • 59 slides
2 3 4 5 8 9 MINNEAPOLIS MILWAUKEE MSA RANK #16 MSA RANK #39 CHICAGO MSA RANK #3

2 3 4 5 8 9 MINNEAPOLIS MILWAUKEE MSA RANK #16 MSA RANK #39 CHICAGO MSA RANK #3

2 3 4 5 8 9 MINNEAPOLIS MILWAUKEE MSA RANK #16 MSA RANK #39 CHICAGO MSA RANK #3 INDIANAPOLIS MSA RANK #34 DETROIT MSA RANK #14 COLUMBUS MSA RANK #33 BALTIMORE/ DC CORRIDOR MSA RANK #21 CINCINNATI MSA RANK #28 ATLANTA MSA RANK

726 views • 44 slides
NetGAN without GAN: From Random Walks to Low-Rank Approximations Luca Rendsburg, Holger Heidrich,

NetGAN without GAN: From Random Walks to Low-Rank Approximations Luca Rendsburg, Holger Heidrich,

NetGAN without GAN: From Random Walks to Low-Rank Approximations Luca Rendsburg, Holger Heidrich, Ulrike von Luxburg ICML 2020 1 3 Input 1 8 1 4 1 1 5 6 9 1 2 1 7 1 9 } 3 1 } NetGAN: Generating Graphs via Random Walks 1 8

2.88k views • 67 slides
An Efficient Gauss-Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations Xin Liu

An Efficient Gauss-Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations Xin Liu

An Efficient Gauss-Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations Xin Liu State Key Laboratory of Scientific and Engineering Computing Institute of Computational Mathematics and Scientific/Engineering Computing Academy of

671 views • 29 slides
Parallel Numerical Algorithms Chapter 6 Structured and Low Rank Matrices Section 6.3

Parallel Numerical Algorithms Chapter 6 Structured and Low Rank Matrices Section 6.3

General Nonlinear Optimization Matrix Completion Parallel Numerical Algorithms Chapter 6 Structured and Low Rank Matrices Section 6.3 Numerical Optimization Michael T. Heath and Edgar Solomonik Department of Computer Science

498 views • 23 slides
Understanding parallel analysis methods for rank selection in PCA David Hong Yue Sheng Edgar

Understanding parallel analysis methods for rank selection in PCA David Hong Yue Sheng Edgar

Understanding parallel analysis methods for rank selection in PCA David Hong Yue Sheng Edgar Dobriban Wharton Statistics, University of Pennsylvania Random Matrices and Complex Data Analysis Workshop 10 December 2019 This work was supported

837 views • 72 slides
Parallel Numerical Algorithms Chapter 6 Structured and Low Rank Matrices Section 6.3

Parallel Numerical Algorithms Chapter 6 Structured and Low Rank Matrices Section 6.3

Alternating Least Squares Coordinate Descent Gradient Descent Nonlinear Optimization Parallel Numerical Algorithms Chapter 6 Structured and Low Rank Matrices Section 6.3 Numerical Optimization Michael T. Heath and Edgar Solomonik

559 views • 21 slides
Linear-Complexity Data-Parallel Earth Movers Distance Approximations Kubilay Atasu, Thomas

Linear-Complexity Data-Parallel Earth Movers Distance Approximations Kubilay Atasu, Thomas

Linear-Complexity Data-Parallel Earth Movers Distance Approximations Kubilay Atasu, Thomas Mittelholzer Earth/Word Movers Distance: Discrete Wasserstein Distance The Queen to tour Canada Royal visit to Halifax Canada Halifax

96 views • 5 slides
1 SVD applications: rank, column, row, and null spaces Rank : the rank of a matrix is equal to:

1 SVD applications: rank, column, row, and null spaces Rank : the rank of a matrix is equal to:

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 3A notes: SVD and Linear Systems 1 SVD applications: rank, column, row, and null spaces Rank : the rank of a matrix is equal to:

462 views • 3 slides
/k Content 2/15 1. Introduction 2. Hamming weight 3. Rank weight 4. Extended rank weight

/k Content 2/15 1. Introduction 2. Hamming weight 3. Rank weight 4. Extended rank weight

On defining the generalized rank weight Ruud Pellikaan joint work with Relinde Jurrius Autonomous University Barcelona, 6 November 2014 /k Content 2/15 1. Introduction 2. Hamming weight 3. Rank weight 4. Extended rank weight enumerator

165 views • 15 slides
Optimal parallel repetition for projection games on low threshold rank graphs Madhur Tulsiani,

Optimal parallel repetition for projection games on low threshold rank graphs Madhur Tulsiani,

Optimal parallel repetition for projection games on low threshold rank graphs Madhur Tulsiani, John Wright, Yuan Zhou TTIC CMU CMU Two-prover one-round games Two-prover one-round games Bipartite graph Two-prover one-round games Bipartite

846 views • 68 slides
10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise,

10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise,

10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise, Listwise 10.3. Gathering User Input 10.4. LeToR Evaluation 10.5. Beyond Search Advanced Topics in Information Retrieval / Learning to Rank 2 10.1.

1.01k views • 34 slides
A Semigroup Approach for Weak Approximations with an Application to Infinite Activity L evy

A Semigroup Approach for Weak Approximations with an Application to Infinite Activity L evy

A Semigroup Approach for Weak Approximations with an Application to Infinite Activity L evy Driven SDEs Hideyuki Tanaka 1 and Arturo Kohatsu-Higa 2 November 20, 2008 1 Mitsubishi UFJ 2 Osaka University. Abstract Weak approximations have

789 views • 47 slides
Identification of parallel Wiener- Hammerstein systems with a decoupled static nonlinearity M.

Identification of parallel Wiener- Hammerstein systems with a decoupled static nonlinearity M.

Identification of parallel Wiener- Hammerstein systems with a decoupled static nonlinearity M. Schoukens, K. Tiels, M. Ishteva, J. Schoukens 1 Parallel Wiener-Hammerstein Flexible Simple LTI SNL LTI 2 Identifiability Full rank linear

603 views • 25 slides
A new family of maximum rank distance codes or: Maximum rank distance codes and finite semifields

A new family of maximum rank distance codes or: Maximum rank distance codes and finite semifields

A new family of maximum rank distance codes or: Maximum rank distance codes and finite semifields John Sheekey Universiteit Gent, Belgium Alcoma, March 2015 Rank metric codes A rank metric code is a set C M n ( F ) of n n matrices over

963 views • 92 slides
On nonlinear approximations and the linear hull effect Anne Canteaut Inria, Paris, France joint

On nonlinear approximations and the linear hull effect Anne Canteaut Inria, Paris, France joint

On nonlinear approximations and the linear hull effect Anne Canteaut Inria, Paris, France joint work with Christof Beierle and Gregor Leander ASK 2018, Kolkota Linear approximations 1 Linear approximations Pr[ x + F ( x ) = 0]

828 views • 27 slides
Introduction to rank-based cryptography Philippe Gaborit University of Limoges, France ASCRYPTO

Introduction to rank-based cryptography Philippe Gaborit University of Limoges, France ASCRYPTO

Rank codes : definitions and basic properties Decoding in rank metric Complexity issues : decoding random rank codes Encryption Authentication and signature Introduction to rank-based cryptography Philippe Gaborit University of Limoges,

1.56k views • 115 slides
On the minimum rank of a graph Jisu Jeong June 21, 2013 Jisu Jeong On the minimum rank of a

On the minimum rank of a graph Jisu Jeong June 21, 2013 Jisu Jeong On the minimum rank of a

Minimum rank The minimum rank of a random graph over the binary field An algorithm to decide the minimum rank for fixed k Future work On the minimum rank of a graph Jisu Jeong June 21, 2013 Jisu Jeong On the minimum rank of a graph Minimum

865 views • 27 slides
Three Air Three Airpor ports Rank at ts Rank at the Bottom the Bottom At 20 At 20 minutes FAA

Three Air Three Airpor ports Rank at ts Rank at the Bottom the Bottom At 20 At 20 minutes FAA

Report Findings Three Air Three Airpor ports Rank at ts Rank at the Bottom the Bottom At 20 At 20 minutes FAA caps flights, which limits use At 10 At 10 minutes to match most major airports, improving level of service Air Passengers

809 views • 47 slides

More recommend