Sparse Matrix Partitioning, Reordering and Vector Multiplication - PowerPoint PPT Presentation

Sparse Matrix Partitioning, Reordering and Vector Multiplication Sparse Matrix Partitioning, Reordering and Vector Multiplication Albert-Jan Yzelman, Utrecht University (NL) May, 2010 Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel Bulk Synchronous Parallel Bulk Synchronous Parallel 1 Matrix partitioning for SpMV 2 Reordering for Sequential SpMV 3 In relation to PSPIKE 4 Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel Supercomputers... Many different processor types: Reduced Instruction Set chips (RISC), (e.g., IBM Power) Intel Itanium x86-type (your average home PC/laptop) Vector (co-)processors GPUs Stream processors ... Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel Supercomputers... Many different connectivity; Ring All-to-all ethernet InfiniBand Cube Hierarchical Internet ... Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel Programming Model One model; bridging models : Message Passing Interface (MPI) Bulk Synchronous Parallel (BSP) Leslie G. Valiant, A bridging model for parallel computation , Communications of the ACM, Volume 33 (1990), pp. 103–111 Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel Bulk Synchronous Parallel A BSP-computer: consists of P processors, each with local memory executes a Single Program on Multiple Data (SPMD) performs no communication during calculation communicates only during barrier synchronisation Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel Superstep 0 Sync Superstep 1 Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel Bulk Synchronous Parallel A BSP-computer furthermore: has homogenous processors, able to do r flops each second takes l time to synchronise has a communication speed of g The model thus only uses four parameters ( P , r , l , g ). Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel Bulk Synchronous Parallel A BSP- algorithm can (using BSPlib, BSPonMPI): Ask for some environment variables: bsp_nprocs() bsp_pid() Synchronise: bsp_sync() Perform “direct” remote memory access (DRMA): bsp_put(source, dest, dest_PID) bsp_get(source, source_PID, dest) Send messages, synchronously (BSMP): bsp_send(data, dest_PID) bsp_move() Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel Exercise Design a parallel inner-product calculation: double spmd_ip( double *x, double *y, int length ) { double sum=... ... return sum; } Using: bsp_nprocs(), bsp_pid(), bsp_put(...) Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel Exercise Design a parallel inner-product calculation: double spmd_ip( double *x, double *y, int length ) { int i = 0; double sum = x[0]*y[0]; double res[ bsp_nprocs() ]; for(i=1; i<length; i++) sum += x[i]*y[i]; for(i=0; i<bsp_nprocs(); i++) bsp_put(&sum,&res[bsp_pid()],i); bsp_sync(); for(i=0; i<bsp_nprocs(); i++) sum += res[i]; return sum; } Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel BSP cost model: let w ( s ) be the work to be done by processor s in superstep i , i let r ( s ) be the communication received by processor s between i superstep i and i + 1, let t ( s ) be the communication transmitted by processor s . i Furthermore, let T be the total number of supersteps and the communication bound of superstep i be given by � � s ∈ [0 , P − 1] r ( s ) s ∈ [0 , P − 1] t ( s ) c i = max max max , . i i Similarly, the upper bound for the amount of work: s ∈ [0 , P − 1] w ( s ) w i = max . i Then the cost of a BSP algorithm is given by: T T − 1 � � w i + ( l + g · c i ) i =0 i =0 Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Bulk Synchronous Parallel Exercise Now think on how to do Sparse Matrix–Vector multiplication (SpMV) using BSP. Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Matrix partitioning for SpMV Matrix partitioning for SpMV Bulk Synchronous Parallel 1 Matrix partitioning for SpMV 2 Reordering for Sequential SpMV 3 In relation to PSPIKE 4 Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Matrix partitioning for SpMV Sparse matrix, dense vector multiplication y=Ax : for each nonzero k from A add x [ k . column ] · k . value to y [ k . row ] �� Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication > Matrix partitioning for SpMV Sparse matrix, dense vector multiplication (parallel) Step 1 ( fan-out ): Not all processors have the elements from x they need; processors need to get the missing items. �� Albert-Jan Yzelman, Utrecht University (NL)

Sparse Matrix Partitioning, Reordering and Vector Multiplication - PowerPoint PPT Presentation

Sparse Matrix Partitioning, Reordering and Vector Multiplication Sparse Matrix Partitioning, Reordering and Vector Multiplication Albert-Jan Yzelman, Utrecht University (NL) May, 2010 Albert-Jan Yzelman, Utrecht University (NL) Sparse Matrix

Sparse matrix partitioning, ordering, and visualisation by Mondriaan 3.0 Outline Partitioning

Parallel Sparse Matrix-Vector and Matrix- Transpose-Vector Multiplication using Compressed Sparse

Exploiting Matrix Reuse and Data Locality in Sparse Matrix-Vector and Matrix-Transpose-Vector

Investigating hypergraph-partitioning-based sparse matrix partitioning methods Bora U car

Fast sparse matrixvector multiplication by partitioning and reordering Albert-Jan Yzelman

Fast sparse matrixvector multiplication by partitioning and reordering Albert-Jan Yzelman

Exploiting GPU Caches in Sparse Matrix Vector Multiplication Yusuke Nagasaka Tokyo Institute of

Vector FPGA Acceleration of 1-D DWT Computations using Sparse Matrix Skeletons Sidharth

Benchmarking Sparse Matrix-Vector Multiply In 5 Minutes Hormozd Gahvari, Mark Hoemmen, James

A Hybrid 2D Method for Sparse Matrix Partitioning Rob Bisseling, Tristan van Leeuwen Utrecht

A Hybrid 2D Method for Sparse Matrix Partitioning Rob Bisseling, Tristan van Leeuwen Utrecht

Mondriaan, partitioning software for sparse matrix computations Rob Bisseling and Brendan

Op#miza#on of Block Sparse Matrix- Vector Mul#plica#on on Shared

Performance Optimizations and Bounds for Sparse Matrix-Vector Multiply Richard Vuduc, James

Optimizations & Bounds for Sparse Symmetric Matrix-Vector Multiply Berkeley Benchmarking and

The Sparse Matrix Vector Product on High-End GPUs SIAM Conference on Parallel Processing for

A structure-driven performance analysis of sparse matrix-vector multiplication Prabhjot Sandhu ,

On partitioning and reordering problems in a hierarchically parallel hybrid linear solver

CS 5220: Graph Partitioning David Bindel 2017-11-07 1 Reminder: Sparsity and partitioning 1 2

Optimizing Sparse Matrix Vector Multiplication on Emerging Multicores Orhan Kislal, Wei Ding,

Performance Models for Evaluation and Automatic Tuning of Symmetric Sparse Matrix-Vector Multiply

Column-Based Matrix Partitioning for Parallel Matrix Multiplication on Heterogeneous Processors

Matrix and Vector Operations Matrix and Vector Operations 1 / 21 Matrix and Vector Operations

Alternating paths and US patent #5,905,666 Alan J. Hoffman John A. Tomlin William R.

Sparse Matrix Partitioning, Reordering and Vector Multiplication - PowerPoint PPT Presentation

Sparse Matrix Partitioning, Reordering and Vector Multiplication Sparse Matrix Partitioning, Reordering and Vector Multiplication Albert-Jan Yzelman, Utrecht University (NL) May, 2010 Albert-Jan Yzelman, Utrecht University (NL) Sparse Matrix

Sparse matrix partitioning, ordering, and visualisation by Mondriaan 3.0 Outline Partitioning

Parallel Sparse Matrix-Vector and Matrix- Transpose-Vector Multiplication using Compressed Sparse

Exploiting Matrix Reuse and Data Locality in Sparse Matrix-Vector and Matrix-Transpose-Vector

Investigating hypergraph-partitioning-based sparse matrix partitioning methods Bora U car

Fast sparse matrixvector multiplication by partitioning and reordering Albert-Jan Yzelman

Fast sparse matrixvector multiplication by partitioning and reordering Albert-Jan Yzelman

Exploiting GPU Caches in Sparse Matrix Vector Multiplication Yusuke Nagasaka Tokyo Institute of

Vector FPGA Acceleration of 1-D DWT Computations using Sparse Matrix Skeletons Sidharth

Benchmarking Sparse Matrix-Vector Multiply In 5 Minutes Hormozd Gahvari, Mark Hoemmen, James

A Hybrid 2D Method for Sparse Matrix Partitioning Rob Bisseling, Tristan van Leeuwen Utrecht

A Hybrid 2D Method for Sparse Matrix Partitioning Rob Bisseling, Tristan van Leeuwen Utrecht

Mondriaan, partitioning software for sparse matrix computations Rob Bisseling and Brendan

Op#miza#on of Block Sparse Matrix- Vector Mul#plica#on on Shared

Performance Optimizations and Bounds for Sparse Matrix-Vector Multiply Richard Vuduc, James

Optimizations &amp; Bounds for Sparse Symmetric Matrix-Vector Multiply Berkeley Benchmarking and

The Sparse Matrix Vector Product on High-End GPUs SIAM Conference on Parallel Processing for

A structure-driven performance analysis of sparse matrix-vector multiplication Prabhjot Sandhu ,

On partitioning and reordering problems in a hierarchically parallel hybrid linear solver

CS 5220: Graph Partitioning David Bindel 2017-11-07 1 Reminder: Sparsity and partitioning 1 2

Optimizing Sparse Matrix Vector Multiplication on Emerging Multicores Orhan Kislal, Wei Ding,

Performance Models for Evaluation and Automatic Tuning of Symmetric Sparse Matrix-Vector Multiply

Column-Based Matrix Partitioning for Parallel Matrix Multiplication on Heterogeneous Processors

Matrix and Vector Operations Matrix and Vector Operations 1 / 21 Matrix and Vector Operations

Alternating paths and US patent #5,905,666 Alan J. Hoffman John A. Tomlin William R.

Optimizations & Bounds for Sparse Symmetric Matrix-Vector Multiply Berkeley Benchmarking and