Flexible and Type-Safe Skeleton Programming for Heterogeneous - PowerPoint PPT Presentation

EXCESS workshop August 26, 2016 (From HLPP 2016, updated) Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems August Ernstsson Lu Li Christoph Kessler {firstname.lastname}@liu.se Linköping University, Sweden

Overview Background • Skeleton Programming • SkePU 1 Results • SkePU 2 • Readability Survey • Performance Evaluation • Conclusions 2 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

Skeleton Programming 3 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

Skeleton Programming :: Motivation Programming parallel systems is hard! • Resource utilization • Synchronization • Communication • Memory consistency • Different hardware architectures • Heterogeneity 4 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

Skeleton Programming :: Introduction Skeleton programming (algorithmic skeletons) • A high-level parallel programming concept • Inspired by functional programming • Generic computational patterns • Abstracts architecture-specific issues 5 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

Skeleton Programming :: Skeletons Skeletons Parametrizable higher-order constructs • Map Map • Reduce • MapReduce • Scan MapReduce • and others 6 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

Skeleton Programming :: User Functions User functions User-defined operators Mult Add Pow Abs … 7 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

Skeleton Programming :: Example Skeleton parametrization example Dot product operation Mult MapReduce Add 8 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU 9 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU :: Features ✓ Efficient parallel algorithms ✓ Memory management and data movement ✓ Automatic backend selection and tuning 10 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU :: Architecture C++ interface (macros, skeletons, smart containers, …) C++ OpenMP OpenCL CUDA … CPU Multi-core CPU Accelerator GPU … g n i o g n o h c r a e s e r l a n o t i i d d A 11 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU :: Syntax BINARY_FUNC(add, int , a, b, return a + b; Mult ) Map<add> vec_sum( new add); Map Mult Mult Mult Mult Mult vec_sum(v1, v2, result); Mult Mult Mult Mult 12 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU :: Syntax BINARY_FUNC_CONSTANT(add, int , int , a, b, m, return (a + b) % m; Mult ) Map<add> vec_sum( new add); Map Mult Mult vec_sum.setConstant(5); Mult Mult Mult vec_sum(v1, v2, result); Mult Mult Mult Mult 13 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU :: Limitations × Non-intuitive macro syntax × Type-unsafe user functions × Constrained skeleton signatures 14 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

15 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU 2 :: Introduction • Builds on the SkePU 1 runtime and algorithms • New, more native -looking interface (API) • Extra source-to-source translation step • Based on Clang compiler front-end libraries 16 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU 2 :: Syntax int add( int a, int b) Mult { return a + b; } auto vec_sum = Map<2>(add); Map Mult Mult Mult Mult vec_sum(result, v1, v2); Mult Mult Mult Mult Mult 17 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU 2 :: Syntax int add( int a, int b, int m) Mult { return (a + b) % m; } auto vec_sum = Map<2>(add); Map Mult Mult Mult Mult vec_sum(result, v1, v2, 5); Mult Mult Mult Mult Mult 18 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU 2 :: Architecture SkePU program source Parallel Sequential backend runtime Parallel Parallel runtime runtime library runtime source-to-source translator C++11 compiler C++11 compiler Sequential executable Parallel executable 19 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU 2 :: Flexibility • Variable arity on Map and MapReduce skeletons • Index argument (of current Map’d container element) • Uniform arguments • Smart container arguments accessible freely inside user function • Read -only / write -only / read-write copy modes • User function templates 20 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU 2 :: Advanced Example Templates template < typename T> T abs(T input) { R e a d o n l y , return input < 0 ? -input : input; n o c o p y b a c k } template < typename T> T mvmult(Index1D row, const Matrix<T> m, const Vector<T> v) { T res = 0; for (size_t i = 0; i < v.size; ++i) res += m[row.i * m.cols + i] * v[i]; return abs(res); } C h a i n e d u s e r f u n c t i o n s 21 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU 2 :: Type Safety Type safety test-case Reduce skeleton with unary user function SkePU 1, at run time [SKEPU_ERROR] Wrong operator type! Reduce operation require binary user function. SkePU 2, at compile time error: no matching function for call to 'Reduce' auto globalSum = Reduce(plus_f); ^~~~~~~~~~~~~~ note: candidate template ignored: failed template argument deduction Reduce(T(*red)(T, T)) 22 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU 2 :: Experimental Features • User function specialization for backends • Extends SkePU for multi-variant components • ”Call” skeleton • Custom types • Chained user functions • In-line lambda syntax for user functions 23 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU 2 :: Lambda Syntax int add( int a, int b) { return a + b; } auto vec_sum = Map<2>(mult); // ... vec_sum(result, v1, v2); 24 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

SkePU 2 :: Lambda Syntax auto vec_sum = Map<2>([]( int a, int b) { return a + b; }); // ... vec_sum(result, v1, v2); 25 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

Readability Survey 26 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

Readability Survey • Survey was made on a development version of SkePU 2 with a slightly different syntax • Main difference: Used C++11 attributes • [[skepu::userfunction]] on user functions • [[skepu::instance]] on skeleton instances • Reason: Guide the source-to-source translator and generate better error messages 27 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems

Readability :: Simple Example 1 BINARY_FUNC(sum, int , a, b, return a + b; ) Vector< float > vector_sum(Vector< float > &v1, Vector< float > &v2) { Map<sum> vsum( new sum); Vector< float > result(v1.size()); vsum(v1, v2, result); return result; } 2 [[skepu::userfunction]] float sum( float a, float b) { return a + b; } Vector< float > vector_sum(Vector< float > &v1, Vector< float > &v2) { auto vsum [[skepu::instance]] = Map<2>(sum); Vector< float > result(v1.size()); vsum(result, v1, v2); return result; } 28 August Ernstsson – 2016-08-26 SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous Parallel Systems