Entangled Polynomial Codes for Secure, Private, and Batch Distributed Matrix Multiplication: Breaking the "Cubic" Barrier QIAN YU (USC) joint work with Salman Avestimehr (USC) IEEE International Symposium on Information Theory (ISIT) July 2020
Backgrounds: Challenges in Modern Distributed Computing Computational Scalability: • Node failure ( stragglers ) • Byzantine attack ( adversaries ) • Privacy breach ( colluding workers ) … colluding stragglers workers adversaries "The Tail at Scale" Google, 2013 Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
An Introduction to Coded Computing Problem Parameters • Input: X • Compute: g(X) N workers • N evaluations of f Coded Computing … f( ) f( ) f( ) Recovery Threshold: # workers ’ results needed for recovering final outputs (equivalent to straggler resiliency and Design Space computation security) • Encoding functions • Decoding functions Goal: minimize recovery threshold given N,g,f Yu et al, NIPS 2017 Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
An Introduction to Coded Computing Coded Computing Computation f - Channel Coded variables – Coded symbols vs Worker – Channel usage Recovery threshold – Communication rate (tolerate errors) Classical Shannon Theory Differences: • Algebraic vs probabilistic or combinatorial Key Challenge: how to design codes so that Require New coding ideas! 1. Computation on coded data is meaningful? • 2. Encoding and Decoding has low complexity Coding complexity Linear codes as a natural assumption Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Solution: Polynomial Coded Computing (PCC) • Basic idea: • Design polynomials for input entries, s.t. the composition of f and polynomials recover the output. • Recovery threshold ≤ degree of composed polynomial + 1 Yu et al, NIPS 2017 • Example: Matrix Multiplication (Column-wise partition) • Input: X = (𝑩 , 𝑪) Compute: g(X)= 𝑩 𝑼 ⋅ 𝑪 • Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Solution: Polynomial Coded Computing (PCC) • Basic idea: • Design polynomials for input entries, s.t. the composition of f and polynomials recover the output. • Recovery threshold ≤ degree of composed polynomial + 1 Yu et al, NIPS 2017 • Example: Matrix Multiplication (Column-wise partition) • Goal: recover all 𝒏𝒐 • Input: X = (𝑩 , 𝑪) pair-wise products Compute: g(X)= 𝑩 𝑼 ⋅ 𝑪 • T B j A i • N evaluations of f= ⋅ Requires 𝒏𝒐 workers 𝒏 submatrices 𝒐 submatrices Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Solution: Polynomial Coded Computing (PCC) • Basic idea: • Design polynomials for input entries, s.t. the composition of f and polynomials recover the output. • Recovery threshold ≤ degree of composed polynomial + 1 Yu et al, NIPS 2017 • Example: Matrix Multiplication (Column-wise partition) • Goal: recover all 𝒏𝒐 • Design polynomials pair-wise products T B j A i Requires 𝒏𝒐 workers Encoding: worker i obtains 𝑏 𝑗 , 𝑐(𝑗) Worker i essentially calculates: First optimal code for Recovery threshold = 𝒏𝒐 non-linear operations! Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Solution: Polynomial Coded Computing (PCC) Recovery threshold Recovery threshold Recovery threshold … First optimal code for non-linear operations! Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Problem Formulation: Block Matrix Multiplication A better design for straggler mitigation Orderwise improvement for Secure, Private, (Entangled polynomial codes) Batch Matrix Multiplication Qian Yu, Mohammad Ali Maddah-Ali, Salman Avestimehr, “ Straggler Mitigation in Distributed Matrix Multiplication: Fundamental Limits and Optimal Coding ” , Jan 2018, ISIT, TIT. Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Problem Formulation: Block Matrix Multiplication A better design for straggler mitigation Orderwise improvement for Secure, Private, (Entangled polynomial codes) Batch Matrix Multiplication p × 𝑛 blocks p × 𝑜 blocks • Flexible tradeoff in computation, storage, communication ෩ 𝑼 ෩ 𝑩 𝒋 ⋅ 𝑪 𝒋 each worker 𝑗 computes • Also studied in Fahim et al, Allerton 2017 Qian Yu, Mohammad Ali Maddah-Ali, Salman Avestimehr , “ Straggler Mitigation in Distributed Matrix Multiplication: Fundamental Limits and Optimal Coding ”, Jan 2018, ISIT, TIT. Qian Yu - Entangled Polynomial Codes (ISIT) July 2020 Qian Yu (USC)
Block Matrix Multiplication (Straggler Mitigation) Best known result p × 𝑛 blocks Theorem (Yu et al, Jan 2018) Entangled polynomial codes achieves Recovery threshold ≤ p × 𝑜 blocks Cubic ver. Sub-cubic ver. Relate works All related works (Cubic interpretation, at least 𝒒𝒏𝒐 workers) • Polydot (Allerton 2017) 𝑛𝑜(2𝑞 − 1) for 𝑛 = 𝑜 • Generalized Polydot (arxiv ver. May 2018) Entangled Polynomial Codes: 𝑞𝑛𝑜 + 𝑞 − 1 Order-wise coding gain for any large p,m,n Qian Yu, Mohammad Ali Maddah-Ali, Salman Avestimehr , “ Straggler Mitigation in Distributed Matrix Multiplication: Fundamental Limits and Optimal Coding ”, Jan 2018, ISIT, TIT. Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Entangled Polynomial Codes - Breaking the “Cubic” Barrier Tensor decomposition since ~1960s Relate works • Cubic interpretation 𝑆(𝑞, 𝑛, 𝑜) = # multiplications needed for multiplying an 𝑛 -by- 𝑞 • matrix by a 𝑞 -by- 𝑜 matrix. recover linear combinations • of 𝒒𝒏𝒐 pair-wise products Bilinear complexity is sub-cubic: 𝑆 𝑞, 𝑛, 𝑜 = 𝑝 𝑞𝑛𝑜 for any T B ik A ij large p,m,n • 𝒋 Lazy proof (special case), 𝑆 2,2,2 = 7, 𝑆 2 𝑦 , 2 𝑦 , 2 𝑦 ≤ 7 𝑦 All requiring at least 𝒒𝒏𝒐 Theorem: does not guarantee straggler resiliency What is missing?: Design codes for subcubic interpretation! Theorem (Yu et al, Jan 2018) Entangled polynomial codes achieves Recovery threshold ≤ What if 𝑆 𝑞, 𝑛, 𝑜 not yet known? Sub-cubic upper bound exists for any large p,m,n Qian Yu, Mohammad Ali Maddah-Ali, Salman Avestimehr, “ Straggler Mitigation in Distributed Matrix Multiplication: Fundamental Limits and Optimal Coding ” , Jan 2018, ISIT, TIT. Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Entangled Polynomial Codes - Breaking the “ Cubic ” Barrier Design codes for subcubic interpretation! Theorem (Yu et al, Jan 2018) Entangled polynomial codes achieves Recovery threshold ≤ Sub-cubic ver. Cubic ver. different coding structures • Not by just plugging in fast matrix multiplication Qian Yu, Mohammad Ali Maddah-Ali, Salman Avestimehr , “ Straggler Mitigation in Distributed Matrix Multiplication: Fundamental Limits and Optimal Coding ”, Jan 2018, ISIT, TIT. Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Entangled Polynomial Codes - Breaking the “Cubic” Barrier Theorem (Entangled Polynomial Codes) All related works (Cubic interpretation, at least 𝒒𝒏𝒐 workers) Entangled Polynomial Codes: Order-wise coding gain for any large p,m,n Unified framework with order-wise improvement Private Matrix Multiplication Fault-Tolerant Computing Coded Computing Entangled Polynomial Batch Matrix Matrix Multiplication Codes Multiplication Secure Matrix Algorithms Multiplication Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Secure, Private, Batch Matrix Multiplication Private Distributed Matrix Multiplication Secure Distributed Matrix Multiplication …and all possible mixtures One/Two-sided secure: Private: One/Two of inputs information theoretically private against 𝑈 colluding workers Query D information theoretically private against any worker Batch Distributed Matrix Multiplication Element-wise multiply two lists of matrices (Length: L) Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Entangled Polynomial Codes - Breaking the “Cubic” Barrier Secure, Private, Batch Matrix Multiplication all possible mixtures Theorem Relate works Entangled polynomial codes achieves • Secure Private Recovery threshold ≤ Θ(𝑆(𝑞, 𝑛, 𝑜)) for all setting Nodehi et al, 2018 • Aliasgari et al, 2019 • Jia and Jafar, 2019 Requiring 𝐩(𝒒𝒏𝒐) workers for all settings … Batch Remark 1 (Polynomial Coded Computing) All built upon cubic interpretation Straggler resiliency comes for free require at least 𝒒𝒏𝒐 workers Remark 2 (Converse) Optimal within a factor of 2 Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Entangled Polynomial Codes - Breaking the “Cubic” Barrier Secure, Private, Batch Matrix Multiplication Example 1: Secure Distributed Matrix Multiplication Secure Distributed Matrix Multiplication Theorem (Secure Distributed Matrix Multiplication) Entangled polynomial codes achieves +𝑈 for one-sided security recovery threshold ≤ 2𝑆 𝑞, 𝑛, 𝑜 − 1 +2𝑈 for two-sided security One/Two-sided secure: One/Two of inputs information theoretically private against 𝑈 colluding workers State of the arts Require at least 𝒒𝒏𝒐 +𝑼 workers for one-sided security +𝟑𝑼 workers for one-sided security Qian Yu - Entangled Polynomial Codes (ISIT) July 2020
Recommend
More recommend
