Fast, General Parallel Computation for Norm Matloff University of - PowerPoint PPT Presentation

Fast, General Parallel Computation for Machine Learning Robin Elizabeth Yancey and Fast, General Parallel Computation for Norm Matloff University of California at Machine Learning Davis Robin Elizabeth Yancey and Norm Matloff University of California at Davis P2PS Workshop, ICPP 2018

Fast, General Parallel Outline Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis

Fast, General Parallel Outline Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis • Motivation. • Software Alchemy. • Theoretical foundations. • Empirical investigation.

Fast, General Parallel Motivation Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis

Fast, General Parallel Motivation Computation for Machine Learning Robin Elizabeth Yancey and Characteristics of machine learning (ML) algorithms: Norm Matloff University of California at Davis

Fast, General Parallel Motivation Computation for Machine Learning Robin Elizabeth Yancey and Characteristics of machine learning (ML) algorithms: Norm Matloff University of California at • Big Data: in n × p ( cases × features ) dataset, both n Davis AND p large.

Fast, General Parallel Motivation Computation for Machine Learning Robin Elizabeth Yancey and Characteristics of machine learning (ML) algorithms: Norm Matloff University of California at • Big Data: in n × p ( cases × features ) dataset, both n Davis AND p large. • Compute-intensive algorithms: sorting, k-NN, matrix inversion, iteration.

Fast, General Parallel Motivation Computation for Machine Learning Robin Elizabeth Yancey and Characteristics of machine learning (ML) algorithms: Norm Matloff University of California at • Big Data: in n × p ( cases × features ) dataset, both n Davis AND p large. • Compute-intensive algorithms: sorting, k-NN, matrix inversion, iteration. • Not generally embarrassingly parallel (EP).

Fast, General Parallel Motivation Computation for Machine Learning Robin Elizabeth Yancey and Characteristics of machine learning (ML) algorithms: Norm Matloff University of California at • Big Data: in n × p ( cases × features ) dataset, both n Davis AND p large. • Compute-intensive algorithms: sorting, k-NN, matrix inversion, iteration. • Not generally embarrassingly parallel (EP). (An exception: Random Forests – grow different trees within different processes.)

Fast, General Parallel Motivation Computation for Machine Learning Robin Elizabeth Yancey and Characteristics of machine learning (ML) algorithms: Norm Matloff University of California at • Big Data: in n × p ( cases × features ) dataset, both n Davis AND p large. • Compute-intensive algorithms: sorting, k-NN, matrix inversion, iteration. • Not generally embarrassingly parallel (EP). (An exception: Random Forests – grow different trees within different processes.) • Memory problems: The computation may not fit on a single machine (esp. in R or GPUs).

Fast, General Parallel Parallel ML: Desired Properties Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis

Fast, General Parallel Parallel ML: Desired Properties Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis • Simple, easily implementable.

Fast, General Parallel Parallel ML: Desired Properties Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis • Simple, easily implementable. (And easily understood by non-techies.)

Fast, General Parallel Parallel ML: Desired Properties Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis • Simple, easily implementable. (And easily understood by non-techies.) • As general in applicability as possible.

Fast, General Parallel Software Alchemy Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis

Fast, General Parallel Software Alchemy Computation for Machine Learning alchemy: Robin The medieval forerunner of chemistry...concerned Elizabeth Yancey and particularly with attempts to convert base metals into Norm Matloff University of gold... a seemingly magical process of California at Davis transformation...

Fast, General Parallel Software Alchemy (cont’d.) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis

Fast, General Parallel Software Alchemy (cont’d.) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis • “Alchemical”: Converts non-EP problems to statistically equivalent EP problems.

Fast, General Parallel Software Alchemy (cont’d.) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis • “Alchemical”: Converts non-EP problems to statistically equivalent EP problems. • Developed independently by (Matloff, JSS, 2013) and several others.

Fast, General Parallel Software Alchemy (cont’d.) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis • “Alchemical”: Converts non-EP problems to statistically equivalent EP problems. • Developed independently by (Matloff, JSS, 2013) and several others. EP: No programming challenge. :-)

Fast, General Parallel Software Alchemy (cont’d.) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis • “Alchemical”: Converts non-EP problems to statistically equivalent EP problems. • Developed independently by (Matloff, JSS, 2013) and several others. EP: No programming challenge. :-) • Not just Embarrassingly Parallel but also Embarrassingly Simple. :-)

Fast, General Parallel Software Alchemy (cont’d) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis

Fast, General Parallel Software Alchemy (cont’d) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of • Break the data into chunks, one chunk per process. California at Davis

Fast, General Parallel Software Alchemy (cont’d) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of • Break the data into chunks, one chunk per process. California at Davis • Apply the procedure, e.g. neural networks (NNs), to each chunk,

Fast, General Parallel Software Alchemy (cont’d) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of • Break the data into chunks, one chunk per process. California at Davis • Apply the procedure, e.g. neural networks (NNs), to each chunk, using off-the-shelf SERIAL algorithms.

Fast, General Parallel Software Alchemy (cont’d) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of • Break the data into chunks, one chunk per process. California at Davis • Apply the procedure, e.g. neural networks (NNs), to each chunk, using off-the-shelf SERIAL algorithms. • In regression case (continuous response variable) take final estimate as average of the chunked estimates.

Fast, General Parallel Software Alchemy (cont’d) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of • Break the data into chunks, one chunk per process. California at Davis • Apply the procedure, e.g. neural networks (NNs), to each chunk, using off-the-shelf SERIAL algorithms. • In regression case (continuous response variable) take final estimate as average of the chunked estimates. • In classification case (categorical response variable), do “voting.”

Fast, General Parallel Software Alchemy (cont’d) Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of • Break the data into chunks, one chunk per process. California at Davis • Apply the procedure, e.g. neural networks (NNs), to each chunk, using off-the-shelf SERIAL algorithms. • In regression case (continuous response variable) take final estimate as average of the chunked estimates. • In classification case (categorical response variable), do “voting.” • If have some kind of parametric model (incl. NNs), can average the parameter values across chunks.

Fast, General Parallel Theory Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at Davis

Fast, General Parallel Theory Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at • Theorem: Davis

Fast, General Parallel Theory Computation for Machine Learning Robin Elizabeth Yancey and Norm Matloff University of California at • Theorem: Davis Say rows of data matrix are i.i.d., output of procedure asymptotically normal. Then the Software Alchemy estimator is fully statistically efficient, i.e. has the same asymptotic variance.