Distributed Training on HPC Presented By: Aaron D. Saxton, PhD - PowerPoint PPT Presentation

7/11/19 Distributed Training on HPC Presented By: Aaron D. Saxton, PhD

Statistics Review Simple y = 𝑛 $ 𝑦 + 𝑐 regression • • Least Squares to find m,b With data set { 𝑦 ) , 𝑧 ) } )-.,..,0 • • Very special, often hard to measure 𝑧 ) • Let the error be 0 [(𝑧 ) − 𝑛 $ 𝑦 ) + 𝑐 ] 7 • 𝑆 = ∑ )-. Minimize 𝑆 with respect to 𝑛 and 𝑐 . • • Simultaneously Solve • 𝑆 8 𝑛, 𝑐 = 0 • 𝑆 : (𝑛, 𝑐) = 0 • Linear System We will consider more general 𝑧 = 𝑔(𝑦) • 𝑆 8 𝑛, 𝑐 = 0 and 𝑆 : 𝑛, 𝑐 = 0 may not be linear • 2

Statistics Review • Regressions with parameterized sets of functions. e.g. • 𝑧 = 𝑏𝑦 7 + 𝑐𝑦 + 𝑑 (quadratic) • 𝑧 = ∑ 𝑏 ) 𝑦 ) (polynomial) • 𝑧 = 𝑂𝑓 AB (exponential) . • 𝑧 = .CD E(FGHI) (logistic) 3

Statistics Review • Polynomial model of degree ‘n’ • “degrees of freedom” - models capacity Deep Learning, Goodfellow et. al., MIT Press, http://www.deeplearningbook.org, 2016 4

Gradient Decent • Searching for minimum 𝛼𝑆 = 𝑆 K L , 𝑆 K M , … , 𝑆 K O • 𝑆 ⃗ 𝜄 RC. = 𝑆 ⃗ 𝜄 R + 𝛿𝛼𝑆 • 𝛿: Learning Rate • • Recall, Loss depends on data Expand notation, • 𝑆 ⃗ 𝜄 R ; { 𝑦 ) , 𝑧 ) } 0 Recall 𝑆 and 𝛼𝑆 is a sum over 𝑗 • Intuitively, want 𝑆 with • 0 [(𝑧 ) − 𝑔 ALL DATA ….. ? ( 𝑆 = ∑ )-. K W (𝑦 ) )] 7 ) 5

Gradient Decent 6

Stochastic Gradient Decent 0 [(𝑧 ) − 𝑔 K W (𝑦 ) )] 7 ) Recall 𝑆 is a sum over 𝑗 ( 𝑆 = ∑ )-. • Single training example, 𝑦 ) , 𝑧 ) , Sum over only one training example • 𝛼𝑆 B X ,Y X = 𝑆 K L , 𝑆 K M , … , 𝑆 K O • B X ,Y X ⃗ ⃗ 𝑆 B X ,Y X 𝜄 RC. = 𝑆 B X ,Y X 𝜄 R + 𝛿𝛼𝑆 B X ,Y X • 𝛿: Learning Rate • Choose next 𝑦 )C. , 𝑧 )C. , (Shuffled training set) • • SGD with mini batches Many training example, 𝑦 ) , 𝑧 ) , Sum over many training example • • Batch Size or Mini Batch Size (This gets ambiguous with distributed training) • SGD often outperforms traditional GD, want small batches. • https://arxiv.org/abs/1609.04836, On Large-Batch Training … Sharp Minima • https://arxiv.org/abs/1711.04325, Extremely Large ... in 15 Minutes 7

Neural Networks • Activation functions Logistic ReLU (Rectified Linear Unit) 1.5 1.5 1 𝜏 𝑦 = 𝜏 𝑦 = 1 0.5 0.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 Arctan 2.5 𝜏 𝑦 = -7.5 -5 -2.5 0 2.5 5 7.5 10 -2.5 • Softmax D I] • 𝑕 [ 𝑦 . , 𝑦 7 , … , 𝑦 \ = ∑ D IX 8

Neural Networks • Parameterized function • 𝑎 ` = 𝜏 𝛽 b8 + 𝛽 8 𝑌 • 𝑈 e = 𝛾 b[ + 𝛾 [ 𝑎 • 𝑔 e 𝑌 = 𝑕 [ 𝑈 • Linear Transformations with pointwise evaluation of nonlinear function, 𝜏 𝑈 → 𝑍 𝑌 𝑎 • 𝛾 b) , 𝛾 ) , 𝛽 b8 , 𝛽 8 • Weights to be optimized 9

Faux Model Example 10

Distributed Training, data distributed 11

Distributed Training, data distributed 12

Distributed Training, All Reduce Collective 13

Distributed TensorFlow: Parameter Sever/Worker Default, Bad Way on HPC ps:0 worker:0 worker:1 Aggregate Model worker:2 Update Model Parameters Loss (Cross Entropy) Model ps:1 Loss (Cross Entropy) Optimize (Gradient Decent) Aggregate Loss (Cross Entropy) Optimize (Gradient Decent) Update Parameters Optimize (Gradient Decent) 14

Other models: Sequence Modeling Autoregression • i 𝜚 ) 𝐶 ) 𝑌 R + 𝜗 R 𝑌 R = 𝑑 + ∑ )-. Back Shift Operatior : 𝐶 ) Autocorrelation • 𝑆 {{ (𝑢 . , 𝑢 7 ) = 𝐹[𝑌 R ~ 𝑌 R M ] Other tasks • Semantic Labeling • [art.] [adj.] [adj.] [n.] [v.] [adverb] [art.] [adj.] [adj.] [d.o.] The quick red fox jumps over the lazy brown dog

Recurrent Neural Networks: Sequence Modeling • Few projects use pure RNNs, this example is only for pedagogy • RNN is a model that is as “deep” as the modeled sequence is long • LSTM’s, Gated recurrent unit, • No Model Parallel distributed training on the market (June 2019) 16