Distributed TensorFlow CSE545 - Spring 2020 Stony Brook University - PowerPoint PPT Presentation

Distributed TensorFlow CSE545 - Spring 2020 Stony Brook University

Big Data Analytics, The Class Goal: Generalizations A model or summarization of the data. Data Frameworks Algorithms and Analyses Similarity Search Hadoop File System Spark Hypothesis Testing Graph Analysis Streaming Recommendation Systems MapReduce Tensorflow Deep Learning

Limitations of Spark Spark Overview Spark is fast for being so flexible ● Fast: RDDs in memory + Lazy evaluation: optimized chain of operations. ● Flexible: Many transformations -- can contain any custom code.

Limitations of Spark Spark Overview Spark is fast for being so flexible ● Fast: RDDs in memory + Lazy evaluation: optimized chain of operations. ● Flexible: Many transformations -- can contain any custom code. However: ● Hadoop MapReduce can still be better for extreme IO, data that will not fit in memory across cluster.

Limitations of Spark Spark Overview Spark is fast for being so flexible ● Fast: RDDs in memory + Lazy evaluation: optimized chain of operations. ● Flexible: Many transformations -- can contain any custom code. However: ● Hadoop MapReduce can still be better for extreme IO, data that will not fit in memory across cluster. Compute Bound IO Bound MapReduce Spark many numeric computations large files (TBs or PBs) (1s of TBs, 100s of GBs)

Limitations of Spark Spark Overview Spark is fast for being so flexible ● Fast: RDDs in memory + Lazy evaluation: optimized chain of operations. ● Flexible: Many transformations -- can contain any custom code. However: ● Hadoop MapReduce can still be better for extreme IO, data that will not fit in memory across cluster. ● Modern machine learning (esp. Deep learning), a common big data task, requires heavy numeric computation. Compute Bound IO Bound MapReduce Spark (many numeric computations) (large files: TBs or PBs) (1s of TBs, 100s of GBs) * this is the subjective approximation of the instructor as of February 2020. A lot of factors at play.

Limitations of Spark Spark Overview Spark is fast for being so flexible ● Fast: RDDs in memory + Lazy evaluation: optimized chain of operations. ● Flexible: Many transformations -- can contain any custom code. However: ● Hadoop MapReduce can still be better for extreme IO, data that will not fit in memory across cluster. ● Modern machine learning (esp. Deep learning), a common big data task, requires heavy numeric computation. Compute Bound IO Bound MapReduce Spark TensorFlow (many numeric computations) (large files: TBs or PBs) (1s of TBs, 100s of GBs) * this is the subjective approximation of the instructor as of February 2020. A lot of factors at play.

Learning Objectives Spark Overview ● Understand TensorFlow as a data workflow system. ○ Know the key components of TensorFlow. ○ Understand the key concepts of distributed TensorFlow. ● Execute basic distributed tensorflow program. ● Establish a foundation to distribute deep learning models: ● Convolutional Neural Networks ● Recurrent Neural Network (or LSTM, GRU)

What is TensorFlow? Spark Overview A workflow system catered to numerical computation. One view: Like Spark, but uses tensors instead of RDDs .

What is Tensor Flow? Spark Overview A workflow system catered to numerical computation. One view: Like Spark, but uses tensors instead of RDDs . A multi-dimensional matrix (i.stack.imgur.com)

What is Tensor Flow? Spark Overview A workflow system catered to numerical computation. One view: Like Spark, but uses tensors instead of RDDs . A 2-d tensor is just a matrix. 1-d: vector 0-d: a constant / scalar Note: Linguistic ambiguity: Dimensions of a Tensor =/= Dimensions of a Matrix (i.stack.imgur.com)

What is Tensor Flow? Spark Overview A workflow system catered to numerical computation. One view: Like Spark, but uses tensors instead of RDDs . Examples > 2-d : Image definitions in terms of RGB per pixel Image[ row ][ column ][ rgb ] Subject, Verb, Object representation of language: Counts[verb][subject][object]

What is Tensor Flow? Spark Overview A workflow system catered to numerical computation. One view: Like Spark, but uses tensors instead of RDDs . Technically, less abstract than RDDs which could hold tensors as well as many other data structures (dictionaries/HashMaps, Trees, ...etc…). Then, why TensorFlow?

What is TensorFlow? Spark Overview Efficient, high-level built-in linear algebra and machine learning optimization operations (i.e. transformations). enables complex models, like deep learning Technically, less abstract than RDDs which could hold tensors as well as many other data structures (dictionaries/HashMaps, Trees, ...etc…). Then, why TensorFlow?

What is TensorFlow? Spark Overview Efficient, high-level built-in linear algebra and machine learning optimization operations . enables complex models, like deep learning (Bakshi, 2016, “What is Deep Learning? Getting Started With Deep Learning”)

What is TensorFlow? Spark Overview Efficient, high-level built-in linear algebra and machine learning operations . (Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., ... & Ghemawat, S. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 .)

Tensor Flow Spark Overview Operations on tensors are often conceptualized as graphs : (Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., ... & Ghemawat, S. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 .)

Tensor Flow Spark Overview Operations on tensors are often conceptualized as graphs: A simpler example: c =mm(A, B) c = tensorflow.matmul(a, b) a b

Tensor Flow Spark Overview Operations on tensors are often conceptualized as graphs: example: d=b+c e=c+2 a=d ∗ e (Adventures in Machine Learning. Python TensorFlow Tutorial , 2017)

Ingredients of a TensorFlow Spark Overview tensors* operations variables - persistent an abstract computation mutable tensors (e.g. matrix multiply, add) constants - constant executed by device kernels placeholders - from data * technically, still operations graph session devices defines the environment in the specific devices (cpus or gpus) on which to run the which operations run . (like a Spark context) session.

Ingredients of a TensorFlow Spark Overview * technically, operations that work with tensors. tensors* operations ○ tf.Variable(initial_value, name) variables - persistent an abstract computation ○ tf.constant(value, type, name) mutable tensors (e.g. matrix multiply, add) ○ tf.placeholder(type, shape, name) constants - constant executed by device kernels placeholders - from data * technically, still operations graph session devices defines the environment in the specific devices (cpus or gpus) on which to run the which operations run . (like a Spark context) session.

Ingredients of a TensorFlow Spark Overview Operations tensors* operations variables - persistent an abstract computation mutable tensors (e.g. matrix multiply, add) constants - constant executed by device kernels placeholders - from data

Ingredients of a TensorFlow Spark Overview tensors* ● Places operations on devices operations variables - persistent an abstract computation mutable tensors ● Stores the values of variables (when not distributed) (e.g. matrix multiply, add) constants - constant executed by device kernels placeholders - from data ● Carries out execution: eval() or run() graph session devices defines the environment in the specific devices (cpus or gpus) on which to run the which operations run . (like a Spark context) session.

Ingredients of a TensorFlow Spark Overview * technically, operations that work with tensors. tensors* operations variables - persistent an abstract computation mutable tensors (e.g. matrix multiply, add) constants - constant executed by device kernels placeholders - from data graph session devices defines the environment in the specific devices (cpus or gpus) on which to run the which operations run . (like a Spark context) session.

Distributed TensorFlow Spark Overview Typical use-case: (Supervised Machine Learning) Determine weights, W , of a function, f , such that ε is minimized: f ( X|W ) = Y + ε

Distributed TensorFlow Spark Overview Typical use-case: Determine weights, W , of a function, f , such that ε is minimized: f ( X|W ) = Y + ε X 1 X 2 X 3 Y

Distributed TensorFlow Spark Overview Typical use-case: Determine weights, W , of a function, f , such that ε is minimized: f ( X|W ) = Y + ε X 1 X 2 X 3 Y X 1 X 2 X 3 X 4 X 5 X 6 X 7 X 8 X 9 X 10 X 11 X 12 Y X 13 X 14 X 15 ... X m

Distributed TensorFlow Spark Overview Typical use-case: Determine weights, W , of a function, f , such that ε is minimized: f ( X|W ) = Y + ε X 1 X 2 X 3 Y X 1 X 2 X 3 X 4 X 5 X 6 X 7 X 8 X 9 X 10 X 11 X 12 Y f given w 1 , w 2 ...., w p X 13 X 14 X 15 ... X m (typically, p >= m)

Distributed TensorFlow Spark Overview f(X|W) = Ŷ Typical use-case: Y = (X|W) + ε Determine weights, W , of a function, f , such that | ε | is minimized: f ( X|W ) = Y + ε Y = Ŷ + ε ε = Ŷ - Y X 1 X 2 X 3 Y X 1 X 2 X 3 X 4 X 5 X 6 X 7 X 8 X 9 X 10 X 11 X 12 Y f given w 1 , w 2 ...., w p X 13 X 14 X 15 ... X m (typically, p >= m)