a functional reboot for deep learning
play

A Functional Reboot for Deep Learning Conal Elliott Target August - PowerPoint PPT Presentation

May 29, 2023 •384 likes •656 views

A Functional Reboot for Deep Learning Conal Elliott Target August 2019 Conal Elliott A Functional Reboot for Deep Learning August 2019 1 / 23 Goal Extract the essence of DL. Shed accidental complexity and artificial limitations, i.e.,

  1. A Functional Reboot for Deep Learning Conal Elliott Target August 2019 Conal Elliott A Functional Reboot for Deep Learning August 2019 1 / 23

  2. Goal Extract the essence of DL. Shed accidental complexity and artificial limitations, i.e., simplify and generalize. Conal Elliott A Functional Reboot for Deep Learning August 2019 2 / 23

  3. Essence Optimization: best element of a set (by objective function). Usually via differentiation and gradient following. For machine learning, sets of functions . Objective function is defined via set of input/output pairs. Conal Elliott A Functional Reboot for Deep Learning August 2019 3 / 23

  4. Accidental complexity in deep learning Conal Elliott A Functional Reboot for Deep Learning August 2019 4 / 23

  5. Accidental complexity in DL (overview) Imperative programming Weak typing Graphs (neural networks ) Layers Tensors/arrays Back propagation Linearity bias Hyper-parameters Manual differentiation Conal Elliott A Functional Reboot for Deep Learning August 2019 5 / 23

  6. Imperative programming Thwarts correctness/dependability (usually “not even wrong”). Thwarts efficiency (parallelism). Unnecessary for expressiveness. Poor fit. DL is math, so express in a math language. Conal Elliott A Functional Reboot for Deep Learning August 2019 6 / 23

  7. Weak typing Requires people to manage detail & consistency. Run-time errors. Conal Elliott A Functional Reboot for Deep Learning August 2019 7 / 23

  8. Graphs (neural networks ) Clutters API, distracting from purpose. Purpose: a representation of functions. We already have a better one: programming language. Can we differentiate? Conal Elliott A Functional Reboot for Deep Learning August 2019 8 / 23

  9. Graphs (neural networks ) Clutters API, distracting from purpose. Purpose: a representation of functions. We already have a better one: programming language. Can we differentiate? An issue of implementation , not language or library definition. Fix accordingly. Conal Elliott A Functional Reboot for Deep Learning August 2019 8 / 23

  10. Layers Strong bias toward sequential composition. Neglects equally important forms: parallel & conditional. Awkward patches: “skip connections”, ResNet, HighwayNet. Don’t patch the problem; eliminate it. Replace with binary sequential, parallel, conditional composition. Conal Elliott A Functional Reboot for Deep Learning August 2019 9 / 23

  11. “Tensors” Really, multi-dimensional arrays. Awkward: imagine you could program only with arrays (Fortran). Unsafe without dependent types. Multiple intents / weakly typed Even as linear maps: meaning of m ˆ n array? Limited: missing almost all differentiable types. Missing more natural & compositional data types, e.g., trees. Conal Elliott A Functional Reboot for Deep Learning August 2019 10 / 23

  12. Back propagation Specialization and rediscovery of reverse-mode auto-diff. Described in terms of graphs. Highly complex due to graph formulation. Stateful: Hinders parallelism/efficiency. High memory use, limiting problem size. Conal Elliott A Functional Reboot for Deep Learning August 2019 11 / 23

  13. Linearity bias “Dense” & “fully connected” mean arbitrary linear transformation. Sprinkle in “activation functions” as exceptions to linearity. Misses simpler and more efficient architectures. Conal Elliott A Functional Reboot for Deep Learning August 2019 12 / 23

  14. Hyper-parameters Same essential purpose as parameters. Different mechanisms for expression and search. Inefficient and ad hoc Conal Elliott A Functional Reboot for Deep Learning August 2019 13 / 23

  15. A functional reboot Conal Elliott A Functional Reboot for Deep Learning August 2019 14 / 23

  16. Values Precision : meaning, reasoning, correctness. Simplicity : practical rigor/dependability. Generality : room to grow; design guidance. Conal Elliott A Functional Reboot for Deep Learning August 2019 15 / 23

  17. Essence Optimization: best element of a set (by objective function). Usually via differentiation and gradient following. For machine learning, sets of functions . Objective function is defined via set of input/output pairs. Conal Elliott A Functional Reboot for Deep Learning August 2019 16 / 23

  18. Optimization Describe a set of values as range of function: f :: p Ñ c . Objective function: q :: c Ñ R . Find argMin p q ˝ f q :: p . When q ˝ f is differentiable, gradient descent can help. Otherwise, other methods. Consider also global optimization, e.g., with interval methods. Conal Elliott A Functional Reboot for Deep Learning August 2019 17 / 23

  19. Learning functions Special case of optimization, where c “ a Ñ b , i.e., f :: p Ñ p a Ñ b q , and q :: p a Ñ b q Ñ R . Objective function often based on sample set S Ď a ˆ b . Measure mis-predictions (loss). Additivity enables parallel, log-time learning step. Conal Elliott A Functional Reboot for Deep Learning August 2019 18 / 23

  20. Differentiable functional programming Directly on Haskell (etc) programs : Not a library/DSEL No graphs/networks/layers Conal Elliott A Functional Reboot for Deep Learning August 2019 19 / 23

  21. Differentiable functional programming Directly on Haskell (etc) programs : Not a library/DSEL No graphs/networks/layers Differentiated at compile time Simple, principled, and general ( The simple essence of automatic differentiation ) Generating efficient run-time code Amenable to massively parallel execution (GPU, etc) Conal Elliott A Functional Reboot for Deep Learning August 2019 19 / 23

  22. Beyond “tensors” Most differentiable types are not vectors (uniform n -tuples), and most derivatives (linear maps) are not matrices. A more general alternative: Free vector space over s : i Ñ s » f s (“ i indexes f ”) Special case: Fin n Ñ s » Vec n s ˆ Algebra of representable functors : f ˆ ˆ g , 1, g ˝ f , Id Your (representable) functor via deriving Generic Linear map p f s ⊸ g s q » g p f s q » p g ˝ f q s (generalized matrix). Other representations for efficient reverse-mode AD (w/o tears). Conal Elliott A Functional Reboot for Deep Learning August 2019 20 / 23

  23. Beyond “tensors” Most differentiable types are not vectors (uniform n -tuples), and most derivatives (linear maps) are not matrices. A more general alternative: Free vector space over s : i Ñ s » f s (“ i indexes f ”) Special case: Fin n Ñ s » Vec n s ˆ Algebra of representable functors : f ˆ ˆ g , 1, g ˝ f , Id Your (representable) functor via deriving Generic Linear map p f s ⊸ g s q » g p f s q » p g ˝ f q s (generalized matrix). Other representations for efficient reverse-mode AD (w/o tears). Use with Functor , Foldable , Traversable , Scannable , etc. No need for special/limited array “reshaping” operations. Compositional and naturally parallel-friendly ( Generic parallel functional programming ) Conal Elliott A Functional Reboot for Deep Learning August 2019 20 / 23

  24. Modularity How to build function families from pieces, as in DL? Category of indexed sets of functions. Extract monolithic function after composing. Other uses, including satisfiability. Prototyped, but problem with GHC type-checker. Conal Elliott A Functional Reboot for Deep Learning August 2019 21 / 23

  25. Progress Simple & efficient reverse-mode AD. Some simple regressions, simple DL, and CNN. Some implementation challenges with robustness. Looking for collaborators, including GHC internals (compiling-to-categories plugin) Background in machine learning and statistics Conal Elliott A Functional Reboot for Deep Learning August 2019 22 / 23

  26. Summary Generalize & simplify DL (more for less). Essence of DL: pure FP with minarg . Generalize from “tensors” (for composition & safety). Collaboration welcome! Conal Elliott A Functional Reboot for Deep Learning August 2019 23 / 23

Recommend

Trend de la Trend Benefits and Pitfalls for Deep Learning with Functional Programming HELLO! I

Trend de la Trend Benefits and Pitfalls for Deep Learning with Functional Programming HELLO! I

Trend de la Trend Benefits and Pitfalls for Deep Learning with Functional Programming HELLO! I am Brad Johns I should write something here but Im lazy. Heres my github: https://github.com/bradjohns94 A Quick Intro to Deep Learning

556 views • 34 slides
Adaptive Activation Network and Functional Regularization for Efficient and Flexible Deep

Adaptive Activation Network and Functional Regularization for Efficient and Flexible Deep

Adaptive Activation Network and Functional Regularization for Efficient and Flexible Deep Multi-Task Learning (AAAI-2020) Reading Group Dec. 11, 2019 Suman Saha (postdoc), Computer Vision Lab @ ETH Zurich M o t t i i v v a a t t i

370 views • 23 slides
Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Deep 3D Representation Learning for Visual Computing Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms Conclusion 2 Outline Overview of 3D deep learning Background 3D deep learning tasks 3D deep

1.66k views • 122 slides
ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY MEDIA & ENTERTAINMENT SECURITY & DEFENSE AUTONOMOUS MACHINES Image

308 views • 26 slides
Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice Scaling up DL 2 What is Deep Learning? 3 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY SECURITY & DEFENSE MEDIA &

1.45k views • 39 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI Mission Helping people solve challenging problems using AI and deep learning. Developers, data scientists and engineers Self-driving cars,

1.44k views • 72 slides
Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization Rie

Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization Rie

Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization Rie Johnson and Tong Zhang RJ Research Consulting Hong Kong University of Science and Technology 1 / 12 Training Deep Neural Networks

458 views • 21 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin Clark Deep Learning for NLP 5 years ago No Seq2Seq No Attention No large-scale QA/reading comprehension datasets No TensorFlow or

1.41k views • 92 slides
An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning Microsoft Research Asia (MSRA) Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. Deep Residual Learning for Image Recognition. arXiv

1.32k views • 54 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides
Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning

Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning

Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning with multi-layer (3~30) neural networks, on a huge training set. State-of-the-art on many AI tasks Computer Vision : Image Classification,

1.53k views • 23 slides
Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL

Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL

Tofu: Parallelizing Deep Learning Systems with Automatic Tiling Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL system is based on dataflow GPU#0 w1 w2 data g1 g2 Forward propagation

405 views • 38 slides
Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations deeplearning.cce2020@gmail.com Deep Networks Intuition Neural networks with multiple hidden layers - Deep networks [Hinton, 2006] Deep Networks Intuition

1.33k views • 28 slides
Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu

Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu

Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu 2020 For the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline Deep Learning Growth, Celebrations, and Limitations

1.1k views • 87 slides
12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Wanli Ouyang, Zsolt Kira, Lawrence Neal, Raymond Yeh, Junting Lou and Teck-Yian Lim Unsupervised Learning in General Unsupervised learning is

1.48k views • 115 slides
Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning

Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning

Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-09-27 Definition Regularization is any modification we make to a learning algorithm that is intended to reduce

179 views • 13 slides
Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples RNNs Some slides borrowed from Fei-Fei Li & Justin Johnson &

1.1k views • 55 slides
Deep learning for NLP: Introduction CS 6956: Deep Learning for NLP Words are a very fantastical

Deep learning for NLP: Introduction CS 6956: Deep Learning for NLP Words are a very fantastical

Deep learning for NLP: Introduction CS 6956: Deep Learning for NLP Words are a very fantastical banquet, just so many strange dishes And yet, we seem to do fine We can understand and generate language effortlessly. Almost 1 Wouldnt it be

1.62k views • 52 slides
Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning for robots? slide from V. Vanhoucke End-to-end Deep Learning for robots? 2010 : Speech Recognition Audio Acoustic Model Phonetic Model

1.05k views • 65 slides
Bayesian Deep Learning Mohd Adnan Problems With Deep Learning What does a model not know?

Bayesian Deep Learning Mohd Adnan Problems With Deep Learning What does a model not know?

Bayesian Deep Learning Mohd Adnan Problems With Deep Learning What does a model not know? Uninterpretable black-boxes Easily fooled (AI safety) Lacks solid mathematical foundation Crucially relies on big dat Why

297 views • 15 slides
Deep Learning on Massively Parallel Processing Databases Frank McQuillan Feb 2019 2 A Brief

Deep Learning on Massively Parallel Processing Databases Frank McQuillan Feb 2019 2 A Brief

Deep Learning on Massively Parallel Processing Databases Frank McQuillan Feb 2019 2 A Brief Introduction to Deep Learning Artificial Intelligence Landscape Deep Learning 4 Example Deep Learning Algorithms Multilayer Recurrent

651 views • 44 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides

More recommend