the fundamentals of deep learning
play

The Fundamentals of Deep Learning Building Blocks Theory with - PowerPoint PPT Presentation

Deep Learning Antecedents Vision Case Study The Fundamentals of Deep Learning Building Blocks Theory with Applications Neural Units Neural Nets Deep Neural Nets Application ConvNets Jon Krohn LSTMs untapt jon@untapt.com


  1. Deep Learning Multi-Layer Perceptron Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  2. Deep Learning Sigmoid Neuron Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  3. Deep Learning tanh Neuron Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  4. Deep Learning ReLU: Rectified Linear Units Nair & Hinton (2010); Maas, Hannun & Ng (2014) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  5. Deep Learning Outline Antecedents Vision Case Study Antecedents 1 Building Blocks Theory Case Study: A History of Biological & Artificial Vision Neural Units Building Blocks Neural Nets Deep Neural Nets Application 2 Theory ConvNets LSTMs Biological & Artificial Neurons untapt Reinforcement Neural Networks Deep Neural Networks Contemporary Applications 3 Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

  6. Deep Learning MNIST LeCun, Cortes & Burges Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  7. Deep Learning Fully-Connected Neural Net Single Hidden Layer Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  8. Deep Learning TensorFlow Playground Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt [demo] Reinforcement

  9. Deep Learning Outline Antecedents Vision Case Study Antecedents 1 Building Blocks Theory Case Study: A History of Biological & Artificial Vision Neural Units Building Blocks Neural Nets Deep Neural Nets Application 2 Theory ConvNets LSTMs Biological & Artificial Neurons untapt Reinforcement Neural Networks Deep Neural Networks Contemporary Applications 3 Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

  10. Deep Learning Deep Fully-Connected Net 3 (or more) Hidden Layers Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  11. Deep Learning TFLearn Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application Caffe Torch Theano TensorFlow ConvNets Language C++, Python Lua Python Python LSTMs untapt Pretrained Yes++ Yes++ Yes (Lasagne) Inception Reinforcement Parallel GPUs: Data Yes Yes Yes Yes Parallel GPUs: Model No Yes Experimental Yes (best) Readable Source Code Yes (C++) Yes No No Good at RNN No Mediocre Yes Yes (best) Higher-Level APIs No No Keras Keras and TFLearn

  12. Deep Learning A Simple Deep Net in TFLearn Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt [notebook] Reinforcement

  13. Deep Learning Synaptic Pruning Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  14. Deep Learning (Stochastic) Gradient Descent Adam = AdaGrad + RMSprop Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  15. Deep Learning Backpropagation computes error & gradient of cost function Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  16. Deep Learning Overfitting ...and avoiding it Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement • L1/L2 regularization • dropout • artificial data set expansion

  17. Deep Learning Improving Neural Networks Mostly Hyperparameter Tuning Antecedents Vision Case Study Building Blocks Theory • problem simplification Neural Units Neural Nets Deep Neural Nets • number and width of layers Application • cost fxn: quadratic, cross-entropy, log-likelihood, &c. ConvNets LSTMs untapt • more epochs, early stopping Reinforcement • clever initialization of weights and biases • learning rate η , variable schedule • regularization parameter λ • mini-batch size • automation, e.g., with Spearmint [Summary Blog Post]

  18. Deep Learning Universality Solve Any Continuous Function (Nielsen, 2015) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  19. Deep Learning Unstable Gradient Typically Vanishes (but can Explode ) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  20. Deep Learning Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  21. Deep Learning Classic Deep Architectures ...introducing Convolutional Layers Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  22. Deep Learning Outline Antecedents Vision Case Study Antecedents 1 Building Blocks Theory Case Study: A History of Biological & Artificial Vision Neural Units Building Blocks Neural Nets Deep Neural Nets Application 2 Theory ConvNets LSTMs Biological & Artificial Neurons untapt Reinforcement Neural Networks Deep Neural Networks Contemporary Applications 3 Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

  23. Deep Learning Hubel & Wiesel (1959) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  24. Deep Learning Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  25. Deep Learning Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  26. Deep Learning AlexNet Krizhevsky, Sutskever & Hinton (2012) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  27. Deep Learning Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  28. Deep Learning ConvNet Visualisation Yosinski et al. (2015) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement [video]

  29. Deep Learning Network Architectures Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  30. Deep Learning AlexNet: ILSVRC ‘12 winner Krizhevsky et al. (2012) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement [TFLearn notebook]

  31. Deep Learning VGGNet: ILSVRC ‘14 runner-up Simonyan & Zisserman (2015) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement [TFLearn notebook]

  32. Deep Learning ConvNet in TensorFlow Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application Caffe Torch Theano TensorFlow ConvNets Language C++, Python Lua Python Python LSTMs untapt Pretrained Yes++ Yes++ Yes (Lasagne) Inception Reinforcement Parallel GPUs: Data Yes Yes Yes Yes Parallel GPUs: Model No Yes Experimental Yes (best) Readable Source Code Yes (C++) Yes No No Good at RNN No Mediocre Yes Yes (best) Higher-Level APIs No No Keras Keras and TFLearn

  33. Deep Learning ConvNet in TensorFlow Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt [notebook] Reinforcement

  34. Deep Learning Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  35. Deep Learning ConvNet in Theano Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application Caffe Torch Theano TensorFlow ConvNets Language C++, Python Lua Python Python LSTMs untapt Pretrained Yes++ Yes++ Yes (Lasagne) Inception Reinforcement Parallel GPUs: Data Yes Yes Yes Yes Parallel GPUs: Model No Yes Experimental Yes (best) Readable Source Code Yes (C++) Yes No No Good at RNN No Mediocre Yes Yes (best) Higher-Level APIs No No Keras Keras and TFLearn

  36. Deep Learning ConvNet in Theano Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt [demo] Reinforcement

  37. Deep Learning ConvNet in Keras calls TensorFlow or Theano Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets Caffe Torch Theano TensorFlow LSTMs untapt Language C++, Python Lua Python Python Reinforcement Pretrained Yes++ Yes++ Yes (Lasagne) Inception Parallel GPUs: Data Yes Yes Yes Yes Parallel GPUs: Model No Yes Experimental Yes (best) Readable Source Code Yes (C++) Yes No No Good at RNN No Mediocre Yes Yes (best) Higher-Level APIs No No Keras Keras and TFLearn

  38. Deep Learning ConvNet in Keras calls TensorFlow or Theano Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement [notebook]

  39. Deep Learning “2.5-dimension” CT Scans Roth et al. (2015) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  40. Deep Learning Computer-Aided Detection Shin et al. (2016); Roth et al. (2016) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  41. Deep Learning Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  42. Deep Learning Kaggle Data Science Bowl 2017 Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement [link]

  43. Deep Learning Transfer Learning Caffe Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets Caffe Torch Theano TensorFlow LSTMs untapt Language C++, Python Lua Python Python Reinforcement Pretrained Yes++ Yes++ Yes (Lasagne) Inception Parallel GPUs: Data Yes Yes Yes Yes Parallel GPUs: Model No Yes Experimental Yes (best) Readable Source Code Yes (C++) Yes No No Good at RNN No Mediocre Yes Yes (best) Higher-Level APIs No No Keras Keras and TFLearn

  44. Deep Learning Transfer Learning Caffe Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement [Model Zoo]

  45. Deep Learning Transfer Learning Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  46. Deep Learning Video Classification Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt [video] Reinforcement

  47. Deep Learning Outline Antecedents Vision Case Study Antecedents 1 Building Blocks Theory Case Study: A History of Biological & Artificial Vision Neural Units Building Blocks Neural Nets Deep Neural Nets Application 2 Theory ConvNets LSTMs Biological & Artificial Neurons untapt Reinforcement Neural Networks Deep Neural Networks Contemporary Applications 3 Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

  48. Deep Learning Sunspring Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  49. Deep Learning Sunspring Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  50. Deep Learning Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  51. Deep Learning Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  52. Deep Learning RNNs; LSTM RNNs Hochreiter & Schmidhuber (1997) Antecedents Graves, ... & Schmidhuber (2009) Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  53. Deep Learning Vector Space Embedding Word2Vec: Mikolov, ... & Dean (2013) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  54. Deep Learning t-SNE Hinton & van der Maaten (2008) Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  55. Deep Learning Word2Vec + t-SNE Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  56. Deep Learning Word2Vec + t-SNE Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  57. Deep Learning ‘Understand’ Language with Word2Vec features in your model Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement [even with small corpora]

  58. Deep Learning Quick, Draw! ConvNet + LSTM Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement [link]

  59. Deep Learning Outline Antecedents Vision Case Study Antecedents 1 Building Blocks Theory Case Study: A History of Biological & Artificial Vision Neural Units Building Blocks Neural Nets Deep Neural Nets Application 2 Theory ConvNets LSTMs Biological & Artificial Neurons untapt Reinforcement Neural Networks Deep Neural Networks Contemporary Applications 3 Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

  60. Deep Learning untapt Digital Recruitment Platform Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  61. Deep Learning untapt Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  62. Deep Learning untapt Candidate-Side Feedback Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  63. Deep Learning untapt Client-Side Feedback Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement

  64. Deep Learning untapt Multi-Stage Bayesian Regression with PyMC3 Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement Krohn, Rives-Corbett & Donner (2016)

  65. Deep Learning untapt Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement Krohn, Rives-Corbett & Donner (2016)

  66. Deep Learning untapt Ensemble with Deep Neural Net Antecedents Vision Case Study Building Blocks Theory Neural Units Neural Nets Deep Neural Nets Application ConvNets LSTMs untapt Reinforcement deep-orange.untapt.com

Recommend


More recommend