Introduction Lecture slides for Chapter 1 of Deep Learning - PowerPoint PPT Presentation

Introduction Lecture slides for Chapter 1 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-09-26

x y Cartesian coordinates Polar coordinates Representations Matter θ r Figure 1.1 (Goodfellow 2016)

Depth: Repeated Composition Output CAR PERSON ANIMAL (object identity) 3rd hidden layer (object parts) 2nd hidden layer (corners and contours) 1st hidden layer (edges) Visible layer (input pixels) Figure 1.2 (Goodfellow 2016)

Computational Graphs Element Element σ Set Set + + × Logistic Logistic × × Regression Regression σ w x x x 1 x 1 w 2 w 2 x 2 x 2 w 1 w 1 Figure 1.3 (Goodfellow 2016)

Machine Learning and AI Deep learning Example: Shallow Example: Example: Example: autoencoders Logistic Knowledge MLPs regression bases Representation learning Machine learning AI Figure 1.4 (Goodfellow 2016)

Learning Multiple Components Figure 1.5 Output Mapping from Output Output features Additional Mapping from Mapping from layers of more Output features features abstract features Hand- Hand- Simple designed designed Features features program features Input Input Input Input Deep Classic learning Rule-based machine systems Representation learning (Goodfellow 2016) learning

Organization of the Book Figure 1.6 1. Introduction Part I: Applied Math and Machine Learning Basics 3. Probability and 2. Linear Algebra Information Theory 4. Numerical 5. Machine Learning Computation Basics Part II: Deep Networks: Modern Practices 6. Deep Feedforward Networks 7. Regularization 8. Optimization 9. CNNs 10. RNNs 11. Practical 12. Applications Methodology Part III: Deep Learning Research 13. Linear Factor 15. Representation 14. Autoencoders Models Learning 16. Structured 17. Monte Carlo Probabilistic Models Methods 18. Partition 19. Inference Function 20. Deep Generative Models (Goodfellow 2016)

Historical Waves 0.000250 Frequency of Word or Phrase cybernetics 0.000200 (connectionism + neural networks) 0.000150 0.000100 0.000050 0.000000 1940 1950 1960 1970 1980 1990 2000 Year Figure 1.7 (Goodfellow 2016)

Historical Trends: Growing Datasets 10 9 Dataset size (number examples) Canadian Hansard 10 8 Sports-1M WMT 10 7 ImageNet10k 10 6 Public SVHN Criminals 10 5 ImageNet ILSVRC 2014 10 4 MNIST CIFAR-10 10 3 10 2 T vs. G vs. F Rotated T vs. C Iris 10 1 10 0 1900 1950 1985 2000 2015 Figure 1.8 (Goodfellow 2016)

The MNIST Dataset Figure 1.9 (Goodfellow 2016)

Connections per Neuron 10 4 Human Cat 6 Connections per neuron 7 9 4 Mouse 10 3 2 10 5 8 10 2 Fruit fly 3 1 10 1 1950 1985 2000 2015 Figure 1.10 (Goodfellow 2016)

Number of Neurons Number of neurons (logarithmic scale) 10 11 Human 10 10 17 20 10 9 19 16 Octopus 18 10 8 14 10 7 Frog 11 8 10 6 Bee 3 10 5 Ant 10 4 10 3 Leech 13 10 2 2 12 10 1 1 Roundworm 15 6 9 10 0 10 5 10 − 1 7 4 10 − 2 Sponge 1950 1985 2000 2015 2056 Figure 1.11 (Goodfellow 2016)

Solving Object Recognition 0 . 30 ILSVRC classification error rate 0 . 25 0 . 20 0 . 15 0 . 10 0 . 05 0 . 00 2010 2011 2012 2013 2014 2015 Figure 1.12 (Goodfellow 2016)