intro to deep learning
play

Intro to Deep Learning Slides Credit: Andrej Karapathy, Derek Hoiem, - PowerPoint PPT Presentation

Mar 18, 2023 •365 likes •1.38k views

Intro to Deep Learning Slides Credit: Andrej Karapathy, Derek Hoiem, Marc Aurelio, Yann LeCunn Why this class? Deep Features Have been able to harness the big data in the most efficient and effective manner. Lead to several state of the

  1. Intro to Deep Learning Slides Credit: Andrej Karapathy, Derek Hoiem, Marc Aurelio, Yann LeCunn

  2. Why this class?

  7. Deep Features • Have been able to harness the big data in the most efficient and effective manner. • Lead to several state of the art results!

  9. r o r r E % 5 < : e c n a m r o f r e P 5 1 0 2

  14. What is deep learning?

  19. Space of Visual Features: Non Linear

  29. Deep Neural Networks

  56. Convolution Layer Filters always extend the full depth of the input volume 32x32x3 image 5x5x3 filter 32 Convolve the filter with the image i.e. “slide over the image spatially, computing dot products” 32 3 Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson 12 56

  57. Convolution Layer 32x32x3 image 5x5x3 filter 32 1 number: the result of taking a dot product between the filter and a small 5x5x3 chunk of the image 32 (i.e. 5*5*3 = 75-dimensional dot product + bias) 3 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 13 27 Jan 2016 27 Jan 2016 57

  58. Convolution Layer activation map 32x32x3 image 5x5x3 filter 32 28 convolve (slide) over all spatial locations 28 32 3 1 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 14 27 Jan 2016 27 Jan 2016 58

  62. Closer look at Spatial Dimensions A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 24 62

  63. Closer look at Spatial Dimensions A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 25 27 Jan 2016 27 Jan 2016 63

  64. Closer look at Spatial Dimensions A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 26 64

  65. Closer look at Spatial Dimensions A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter => 5x5 output 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 28 65

  66. Closer look at Spatial Dimensions A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 2 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 29 66

  67. Closer look at Spatial Dimensions A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 2 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 30 27 Jan 2016 27 Jan 2016 67

  68. Closer look at Spatial Dimensions A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 2 => 3x3 output! 7 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 31 27 Jan 2016 27 Jan 2016 68

  69. Closer look at Spatial Dimensions A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 3? doesn’t fit! 7 cannot apply 3x3 filter on 7x7 input with stride 3. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 33 69

  70. N Output size: (N - F) / stride + 1 F e.g. N = 7, F = 3: N F stride 1 => (7 - 3)/1 + 1 = 5 stride 2 => (7 - 3)/2 + 1 = 3 stride 3 => (7 - 3)/3 + 1 = 2.33 :\ Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 34 70

  71. In practice: Common to zero pad the border 0 0 0 0 0 0 e.g. input 7x7 3x3 filter, applied with stride 1 0 pad with 1 pixel border => what is the output? 0 0 0 (recall:) (N - F) / stride + 1 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 35 27 Jan 2016 27 Jan 2016 71

  72. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 43 27 Jan 2016 27 Jan 2016 72

  73. Examples time: Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Output volume size: ? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 39 27 Jan 2016 27 Jan 2016 73

  74. Examples time: Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Output volume size: (32+2*2-5)/1+1 = 32 spatially, so 32x32x10 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 40 27 Jan 2016 27 Jan 2016 74

  75. Examples time: Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Number of parameters in this layer? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 41 27 Jan 2016 27 Jan 2016 75

  76. Examples time: Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Number of parameters in this layer? each filter has 5*5*3 + 1 = 76 params (+1 for bias) => 76*10 = 760 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 42 76

  77. Fully Connected Layer Connects to entire input volume similar to DNN 77

  78. Fully Connected Layer Number of parameters: 10x10x10 connected to 1000 FC 1000 FC to 1000 FC. 78

  79. Special Layers • Pooling • Contrast Normalization

  80. Pooling layer - makes the representations smaller and more manageable - operates over each activation map independently: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 54 27 Jan 2016 27 Jan 2016 80

  82. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 82 56

  83. Local Response Normalization 83

  84. Example Architectures 84

  85. Case Study: LeNet-5 [LeCun et al., 1998] Conv filters were 5x5, applied at stride 1 Subsampling (Pooling) layers were 2x2 applied at stride 2 i.e. architecture is [CONV-POOL-CONV-POOL-CONV-FC] Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 60 27 Jan 2016 27 Jan 2016 85

  86. Imagenet Network

  87. Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11 filters applied at stride 4 => Q: what is the output volume size? Hint: (227-11)/4+1 = 55 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 61 87

  88. Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11 filters applied at stride 4 => Output volume [55x55x96] Q: What is the total number of parameters in this layer? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 62 27 Jan 2016 27 Jan 2016 88

  89. Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11 filters applied at stride 4 => Output volume [55x55x96] Parameters: (11*11*3)*96 = 35K Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson 63 89

Recommend

Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine

Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine learning pipeline Temporal Frame-level Unsupervised Classifier Summary

613 views • 27 slides
CS 4803 / 7643: Deep Learning Topics: Image Classification Supervised Learning view

CS 4803 / 7643: Deep Learning Topics: Image Classification Supervised Learning view

CS 4803 / 7643: Deep Learning Topics: Image Classification Supervised Learning view K-NN Linear Classifier Zsolt Kira Georgia Tech Last Time High-level intro to what deep learning is Fast brief of logistics

1.24k views • 100 slides
Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning?

Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning?

CS6501: Deep Learning for Visual Recognition Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning? Supervised Learning: Classification with K-nearest neighbors Unsupervised Learning: Clustering with

1.36k views • 91 slides
Deep Dream CSC321: Intro to Machine Learning and Neural Networks, Winter 2016 Michael Guerzhoy

Deep Dream CSC321: Intro to Machine Learning and Neural Networks, Winter 2016 Michael Guerzhoy

Deep Dream CSC321: Intro to Machine Learning and Neural Networks, Winter 2016 Michael Guerzhoy Deep Dream Idea: want to exaggerate details in the image that look a little bit like recognizable objects Overview Pick a layer in the

609 views • 5 slides
Full Stack Deep Learning Lecture 1: Intro Pieter Abbeel, Sergey Karayev, Josh Tobin Organizer

Full Stack Deep Learning Lecture 1: Intro Pieter Abbeel, Sergey Karayev, Josh Tobin Organizer

Full Stack Deep Learning Lecture 1: Intro Pieter Abbeel, Sergey Karayev, Josh Tobin Organizer Intros Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L1: Intro 1 Guest Instructors Saturday Sunday Sunday

747 views • 32 slides
Trend de la Trend Benefits and Pitfalls for Deep Learning with Functional Programming HELLO! I

Trend de la Trend Benefits and Pitfalls for Deep Learning with Functional Programming HELLO! I

Trend de la Trend Benefits and Pitfalls for Deep Learning with Functional Programming HELLO! I am Brad Johns I should write something here but Im lazy. Heres my github: https://github.com/bradjohns94 A Quick Intro to Deep Learning

556 views • 34 slides
Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Deep 3D Representation Learning for Visual Computing Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms Conclusion 2 Outline Overview of 3D deep learning Background 3D deep learning tasks 3D deep

1.66k views • 122 slides
CS 4803 / 7643: Deep Learning Topics: Dynamic Programming (Q-Value Iteration)

CS 4803 / 7643: Deep Learning Topics: Dynamic Programming (Q-Value Iteration)

CS 4803 / 7643: Deep Learning Topics: Dynamic Programming (Q-Value Iteration) Reinforcement Learning (Intro, Q-Learning, DQNs) Nirbhay Modhe Georgia Tech Topics well cover Overview of RL RL vs other forms of learning

1.01k views • 65 slides
Machine learning techniques to probe theoretical physics Intro In inSPIRE, search find t

Machine learning techniques to probe theoretical physics Intro In inSPIRE, search find t

Akinori Tanaka (RIKEN AIP/iTHEMS) Machine learning techniques to probe theoretical physics Intro In inSPIRE, search find t machine learning OR deep learning and date 20xx-&gt;20xx+1 year number of results Intro Mainly experimental, a few

908 views • 70 slides
ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP LEARNING EVERYWHERE INTERNET &amp; CLOUD MEDICINE &amp; BIOLOGY MEDIA &amp; ENTERTAINMENT SECURITY &amp; DEFENSE AUTONOMOUS MACHINES Image

307 views • 26 slides
Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications Neurons &amp; Nets dimer Learning &amp; Depth G1E model Feature extraction Theory 1 Layer Nets Neural computation

1.32k views • 45 slides
Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2020 1 Story so far

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2020 1 Story so far

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2020 1 Story so far Neural networks are universal approximators Can model any odd thing Provided they have the right architecture We must train them to approximate

1.89k views • 143 slides
Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 1 Story so far

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 1 Story so far

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 1 Story so far Neural networks are universal approximators Can model any odd thing Provided they have the right architecture We must train them to approximate

1.58k views • 134 slides
Neural Networks: Optimization Part 2 Intro to Deep Learning, Fall 2017 Quiz 3 Quiz 3 Which

Neural Networks: Optimization Part 2 Intro to Deep Learning, Fall 2017 Quiz 3 Quiz 3 Which

Neural Networks: Optimization Part 2 Intro to Deep Learning, Fall 2017 Quiz 3 Quiz 3 Which of the following are necessary conditions for a value x to be a local minimum of a twice differentiable function f defined over the reals having

2.44k views • 171 slides
Deep Learning for Vision Presented by Kevin Matzen Wednesday, April 9, 14 Quick Intro - DNN

Deep Learning for Vision Presented by Kevin Matzen Wednesday, April 9, 14 Quick Intro - DNN

Deep Learning for Vision Presented by Kevin Matzen Wednesday, April 9, 14 Quick Intro - DNN Feed-forward Sparse connectivity (layer to layer) Different layer types Recently popularized for vision [Krizhevsky, et. al. NIPS 2012]

717 views • 42 slides
Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1

Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1

Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1 Quick Recap: Training a network Divergence between desired output and Average over all actual output of net for a given input training instances

1.75k views • 140 slides
Generative Deep Learning Prof. Kuan-Ting Lai 2020/5/12 DeepFake (Intro) Generative Recurrent

Generative Deep Learning Prof. Kuan-Ting Lai 2020/5/12 DeepFake (Intro) Generative Recurrent

Generative Deep Learning Prof. Kuan-Ting Lai 2020/5/12 DeepFake (Intro) Generative Recurrent Networks Douglas Eck (2002), Music Generation using LSTM Alex Graves, Generating Sequences With Recurrent Neural Networks, arXiv (2013),

854 views • 67 slides
Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice Scaling up DL 2 What is Deep Learning? 3 DEEP LEARNING EVERYWHERE INTERNET &amp; CLOUD MEDICINE &amp; BIOLOGY SECURITY &amp; DEFENSE MEDIA &amp;

1.45k views • 39 slides
Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural networks are universal approximators Can model any odd thing Provided they have the right architecture We must train them to approximate

1.37k views • 134 slides
Fully Convolutional Network (FCN) Prof. Seungchul Lee Industrial AI Lab. Deep Learning for

Fully Convolutional Network (FCN) Prof. Seungchul Lee Industrial AI Lab. Deep Learning for

Fully Convolutional Network (FCN) Prof. Seungchul Lee Industrial AI Lab. Deep Learning for Computer Vision: Review Source: 6.S191 Intro. to Deep Learning at MIT 2 Segmentation Segmentation task is different from classification task because

522 views • 15 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI Mission Helping people solve challenging problems using AI and deep learning. Developers, data scientists and engineers Self-driving cars,

1.44k views • 72 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin Clark Deep Learning for NLP 5 years ago No Seq2Seq No Attention No large-scale QA/reading comprehension datasets No TensorFlow or

1.41k views • 92 slides

More recommend