csc 411 lecture 11 neural networks ii
play

CSC 411: Lecture 11: Neural Networks II Class based on Raquel - PowerPoint PPT Presentation

Apr 09, 2024 •414 likes •1.42k views

CSC 411: Lecture 11: Neural Networks II Class based on Raquel Urtasun & Rich Zemels lectures Sanja Fidler University of Toronto March 2, 2016 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 1 / 55 Today

  1. CSC 411: Lecture 11: Neural Networks II Class based on Raquel Urtasun & Rich Zemel’s lectures Sanja Fidler University of Toronto March 2, 2016 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 1 / 55

  2. Today Deep learning for Object Recognition Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 2 / 55

  3. Neural Nets for Object Recognition People are very good at recognizing shapes Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 3 / 55

  4. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 3 / 55

  5. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Why is it di ffi cult? Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 3 / 55

  6. Why is it a Problem? Di ffi cult scene conditions [From: Grauman & Leibe] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 4 / 55

  7. Why is it a Problem? Huge within-class variations. Recognition is mainly about modeling variation. [Pic from: S. Lazebnik] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 5 / 55

  8. Why is it a Problem? Tones of classes [Biederman] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 6 / 55

  9. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Some reasons why it is di ffi cult: Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 7 / 55

  10. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Some reasons why it is di ffi cult: I Segmentation: Real scenes are cluttered Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 7 / 55

  11. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Some reasons why it is di ffi cult: I Segmentation: Real scenes are cluttered I Invariances: We are very good at ignoring all sorts of variations that do not a ff ect shape Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 7 / 55

  12. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Some reasons why it is di ffi cult: I Segmentation: Real scenes are cluttered I Invariances: We are very good at ignoring all sorts of variations that do not a ff ect shape I Deformations: Natural shape classes allow variations (faces, letters, chairs) Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 7 / 55

  13. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Some reasons why it is di ffi cult: I Segmentation: Real scenes are cluttered I Invariances: We are very good at ignoring all sorts of variations that do not a ff ect shape I Deformations: Natural shape classes allow variations (faces, letters, chairs) I A huge amount of computation is required Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 7 / 55

  14. How to Deal with Large Input Spaces How can we apply neural nets to images? Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  15. How to Deal with Large Input Spaces How can we apply neural nets to images? Images can have millions of pixels, i.e., x is very high dimensional Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  16. How to Deal with Large Input Spaces How can we apply neural nets to images? Images can have millions of pixels, i.e., x is very high dimensional How many parameters do I have? Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  17. How to Deal with Large Input Spaces How can we apply neural nets to images? Images can have millions of pixels, i.e., x is very high dimensional How many parameters do I have? Prohibitive to have fully-connected layers Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  18. How to Deal with Large Input Spaces How can we apply neural nets to images? Images can have millions of pixels, i.e., x is very high dimensional How many parameters do I have? Prohibitive to have fully-connected layers What can we do? Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  19. How to Deal with Large Input Spaces How can we apply neural nets to images? Images can have millions of pixels, i.e., x is very high dimensional How many parameters do I have? Prohibitive to have fully-connected layers What can we do? We can use a locally connected layer Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  20. Locally Connected Layer Example: 200x200 image 40K hidden units Filter size: 10x10 4M parameters Note: This parameterization is good when input image is registered (e.g., 34 face recognition). Ra Ranzato Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 9 / 55

  21. When Will this Work? When Will this Work? Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 10 / 55

  22. When Will this Work? When Will this Work? This is good when the input is (roughly) registered Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 10 / 55

  23. General Images The object can be anywhere [Slide: Y. Zhu] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 11 / 55

  24. General Images The object can be anywhere [Slide: Y. Zhu] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 12 / 55

  25. General Images The object can be anywhere [Slide: Y. Zhu] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 13 / 55

  26. Locally Connected Layer STATIONARITY? Statistics is similar at different locations Example: 200x200 image 40K hidden units Filter size: 10x10 4M parameters Note: This parameterization is good when input image is registered (e.g., 35 face recognition). Ranzato Ra Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 14 / 55

  27. The replicated feature approach Adopt approach apparently used in monkey visual systems The red connections all Use many di ff erent copies of the same have the same weight. feature detector. I Copies have slightly di ff erent positions. I Could also replicate across scale and orientation. I Tricky and expensive I Replication reduces number of free parameters to be learned. Use several di ff erent feature types, each with its own replicated pool of detectors. 5 I Allows each patch of image to be represented in several ways. Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 15 / 55

  28. Convolutional Neural Net Idea: statistics are similar at di ff erent locations (Lecun 1998) Connect each hidden unit to a small input patch and share the weight across space This is called a convolution layer and the network is a convolutional network Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 16 / 55

  29. Convolutional Layer Ra Ranzato K h n X h n − 1 ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 17 / 55

  30. Convolutional Layer Ra Ranzato K X h n − 1 h n ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 18 / 55

  31. Convolutional Layer Ra Ranzato K X h n − 1 h n ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 19 / 55

  32. Convolutional Layer Ra Ranzato K X h n − 1 h n ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 20 / 55

  33. Convolutional Layer Ra Ranzato K h n X h n − 1 ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 21 / 55

  34. Convolutional Layer Ra Ranzato K h n X h n − 1 ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 22 / 55

  35. Convolutional Layer Learn multiple filters. E.g.: 200x200 image 100 Filters Filter size: 10x10 10K parameters 54 Ra Ranzato Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 23 / 55

  36. Convolutional Layer Figure: Left: CNN, right: Each neuron computes a linear and activation function Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 24 / 55

  37. Convolutional Layer Figure: Left: CNN, right: Each neuron computes a linear and activation function Hyperparameters of a convolutional layer: Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 24 / 55

  38. Convolutional Layer Figure: Left: CNN, right: Each neuron computes a linear and activation function Hyperparameters of a convolutional layer: The number of filters (controls the depth of the output volume) Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 24 / 55

Recommend

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are the commonest type of neural

534 views • 32 slides
Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views • 34 slides
Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Convolutional Neural Networks We will now study a special type of neural networks convolutional neural

662 views • 6 slides
Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu

Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu

Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu courses.engr.illinois.edu/cs447 CS447: Natural Language Processing 1 Todays Lecture Feed-forward neural networks as classifiers simple architecture in which

724 views • 37 slides
Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin

Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin

Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin Nov 21, 2019 1 / 32 In the previous lecture, we derived upper bounds on Rademacher averages of a set of neural networks in terms of norms of weight

640 views • 32 slides
Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr.

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr.

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu Recap Previous Lecture 2 C. Long Lecture

1.12k views • 68 slides
Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent Neural Networks and Bayesian Networks Reference: We will be referring to sections etc of Deep Learning by Yoshua Bengio, Ian J. Goodfellow

974 views • 54 slides
Evolving Neural Networks This lecture is based on Xin Yaos tutorial slides From Evolving

Evolving Neural Networks This lecture is based on Xin Yaos tutorial slides From Evolving

Evolving Neural Networks This lecture is based on Xin Yaos tutorial slides From Evolving Single Neural Networks to Evolving Ensembles presented at CEC 2007. Literature: Evolving Artificial Neural Networks , Xin Yao, Proceedings of the

281 views • 16 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Learning From Data Lecture 22 Neural Networks and Overfitting Approximation vs. Generalization

Learning From Data Lecture 22 Neural Networks and Overfitting Approximation vs. Generalization

Learning From Data Lecture 22 Neural Networks and Overfitting Approximation vs. Generalization Regularization and Early Stopping Minimizing E in More Efficienty M. Magdon-Ismail CSCI 4100/6100 recap: Neural Networks and Fitting the Data

464 views • 15 slides
Decision support systems and machine learning Lecture 11 Lecture 11 p. 1/24 Neural networks:

Decision support systems and machine learning Lecture 11 Lecture 11 p. 1/24 Neural networks:

Decision support systems and machine learning Lecture 11 Lecture 11 p. 1/24 Neural networks: Biological and artificial Consider humans: Properties of artificial neural nets (ANNs): Neuron switching time 0 . 001 second Many

404 views • 36 slides
CS485/685 Lecture 7: Jan 24, 2012 Perceptrons, Neural Networks [B]: Sections 4.1.7, 5.1 CS485/685

CS485/685 Lecture 7: Jan 24, 2012 Perceptrons, Neural Networks [B]: Sections 4.1.7, 5.1 CS485/685

25/01/2012 CS485/685 Lecture 7: Jan 24, 2012 Perceptrons, Neural Networks [B]: Sections 4.1.7, 5.1 CS485/685 (c) 2012 P. Poupart 1 Outline Neural networks Perceptron Supervised learning algorithms for neural networks CS485/685 (c)

326 views • 15 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
Lecture 5 Intro to machine learning and single-layer neural networks Kai Olav Ellefsen This

Lecture 5 Intro to machine learning and single-layer neural networks Kai Olav Ellefsen This

Biologically inspired computing Lecture 5 Intro to machine learning and single-layer neural networks Kai Olav Ellefsen This Lecture 1. Introduction to learning/classification 2. Biological neuron 3. Perceptron and artificial neural networks

585 views • 55 slides
Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture you should understand the following concepts convolutional neural networks (CNN) convolution and its advantage pooling and its advantage 2

750 views • 50 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10

CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10

CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Variable length data Traditional feed forward neural networks can only handle fixed

609 views • 18 slides
Neural Networks for Machine Learning Lecture 5a Why

Neural Networks for Machine Learning Lecture 5a Why

Neural Networks for Machine Learning Lecture 5a Why object recogni:on is difficult Geoffrey Hinton with Ni:sh Srivastava Kevin Swersky Things

978 views • 30 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides
Lecture 7 Introduction to Neural Networks Julia Hockenmaier juliahmr@illinois.edu 3324

Lecture 7 Introduction to Neural Networks Julia Hockenmaier juliahmr@illinois.edu 3324

CS447: Natural Language Processing http://courses.engr.illinois.edu/cs447 Lecture 7 Introduction to Neural Networks Julia Hockenmaier juliahmr@illinois.edu 3324 Siebel Center Lecture 7: Introduction to Neural Networks : 1 t r a

669 views • 28 slides
Introduction CSCE 970 CSCE 970 Lecture 4: Lecture 4: Convolutional Convolutional Neural

Introduction CSCE 970 CSCE 970 Lecture 4: Lecture 4: Convolutional Convolutional Neural

Introduction CSCE 970 CSCE 970 Lecture 4: Lecture 4: Convolutional Convolutional Neural Neural CSCE 970 Lecture 4: Networks Networks Good for data with a grid-like topology Stephen Scott Convolutional Neural Networks Stephen Scott

355 views • 3 slides

More recommend