mean field theory of two layers neural networks dimension
play

Mean-field theory of two-layers neural networks: dimension-free - PowerPoint PPT Presentation

Feb 27, 2024 •820 likes •1.12k views

Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei, Theodor Misiakiewicz, and Andrea Montanari Stanford University June 26, 2019 COLT 2019 Song Mei (Stanford University) Mean Field Dynamics for

  1. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei, Theodor Misiakiewicz, and Andrea Montanari Stanford University June 26, 2019 COLT 2019 Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 1 / 12

  2. Gradient dynamics of two-layers neural network ◮ Two layers neural network: θ ✐ ❂ ✭ ❛ ✐ ❀ w ✐ ✮ ✷ R ❉ ✿ Θ ❂✭ θ ✶ ❀ ✿ ✿ ✿ ❀ θ ◆ ✮ ❀ ◆ ② ✭ x ❀ Θ ✮ ❂ ✶ ❳ ❫ ❛ ✐ ✛ ✭ ❤ w ✐ ❀ x ✐ ✮ ✿ ◆ ✐ ❂✶ ◮ Risk function: ◆ ② � ✶ ✑ ✷ ✐ ❤✏ ❳ ❘ ◆ ✭ Θ ✮ ❂ E x ❀② ❛ ✐ ✛ ✭ ❤ w ✐ ❀ x ✐ ✮✮ ✿ ◆ ✐ ❂✶ ◮ SGD/gradient flow: Θ ❦ ✰✶ ❂ Θ ❦ � ✑ ❦ r ❵ ◆ ✭ Θ ❦ ❀ x ❦ ❀ ② ❦ ✮ ❀ ❞ ❞ t Θ t ❂ � r ❘ ◆ ✭ Θ t ✮ ✿ Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 2 / 12

  3. Gradient dynamics of two-layers neural network ◮ Two layers neural network: θ ✐ ❂ ✭ ❛ ✐ ❀ w ✐ ✮ ✷ R ❉ ✿ Θ ❂✭ θ ✶ ❀ ✿ ✿ ✿ ❀ θ ◆ ✮ ❀ ◆ ② ✭ x ❀ Θ ✮ ❂ ✶ ❳ ❫ ❛ ✐ ✛ ✭ ❤ w ✐ ❀ x ✐ ✮ ✿ ◆ ✐ ❂✶ ◮ Risk function: ◆ ② � ✶ ✑ ✷ ✐ ❤✏ ❳ ❘ ◆ ✭ Θ ✮ ❂ E x ❀② ❛ ✐ ✛ ✭ ❤ w ✐ ❀ x ✐ ✮✮ ✿ ◆ ✐ ❂✶ ◮ SGD/gradient flow: Θ ❦ ✰✶ ❂ Θ ❦ � ✑ ❦ r ❵ ◆ ✭ Θ ❦ ❀ x ❦ ❀ ② ❦ ✮ ❀ ❞ ❞ t Θ t ❂ � r ❘ ◆ ✭ Θ t ✮ ✿ Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 2 / 12

  4. Gradient dynamics of two-layers neural network ◮ Two layers neural network: θ ✐ ❂ ✭ ❛ ✐ ❀ w ✐ ✮ ✷ R ❉ ✿ Θ ❂✭ θ ✶ ❀ ✿ ✿ ✿ ❀ θ ◆ ✮ ❀ ◆ ② ✭ x ❀ Θ ✮ ❂ ✶ ❳ ❫ ❛ ✐ ✛ ✭ ❤ w ✐ ❀ x ✐ ✮ ✿ ◆ ✐ ❂✶ ◮ Risk function: ◆ ② � ✶ ✑ ✷ ✐ ❤✏ ❳ ❘ ◆ ✭ Θ ✮ ❂ E x ❀② ❛ ✐ ✛ ✭ ❤ w ✐ ❀ x ✐ ✮✮ ✿ ◆ ✐ ❂✶ ◮ SGD/gradient flow: Θ ❦ ✰✶ ❂ Θ ❦ � ✑ ❦ r ❵ ◆ ✭ Θ ❦ ❀ x ❦ ❀ ② ❦ ✮ ❀ ❞ ❞ t Θ t ❂ � r ❘ ◆ ✭ Θ t ✮ ✿ Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 2 / 12

  5. Two-layers neural networks Input layer Hidden layer Output layer w 1 a 1 w 2 a 2 w 3 a 3 a 4 w 4 Figure: Architecture for ◆ ❂ ✹ . θ ✐ ❂ ✭ ❛ ✐ ❀ w ✐ ✮ Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 3 / 12

  6. Related literatures ◮ Mean field distributional dynamics: ❅ t ✚ t ✭ θ ✮ ❂ r ✁ ✭ r ✠✭ θ ❀ ✚ t ✮ ✚ t ✮ ✿ ◮ Non-linear dynamics. Converges in some cases. ◮ [Mei, Montanari, Nguyen, 2018], [Rotskoff and Vanden-Eijnden, 2018], [Chizat and Bach, 2018a], [Sirignano and Spiliopoulos, 2018]. ◮ Neural tangent kernel (NTK) dynamics: ❅ t ❦ u t ❦ ✷ ✷ ❂ �❤ u t ❀ ❍ u t ✐ ✿ ◮ Linear dynamics. Always converges to ✵ empirical risk. ◮ [Jacot, Gabriel, and Clement, 2018], [Li and Liang, 2018], [Du, Zhai, Poczos, Singh, 2018]. Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 4 / 12

  7. Related literatures ◮ Mean field distributional dynamics: ❅ t ✚ t ✭ θ ✮ ❂ r ✁ ✭ r ✠✭ θ ❀ ✚ t ✮ ✚ t ✮ ✿ ◮ Non-linear dynamics. Converges in some cases. ◮ [Mei, Montanari, Nguyen, 2018], [Rotskoff and Vanden-Eijnden, 2018], [Chizat and Bach, 2018a], [Sirignano and Spiliopoulos, 2018]. ◮ Neural tangent kernel (NTK) dynamics: ❅ t ❦ u t ❦ ✷ ✷ ❂ �❤ u t ❀ ❍ u t ✐ ✿ ◮ Linear dynamics. Always converges to ✵ empirical risk. ◮ [Jacot, Gabriel, and Clement, 2018], [Li and Liang, 2018], [Du, Zhai, Poczos, Singh, 2018]. Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 4 / 12

  8. Related literatures ◮ Mean field distributional dynamics: ❅ t ✚ t ✭ θ ✮ ❂ r ✁ ✭ r ✠✭ θ ❀ ✚ t ✮ ✚ t ✮ ✿ ◮ Non-linear dynamics. Converges in some cases. ◮ [Mei, Montanari, Nguyen, 2018], [Rotskoff and Vanden-Eijnden, 2018], [Chizat and Bach, 2018a], [Sirignano and Spiliopoulos, 2018]. ◮ Neural tangent kernel (NTK) dynamics: ❅ t ❦ u t ❦ ✷ ✷ ❂ �❤ u t ❀ ❍ u t ✐ ✿ ◮ Linear dynamics. Always converges to ✵ empirical risk. ◮ [Jacot, Gabriel, and Clement, 2018], [Li and Liang, 2018], [Du, Zhai, Poczos, Singh, 2018]. Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 4 / 12

  9. Related literatures ◮ Mean field distributional dynamics: ❅ t ✚ t ✭ θ ✮ ❂ r ✁ ✭ r ✠✭ θ ❀ ✚ t ✮ ✚ t ✮ ✿ ◮ Non-linear dynamics. Converges in some cases. ◮ [Mei, Montanari, Nguyen, 2018], [Rotskoff and Vanden-Eijnden, 2018], [Chizat and Bach, 2018a], [Sirignano and Spiliopoulos, 2018]. ◮ Neural tangent kernel (NTK) dynamics: ❅ t ❦ u t ❦ ✷ ✷ ❂ �❤ u t ❀ ❍ u t ✐ ✿ ◮ Linear dynamics. Always converges to ✵ empirical risk. ◮ [Jacot, Gabriel, and Clement, 2018], [Li and Liang, 2018], [Du, Zhai, Poczos, Singh, 2018]. Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 4 / 12

  10. Related literatures ◮ Mean field distributional dynamics: ❅ t ✚ t ✭ θ ✮ ❂ r ✁ ✭ r ✠✭ θ ❀ ✚ t ✮ ✚ t ✮ ✿ ◮ Non-linear dynamics. Converges in some cases. ◮ [Mei, Montanari, Nguyen, 2018], [Rotskoff and Vanden-Eijnden, 2018], [Chizat and Bach, 2018a], [Sirignano and Spiliopoulos, 2018]. ◮ Neural tangent kernel (NTK) dynamics: ❅ t ❦ u t ❦ ✷ ✷ ❂ �❤ u t ❀ ❍ u t ✐ ✿ ◮ Linear dynamics. Always converges to ✵ empirical risk. ◮ [Jacot, Gabriel, and Clement, 2018], [Li and Liang, 2018], [Du, Zhai, Poczos, Singh, 2018]. Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 4 / 12

  11. Related literatures ◮ Mean field distributional dynamics: ❅ t ✚ t ✭ θ ✮ ❂ r ✁ ✭ r ✠✭ θ ❀ ✚ t ✮ ✚ t ✮ ✿ ◮ Non-linear dynamics. Converges in some cases. ◮ [Mei, Montanari, Nguyen, 2018], [Rotskoff and Vanden-Eijnden, 2018], [Chizat and Bach, 2018a], [Sirignano and Spiliopoulos, 2018]. ◮ Neural tangent kernel (NTK) dynamics: ❅ t ❦ u t ❦ ✷ ✷ ❂ �❤ u t ❀ ❍ u t ✐ ✿ ◮ Linear dynamics. Always converges to ✵ empirical risk. ◮ [Jacot, Gabriel, and Clement, 2018], [Li and Liang, 2018], [Du, Zhai, Poczos, Singh, 2018]. Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 4 / 12

  12. This work (a) Improved bound for SGD - PDE interpolation. (b) Relationship of the mean field limit and the kernel limit. Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 5 / 12

  13. SGD and distributional dynamics (DD) ◮ SGD for Θ ❦ , with ✭ x ❦ ❀ ② ❦ ✮ ✘ P x ❀② , ✐ ✷ ❬ ◆ ❪ , θ ❦ ✰✶ ❂ θ ❦ ✐ � ✷ s ❦ ◆ r θ ✐ ❵ ◆ ✭ Θ ❦ ❀ x ❦ ❀ ② ❦ ✮ ✿ (SGD) ✐ ◮ [MMN18]: s ❦ ❂ ✧✘ ✭ ❦✧ ✮ , ❦ ❂ t❂✧ , ◆ ✦ ✶ , ✧ ✦ ✵ : ◆ ✑ ✶ ✚ ✭ ◆ ✮ ✐ ✮ ✚ t ✷ P ✭ R ❉ ✮ ✂ ❬✵ ❀ ✶ ✮ ✿ ❳ ❫ ✍ θ ❦ ❦ ◆ ✐ ❂✶ ◮ Distributional dynamics (DD) for ✚ t , ❅ t ✚ t ✭ θ ✮ ❂✷ ✘ ✭ t ✮ r θ ✁ ✭ ✚ t ✭ θ ✮ r θ ✠✭ θ ❀ ✚ t ✮✮ ❀ (DD) where ✠✭ θ ❀ ✚ ✮ ❂ ✍❘ ✭ ✚ ✮ ❩ ❯ ✭ θ ❀ θ ✵ ✮ ✚ ✭❞ θ ✵ ✮ ✿ ✍✚ ✭ θ ✮ ❂ ❱ ✭ θ ✮ ✰ Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 6 / 12

  14. SGD and distributional dynamics (DD) ◮ SGD for Θ ❦ , with ✭ x ❦ ❀ ② ❦ ✮ ✘ P x ❀② , ✐ ✷ ❬ ◆ ❪ , θ ❦ ✰✶ ❂ θ ❦ ✐ � ✷ s ❦ ◆ r θ ✐ ❵ ◆ ✭ Θ ❦ ❀ x ❦ ❀ ② ❦ ✮ ✿ (SGD) ✐ ◮ [MMN18]: s ❦ ❂ ✧✘ ✭ ❦✧ ✮ , ❦ ❂ t❂✧ , ◆ ✦ ✶ , ✧ ✦ ✵ : ◆ ✑ ✶ ✚ ✭ ◆ ✮ ✐ ✮ ✚ t ✷ P ✭ R ❉ ✮ ✂ ❬✵ ❀ ✶ ✮ ✿ ❳ ❫ ✍ θ ❦ ❦ ◆ ✐ ❂✶ ◮ Distributional dynamics (DD) for ✚ t , ❅ t ✚ t ✭ θ ✮ ❂✷ ✘ ✭ t ✮ r θ ✁ ✭ ✚ t ✭ θ ✮ r θ ✠✭ θ ❀ ✚ t ✮✮ ❀ (DD) where ✠✭ θ ❀ ✚ ✮ ❂ ✍❘ ✭ ✚ ✮ ❩ ❯ ✭ θ ❀ θ ✵ ✮ ✚ ✭❞ θ ✵ ✮ ✿ ✍✚ ✭ θ ✮ ❂ ❱ ✭ θ ✮ ✰ Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 6 / 12

  15. SGD and distributional dynamics (DD) ◮ SGD for Θ ❦ , with ✭ x ❦ ❀ ② ❦ ✮ ✘ P x ❀② , ✐ ✷ ❬ ◆ ❪ , θ ❦ ✰✶ ❂ θ ❦ ✐ � ✷ s ❦ ◆ r θ ✐ ❵ ◆ ✭ Θ ❦ ❀ x ❦ ❀ ② ❦ ✮ ✿ (SGD) ✐ ◮ [MMN18]: s ❦ ❂ ✧✘ ✭ ❦✧ ✮ , ❦ ❂ t❂✧ , ◆ ✦ ✶ , ✧ ✦ ✵ : ◆ ✑ ✶ ✚ ✭ ◆ ✮ ✐ ✮ ✚ t ✷ P ✭ R ❉ ✮ ✂ ❬✵ ❀ ✶ ✮ ✿ ❳ ❫ ✍ θ ❦ ❦ ◆ ✐ ❂✶ ◮ Distributional dynamics (DD) for ✚ t , ❅ t ✚ t ✭ θ ✮ ❂✷ ✘ ✭ t ✮ r θ ✁ ✭ ✚ t ✭ θ ✮ r θ ✠✭ θ ❀ ✚ t ✮✮ ❀ (DD) where ✠✭ θ ❀ ✚ ✮ ❂ ✍❘ ✭ ✚ ✮ ❩ ❯ ✭ θ ❀ θ ✵ ✮ ✚ ✭❞ θ ✵ ✮ ✿ ✍✚ ✭ θ ✮ ❂ ❱ ✭ θ ✮ ✰ Song Mei (Stanford University) Mean Field Dynamics for Neural Network June 26, 2019 6 / 12

Recommend

A Mean Field View of the Landscape of Two-Layers Neural Networks Song Mei Stanford University

A Mean Field View of the Landscape of Two-Layers Neural Networks Song Mei Stanford University

A Mean Field View of the Landscape of Two-Layers Neural Networks Song Mei Stanford University November 14, 2018 Joint work with Andrea Montanari and Phan-Minh Nguyen Song Mei (Stanford University) Mean Field Dynamics for Neural Network

807 views • 70 slides
Nearly-tight VC-dimension bounds for piecewise linear neural networks Nicholas J. A. Harvey,

Nearly-tight VC-dimension bounds for piecewise linear neural networks Nicholas J. A. Harvey,

Nearly-tight VC-dimension bounds for piecewise linear neural networks Nicholas J. A. Harvey, Christopher Liaw , Abbas Mehrabian University of British Columbia COLT 17 July 10, 2017 Neural networks () = max {, 0} (ReLU)

531 views • 20 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
Limitations of recurrent neural networks N ATURAL LAN GUAGE GEN ERATION IN P YTH ON Biswanath

Limitations of recurrent neural networks N ATURAL LAN GUAGE GEN ERATION IN P YTH ON Biswanath

Limitations of recurrent neural networks N ATURAL LAN GUAGE GEN ERATION IN P YTH ON Biswanath Halder Data Scientist Simple neural networks Nodes arranged in layers. Nodes in different layers connected by weights. Input to rst layer :

802 views • 29 slides
Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi Convolutional Neural Networks (CNNs) Fully connected (dense) layers have no awareness of spatial information Key concept behind convolutional layers is

983 views • 41 slides
Prospects of Lattice Field Theory Simulations powered by Deep Neural Networks Julian Urban ITP

Prospects of Lattice Field Theory Simulations powered by Deep Neural Networks Julian Urban ITP

Prospects of Lattice Field Theory Simulations powered by Deep Neural Networks Julian Urban ITP Heidelberg 2019/11/06 " this will " this is never work " revolutionary " Prospects of Lattice Field Theory Simulations

605 views • 23 slides
Analyzing Backprop 3-4-16 Reading Quiz Q1: If a neural network has 3 layers with 10 input, 6

Analyzing Backprop 3-4-16 Reading Quiz Q1: If a neural network has 3 layers with 10 input, 6

Analyzing Backprop 3-4-16 Reading Quiz Q1: If a neural network has 3 layers with 10 input, 6 hidden, and 8 output units, what is the dimension of backpropagations local search space? a) 10 + 6 + 8 = 24 b) 10 + 6 * 8 = 58 c) 10 * 6 + 6 *

381 views • 12 slides
Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Transfer Learning with Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural Networks A breakthough Convolutional Neural Networks VGG-16 example Layers of Convolutional filters Bottleneck

625 views • 23 slides
First-order Adversarial Vulnerability of Neural Networks and Input Dimension C.-J. Simon-Gabriel

First-order Adversarial Vulnerability of Neural Networks and Input Dimension C.-J. Simon-Gabriel

First-order Adversarial Vulnerability of Neural Networks and Input Dimension C.-J. Simon-Gabriel , Y. Ollivier , L. Bottou , B. Schlkopf , D. Lopez-Paz Max-Planck-Institute for Intelligent Systems Facebook AI Research

805 views • 28 slides
Understanding Neural Networks Part II: Convolutional Layers and Collaborative Filters Nick

Understanding Neural Networks Part II: Convolutional Layers and Collaborative Filters Nick

TensorFlow Workshop 2018 Understanding Neural Networks Part II: Convolutional Layers and Collaborative Filters Nick Winovich Department of Mathematics Purdue University July 2018 SIAM@Purdue 2018 - Nick Winovich Understanding Neural Networks

1.21k views • 64 slides
Neural Networks as Stat Mech Systems Based on arXiv:1710.06570 [stat.ML], A

Neural Networks as Stat Mech Systems Based on arXiv:1710.06570 [stat.ML], A

Neural Networks as Stat Mech Systems Based on arXiv:1710.06570 [stat.ML], A Correspondence Between Random Neural Networks and Statistical Field Theory Yoni Kahn, KICP/UIUC hep-ai 1/8/19 <latexit

462 views • 17 slides
Lecture 11: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas, Mark

Lecture 11: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas, Mark

Lecture 11: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas, Mark Glickman and Chris Tanner 1 Outline Review from last lecture 1. Training CNNs 2. BackProp of MaxPooling layer 3. Layers Receptive Field 4.

2.22k views • 89 slides
Lecture 9: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas and Mark

Lecture 9: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas and Mark

Lecture 9: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas and Mark Glickman 1 Outline 1. Review from last lecture 2. BackProp of MaxPooling layer 3. A bit of history 4. Layers Receptive Field 5. Saliency maps 6.

1.15k views • 68 slides
Gauge Theories from the 11th Dimension Neil Lambert Birmingham 21 November 2018 Plan of Attack

Gauge Theories from the 11th Dimension Neil Lambert Birmingham 21 November 2018 Plan of Attack

Gauge Theories from the 11th Dimension Neil Lambert Birmingham 21 November 2018 Plan of Attack Symmetry and Quantum Field theory Supersymmetry String Theory and Quantum Field theory M-theory Back to Quantum Field theory

453 views • 32 slides
Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations deeplearning.cce2020@gmail.com Deep Networks Intuition Neural networks with multiple hidden layers - Deep networks [Hinton, 2006] Deep Networks Intuition

1.33k views • 28 slides
Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech

Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech

Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech Processing gkumar@cs.jhu.edu Universal Approximators Neural networks can approximate [1] function. any Capacity Layers hidden layer size

1.19k views • 17 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Monte Carlo Methods and Neural Networks Noah Gamboa and Alexander Keller Neural Networks Fully

Monte Carlo Methods and Neural Networks Noah Gamboa and Alexander Keller Neural Networks Fully

Monte Carlo Methods and Neural Networks Noah Gamboa and Alexander Keller Neural Networks Fully connected layers neurons a 0 , 0 a L , 0 a 1 , 0 a 2 , 0 a 0 , 1 a L , 1 a 1 , 1 a 2 , 1 a 0 , 2 a L , 2 a 1 , 2 a 2 , 2 . . . . . . . . .

548 views • 44 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
A Neural-Symbolic Approach to the Contemporary Theory of Metaphor Guido Boella - University of

A Neural-Symbolic Approach to the Contemporary Theory of Metaphor Guido Boella - University of

Contemporary Theory of Metaphor Sets, Functions and Networks Example Applications and conclusion Philosophy and Machine Learning - Workshop (@ Neural Information Processing Systems 2011) A Neural-Symbolic Approach to the Contemporary Theory

682 views • 19 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Training Neural Networks with Local Error Signals Arild Nkland Lars H. Eidnes Local learning

Training Neural Networks with Local Error Signals Arild Nkland Lars H. Eidnes Local learning

Training Neural Networks with Local Error Signals Arild Nkland Lars H. Eidnes Local learning Typically we train neural networks by backpropagating errors from the loss function and back through the layers. Hard to explain how the

647 views • 13 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides

More recommend