when do neural networks outperform kernel methods
play

When do neural networks outperform kernel methods? Song Mei - PowerPoint PPT Presentation

Jun 04, 2023 •382 likes •645 views

When do neural networks outperform kernel methods? Song Mei Stanford University June 29, 2020 Joint work with Behrooz Ghorbani, Theodor Misiakiewicz, and Andrea Montanari Song Mei (Stanford University) Neural Networks and Kernel Methods June

  1. When do neural networks outperform kernel methods? Song Mei Stanford University June 29, 2020 Joint work with Behrooz Ghorbani, Theodor Misiakiewicz, and Andrea Montanari Song Mei (Stanford University) Neural Networks and Kernel Methods June 29, 2020 1 / 15

  2. Neural tangent model ◮ Multi-layers NN : ❢ ◆ ✭ x ❀ θ ✮ , x ✷ R ❞ , θ ✷ R ◆ ◮ Expanding around θ ✵ : ❢ ◆ ✭ x ❀ θ ✮ ❂ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✰ ❤ θ � θ ✵ ❀ r θ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✐ ✰ ♦ ✭ ❦ θ � θ ✵ ❦ ✷ ✮ ✿ ◮ Neural tangent model: ❢ NT ❀◆ ✭ x ❀ β ❀ θ ✵ ✮ ❂ ❤ β ❀ r θ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✐ ✿ ◮ Coupled gradient flow: ❞ ❞ t θ t ❂ � r θ ❫ θ ✵ ❂ θ ✵ ❀ E ❬✭ ② � ❢ ◆ ✭ x ❀ θ t ✮✮ ✷ ❪ ❀ ❞ ❞ t β t ❂ � r β ❫ β ✵ ❂ 0 ✿ E ❬✭ ② � ❢ NT ❀◆ ✭ x ❀ β t ❀ θ ✵ ✮✮ ✷ ❪ ❀ ◮ Under proper initialization and over-parameterization: ◆ ✦✶ ❥ ❢ ◆ ✭ x ❀ θ t ✮ � ❢ NT ❀◆ ✭ x ❀ β t ✮ ❥ ❂ ✵ ✿ ❧✐♠ [Jacot, Gabriel, Hongler, 2018], [Du, Zhai, Poczos, Singh, 2018], [Chizat, Bach, 2018b], .... Song Mei (Stanford University) Neural Networks and Kernel Methods June 29, 2020 2 / 15

  3. Neural tangent model ◮ Multi-layers NN : ❢ ◆ ✭ x ❀ θ ✮ , x ✷ R ❞ , θ ✷ R ◆ ◮ Expanding around θ ✵ : ❢ ◆ ✭ x ❀ θ ✮ ❂ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✰ ❤ θ � θ ✵ ❀ r θ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✐ ✰ ♦ ✭ ❦ θ � θ ✵ ❦ ✷ ✮ ✿ ◮ Neural tangent model: ❢ NT ❀◆ ✭ x ❀ β ❀ θ ✵ ✮ ❂ ❤ β ❀ r θ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✐ ✿ ◮ Coupled gradient flow: ❞ ❞ t θ t ❂ � r θ ❫ θ ✵ ❂ θ ✵ ❀ E ❬✭ ② � ❢ ◆ ✭ x ❀ θ t ✮✮ ✷ ❪ ❀ ❞ ❞ t β t ❂ � r β ❫ β ✵ ❂ 0 ✿ E ❬✭ ② � ❢ NT ❀◆ ✭ x ❀ β t ❀ θ ✵ ✮✮ ✷ ❪ ❀ ◮ Under proper initialization and over-parameterization: ◆ ✦✶ ❥ ❢ ◆ ✭ x ❀ θ t ✮ � ❢ NT ❀◆ ✭ x ❀ β t ✮ ❥ ❂ ✵ ✿ ❧✐♠ [Jacot, Gabriel, Hongler, 2018], [Du, Zhai, Poczos, Singh, 2018], [Chizat, Bach, 2018b], .... Song Mei (Stanford University) Neural Networks and Kernel Methods June 29, 2020 2 / 15

  4. Neural tangent model ◮ Multi-layers NN : ❢ ◆ ✭ x ❀ θ ✮ , x ✷ R ❞ , θ ✷ R ◆ ◮ Expanding around θ ✵ : ❢ ◆ ✭ x ❀ θ ✮ ❂ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✰ ❤ θ � θ ✵ ❀ r θ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✐ ✰ ♦ ✭ ❦ θ � θ ✵ ❦ ✷ ✮ ✿ ◮ Neural tangent model: ❢ NT ❀◆ ✭ x ❀ β ❀ θ ✵ ✮ ❂ ❤ β ❀ r θ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✐ ✿ ◮ Coupled gradient flow: ❞ ❞ t θ t ❂ � r θ ❫ θ ✵ ❂ θ ✵ ❀ E ❬✭ ② � ❢ ◆ ✭ x ❀ θ t ✮✮ ✷ ❪ ❀ ❞ ❞ t β t ❂ � r β ❫ β ✵ ❂ 0 ✿ E ❬✭ ② � ❢ NT ❀◆ ✭ x ❀ β t ❀ θ ✵ ✮✮ ✷ ❪ ❀ ◮ Under proper initialization and over-parameterization: ◆ ✦✶ ❥ ❢ ◆ ✭ x ❀ θ t ✮ � ❢ NT ❀◆ ✭ x ❀ β t ✮ ❥ ❂ ✵ ✿ ❧✐♠ [Jacot, Gabriel, Hongler, 2018], [Du, Zhai, Poczos, Singh, 2018], [Chizat, Bach, 2018b], .... Song Mei (Stanford University) Neural Networks and Kernel Methods June 29, 2020 2 / 15

  5. Neural tangent model ◮ Multi-layers NN : ❢ ◆ ✭ x ❀ θ ✮ , x ✷ R ❞ , θ ✷ R ◆ ◮ Expanding around θ ✵ : ❢ ◆ ✭ x ❀ θ ✮ ❂ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✰ ❤ θ � θ ✵ ❀ r θ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✐ ✰ ♦ ✭ ❦ θ � θ ✵ ❦ ✷ ✮ ✿ ◮ Neural tangent model: ❢ NT ❀◆ ✭ x ❀ β ❀ θ ✵ ✮ ❂ ❤ β ❀ r θ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✐ ✿ ◮ Coupled gradient flow: ❞ ❞ t θ t ❂ � r θ ❫ θ ✵ ❂ θ ✵ ❀ E ❬✭ ② � ❢ ◆ ✭ x ❀ θ t ✮✮ ✷ ❪ ❀ ❞ ❞ t β t ❂ � r β ❫ β ✵ ❂ 0 ✿ E ❬✭ ② � ❢ NT ❀◆ ✭ x ❀ β t ❀ θ ✵ ✮✮ ✷ ❪ ❀ ◮ Under proper initialization and over-parameterization: ◆ ✦✶ ❥ ❢ ◆ ✭ x ❀ θ t ✮ � ❢ NT ❀◆ ✭ x ❀ β t ✮ ❥ ❂ ✵ ✿ ❧✐♠ [Jacot, Gabriel, Hongler, 2018], [Du, Zhai, Poczos, Singh, 2018], [Chizat, Bach, 2018b], .... Song Mei (Stanford University) Neural Networks and Kernel Methods June 29, 2020 2 / 15

  6. Neural tangent model ◮ Multi-layers NN : ❢ ◆ ✭ x ❀ θ ✮ , x ✷ R ❞ , θ ✷ R ◆ ◮ Expanding around θ ✵ : ❢ ◆ ✭ x ❀ θ ✮ ❂ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✰ ❤ θ � θ ✵ ❀ r θ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✐ ✰ ♦ ✭ ❦ θ � θ ✵ ❦ ✷ ✮ ✿ ◮ Neural tangent model: ❢ NT ❀◆ ✭ x ❀ β ❀ θ ✵ ✮ ❂ ❤ β ❀ r θ ❢ ◆ ✭ x ❀ θ ✵ ✮ ✐ ✿ ◮ Coupled gradient flow: ❞ ❞ t θ t ❂ � r θ ❫ θ ✵ ❂ θ ✵ ❀ E ❬✭ ② � ❢ ◆ ✭ x ❀ θ t ✮✮ ✷ ❪ ❀ ❞ ❞ t β t ❂ � r β ❫ β ✵ ❂ 0 ✿ E ❬✭ ② � ❢ NT ❀◆ ✭ x ❀ β t ❀ θ ✵ ✮✮ ✷ ❪ ❀ ◮ Under proper initialization and over-parameterization: ◆ ✦✶ ❥ ❢ ◆ ✭ x ❀ θ t ✮ � ❢ NT ❀◆ ✭ x ❀ β t ✮ ❥ ❂ ✵ ✿ ❧✐♠ [Jacot, Gabriel, Hongler, 2018], [Du, Zhai, Poczos, Singh, 2018], [Chizat, Bach, 2018b], .... Song Mei (Stanford University) Neural Networks and Kernel Methods June 29, 2020 2 / 15

  7. How about generalization? ◮ [Arora, Du, Hu, Li, Salakhutdinov, Wang, 2019]: Cifar10 experiments. NT: ✷✸✪ test error. NN: less than ✺✪ test error. ◮ [Arora, Du, Li, Salakhutdinov, Wang, Yu, 2019]: Small dataset, NT sometimes generalize better than NN. ◮ [Shankar, Fang, Guo, Fridovich-Keil, Schmidt, Ragan-Kelley, Recht, 2020] [Li, Wang, Yu, Du, Hu, Salakhutdinov, Arora, 2019]: Smaller gap between NT and NN on Cifar10 (10 ✪ for NT). Sometimes there is a large gap, while sometimes the gap is small. Song Mei (Stanford University) Neural Networks and Kernel Methods June 29, 2020 3 / 15

  8. How about generalization? ◮ [Arora, Du, Hu, Li, Salakhutdinov, Wang, 2019]: Cifar10 experiments. NT: ✷✸✪ test error. NN: less than ✺✪ test error. ◮ [Arora, Du, Li, Salakhutdinov, Wang, Yu, 2019]: Small dataset, NT sometimes generalize better than NN. ◮ [Shankar, Fang, Guo, Fridovich-Keil, Schmidt, Ragan-Kelley, Recht, 2020] [Li, Wang, Yu, Du, Hu, Salakhutdinov, Arora, 2019]: Smaller gap between NT and NN on Cifar10 (10 ✪ for NT). Sometimes there is a large gap, while sometimes the gap is small. Song Mei (Stanford University) Neural Networks and Kernel Methods June 29, 2020 3 / 15

  9. Focus of this talk When is there a large performance gap between NN and NT? Song Mei (Stanford University) Neural Networks and Kernel Methods June 29, 2020 4 / 15

  10. Two-layers neural networks Neural networks: ◆ ♥ ❛ ✐ ✛ ✭ ❤ w ✐ ❀ x ✐ ✮ ✿ ❛ ✐ ✷ R ❀ w ✐ ✷ R ❞ ♦ ❳ ❋ NN ❀◆ ❂ ❢ ◆ ✭ x ❀ Θ ✮ ❂ ✿ ✐ ❂✶ Linearization: ◆ ◆ ❳ ❳ ❢ ◆ ✭ x ❀ Θ ✮ ❂ ❢ ◆ ✭ x ❀ Θ ✵ ✮ ✰ ✁ ❛ ✐ ✛ ✭ ❤ w ✵ ❛ ✵ ✐ ✛ ✵ ✭ ❤ w ✵ ✐ ❀ x ✐ ✮ ✰ ✐ ❀ x ✐ ✮ ❤ ✁ w ✐ ❀ x ✐ ✰ ♦ ✭ ✁ ✮ ✿ ✐ ❂✶ ✐ ❂✶ ⑤ ④③ ⑥ ⑤ ④③ ⑥ Top layer linearization Bottom layer linearization Linearized neural networks ( W ❂ ✭ w ✐ ✮ ✐ ✷ ❬ ◆ ❪ ✘ ✐✐❞ ❯♥✐❢✭ S ❞ � ✶ ✮ ): ◆ ♥ ♦ ❳ ❋ RF ❀◆ ✭ W ✮ ❂ ❢ ❂ ❛ ✐ ✛ ✭ ❤ w ✐ ❀ x ✐ ✮ ✿ ❛ ✐ ✷ R ❀ ✐ ✷ ❬ ◆ ❪ ❀ ✐ ❂✶ ◆ ♥ ♦ ❳ ✛ ✵ ✭ ❤ w ✐ ❀ x ✐ ✮ ❤ b ✐ ❀ x ✐ ✿ b ✐ ✷ R ❞ ❀ ✐ ✷ ❬ ◆ ❪ ❋ NT ❀◆ ✭ W ✮ ❂ ❢ ❂ ✿ ✐ ❂✶ Song Mei (Stanford University) Neural Networks and Kernel Methods June 29, 2020 5 / 15

  11. Two-layers neural networks Neural networks: ◆ ♥ ❛ ✐ ✛ ✭ ❤ w ✐ ❀ x ✐ ✮ ✿ ❛ ✐ ✷ R ❀ w ✐ ✷ R ❞ ♦ ❳ ❋ NN ❀◆ ❂ ❢ ◆ ✭ x ❀ Θ ✮ ❂ ✿ ✐ ❂✶ Linearization: ◆ ◆ ❳ ❳ ❢ ◆ ✭ x ❀ Θ ✮ ❂ ❢ ◆ ✭ x ❀ Θ ✵ ✮ ✰ ✁ ❛ ✐ ✛ ✭ ❤ w ✵ ❛ ✵ ✐ ✛ ✵ ✭ ❤ w ✵ ✐ ❀ x ✐ ✮ ✰ ✐ ❀ x ✐ ✮ ❤ ✁ w ✐ ❀ x ✐ ✰ ♦ ✭ ✁ ✮ ✿ ✐ ❂✶ ✐ ❂✶ ⑤ ④③ ⑥ ⑤ ④③ ⑥ Top layer linearization Bottom layer linearization Linearized neural networks ( W ❂ ✭ w ✐ ✮ ✐ ✷ ❬ ◆ ❪ ✘ ✐✐❞ ❯♥✐❢✭ S ❞ � ✶ ✮ ): ◆ ♥ ♦ ❳ ❋ RF ❀◆ ✭ W ✮ ❂ ❢ ❂ ❛ ✐ ✛ ✭ ❤ w ✐ ❀ x ✐ ✮ ✿ ❛ ✐ ✷ R ❀ ✐ ✷ ❬ ◆ ❪ ❀ ✐ ❂✶ ◆ ♥ ♦ ❳ ✛ ✵ ✭ ❤ w ✐ ❀ x ✐ ✮ ❤ b ✐ ❀ x ✐ ✿ b ✐ ✷ R ❞ ❀ ✐ ✷ ❬ ◆ ❪ ❋ NT ❀◆ ✭ W ✮ ❂ ❢ ❂ ✿ ✐ ❂✶ Song Mei (Stanford University) Neural Networks and Kernel Methods June 29, 2020 5 / 15

Recommend

Can Deep Learning Be Interpreted with Kernel Methods ? Ben Edelman & Preetum Nakkiran

Can Deep Learning Be Interpreted with Kernel Methods ? Ben Edelman & Preetum Nakkiran

Can Deep Learning Be Interpreted with Kernel Methods ? Ben Edelman & Preetum Nakkiran Opening the black box of neural networks Weve seen various post-hoc explanation methods (LIME, SHAP , etc.), but none that are faithful and robust.

1.03k views • 47 slides
Bayesian Methods for Neural Networks Readings: Bishop, Neural Networks for Pattern Recognition .

Bayesian Methods for Neural Networks Readings: Bishop, Neural Networks for Pattern Recognition .

Bayesian Methods for Neural Networks Readings: Bishop, Neural Networks for Pattern Recognition . Chapter 10. Aaron Courville Bayesian Methods for Neural Networks p.1/29 Bayesian Inference Weve seen Bayesian inference before, remember

542 views • 29 slides
IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks (2:2)

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks (2:2)

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks (2:2) Erik Velldal & Lilja vrelid University of Oslo 3 March 2020 Agenda Brief recap from last week on CNNs. Extensions of the basic CNN

592 views • 34 slides
IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen University of Oslo March 10, 2019 Our Roadmap Today Language structure: sequences, trees, graphs Recurrent Neural Networks Different

780 views • 76 slides
IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks Stephan Oepen University of Oslo March 24, 2019 Our Roadmap Last Week Language structure: sequences, trees, graphs Recurrent Neural

217 views • 19 slides
Categorical Data Clustering Using Statistical Methods and Neural Networks . Kudov 1 , H.

Categorical Data Clustering Using Statistical Methods and Neural Networks . Kudov 1 , H.

Introduction Clustering Statistical methods Neural Networks Experiments Conclusion Categorical Data Clustering Using Statistical Methods and Neural Networks . Kudov 1 , H. Rezankov 2 , D. Hsek 1 , V. Snel 3 P 1 Institute of

929 views • 33 slides
A Neural Network View of Kernel Methods Shuiwang Ji Department of Computer Science &

A Neural Network View of Kernel Methods Shuiwang Ji Department of Computer Science &

A Neural Network View of Kernel Methods Shuiwang Ji Department of Computer Science & Engineering Texas A&M University 1 / 24 Linear Models 1 In a binary classification problem, we have training data { , y i } m i =1 ,

351 views • 24 slides
Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions Authors : S.

Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions Authors : S.

2018 IEEE International Workshop on Machine Learning for Signal Processing (MLSP18) Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions Authors : S. Scardapane, S. Van Vaerenbergh, D. Comminiello, S. Totaro and A.

545 views • 40 slides
IN5550: Neural Methods in Natural Language Processing IN5550 Neural Methods in Natural

IN5550: Neural Methods in Natural Language Processing IN5550 Neural Methods in Natural

IN5550: Neural Methods in Natural Language Processing IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Lilja vrelid & Stephan Oepen University of Oslo March 14, 2019 Obligatory assignment 3

884 views • 27 slides
IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik Velldal University of Oslo 25 February 2020 So far: MLPs + embeddings as inputs Embeddings have benefits over discrete feature vectors; makes

806 views • 58 slides
IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik

IN5550 Neural Methods in Natural Language Processing Convolutional Neural Networks Erik Velldal University of Oslo 25 February 2020 So far: MLPs + embeddings as inputs Embeddings have benefits over discrete feature vectors; makes

902 views • 35 slides
Monte Carlo Methods and Neural Networks Noah Gamboa and Alexander Keller Neural Networks Fully

Monte Carlo Methods and Neural Networks Noah Gamboa and Alexander Keller Neural Networks Fully

Monte Carlo Methods and Neural Networks Noah Gamboa and Alexander Keller Neural Networks Fully connected layers neurons a 0 , 0 a L , 0 a 1 , 0 a 2 , 0 a 0 , 1 a L , 1 a 1 , 1 a 2 , 1 a 0 , 2 a L , 2 a 1 , 2 a 2 , 2 . . . . . . . . .

548 views • 44 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Recurrent Neural Networks LING572 Advanced Statistical Methods for NLP March 5 2020 1 Outline

Recurrent Neural Networks LING572 Advanced Statistical Methods for NLP March 5 2020 1 Outline

Recurrent Neural Networks LING572 Advanced Statistical Methods for NLP March 5 2020 1 Outline Word representations and MLPs for NLP tasks Recurrent neural networks for sequences Fancier RNNs Vanishing/exploding gradients

921 views • 91 slides
Neural Networks, Chapter 11 in ESL II STK-IN4300 Statistical Learning Methods in Data Science

Neural Networks, Chapter 11 in ESL II STK-IN4300 Statistical Learning Methods in Data Science

Neural Networks, Chapter 11 in ESL II STK-IN4300 Statistical Learning Methods in Data Science Odd Kolbjrnsen, oddkol@math.uio.no Learning today Neural nets Projection pursuit What is it? How to solve it: Stagewise Neural nets

979 views • 40 slides
A Kernel Perspective for Regularizing Deep Neural Networks Julien Mairal Inria Grenoble Imaging

A Kernel Perspective for Regularizing Deep Neural Networks Julien Mairal Inria Grenoble Imaging

A Kernel Perspective for Regularizing Deep Neural Networks Julien Mairal Inria Grenoble Imaging and Machine Learning, IHP, 2019 Julien Mairal A Kernel Perspective for Regularizing NN 1/1 Publications Theoretical Foundations A. Bietti and

1.2k views • 105 slides
Monte Carlo Methods and Neural Networks Alexander Keller, partially joint work with Noah Gamboa

Monte Carlo Methods and Neural Networks Alexander Keller, partially joint work with Noah Gamboa

Monte Carlo Methods and Neural Networks Alexander Keller, partially joint work with Noah Gamboa Artificial Neural Networks in a Nutshell Supervised learning of high dimensional function approximation input layer a 0 , L 1 fully connected

1.04k views • 58 slides
Chapter 11. Stochastic Methods Rooted in Statistical Mechanics Neural Networks and Learning

Chapter 11. Stochastic Methods Rooted in Statistical Mechanics Neural Networks and Learning

Chapter 11. Stochastic Methods Rooted in Statistical Mechanics Neural Networks and Learning Machines (Haykin) Lecture Notes on Self-learning Neural Algorithms Byoung-Tak Zhang School of Computer Science and Engineering Seoul National

680 views • 41 slides
ICASSP 2017 Tutorial on Methods for Interpreting and Understanding Deep Neural Networks G.

ICASSP 2017 Tutorial on Methods for Interpreting and Understanding Deep Neural Networks G.

ICASSP 2017 Tutorial on Methods for Interpreting and Understanding Deep Neural Networks G. Montavon, W. Samek, K.-R. Mller Part 2: Making Deep Neural Networks Transparent 5 March 2017 Making Deep Neural Nets Transparent DNN transparency

448 views • 44 slides
Neural Tangent Kernel Convergence and Generalization in Neural Networks Arthur Jacot Franck

Neural Tangent Kernel Convergence and Generalization in Neural Networks Arthur Jacot Franck

Neural Tangent Kernel Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clment Hongler arthur.jacot@epfl.ch franck.gabriel@epfl.ch clement.hongler@epfl.ch What happens during training? One step of Gradient

248 views • 5 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
1 Kernel methods & optimization One example of a kernel that is frequently used in practice

1 Kernel methods & optimization One example of a kernel that is frequently used in practice

Machine Learning Class Notes 9-25-12 Prof. David Sontag 1 Kernel methods & optimization One example of a kernel that is frequently used in practice and which allows for highly non-linear discriminant functions is the Gaussian kernel,

241 views • 5 slides
CS489/698 Lecture 10: Feb 6, 2017 Kernel methods [D] Chap. 11 [B] Sec. 6.1, 6.2 [M] Sec. 14.1,

CS489/698 Lecture 10: Feb 6, 2017 Kernel methods [D] Chap. 11 [B] Sec. 6.1, 6.2 [M] Sec. 14.1,

CS489/698 Lecture 10: Feb 6, 2017 Kernel methods [D] Chap. 11 [B] Sec. 6.1, 6.2 [M] Sec. 14.1, 14.2 [H] Chap. 9 [HTF] Chap. 6 CS489/698 (c) 2017 P. Poupart 1 Non-linear Models Recap Generalized linear models: Neural networks:

275 views • 15 slides

More recommend