neural tangent kernel
play

Neural Tangent Kernel Convergence and Generalization in Neural - PowerPoint PPT Presentation

Feb 16, 2024 •190 likes •248 views

Neural Tangent Kernel Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clment Hongler arthur.jacot@epfl.ch franck.gabriel@epfl.ch clement.hongler@epfl.ch What happens during training? One step of Gradient

  1. Neural Tangent Kernel Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler arthur.jacot@epfl.ch franck.gabriel@epfl.ch clement.hongler@epfl.ch

  2. What happens during training? One step of Gradient Descent Neural Tangent Kernel: One datapoint x0 Describes the effect of gradient descent on the network function

  3. Determines the trajectory of the network function during training In the Infinite width limit: - Deterministic - Fixed in time - Explicit formula

  4. Kernel methods Neural Networks Kernel Gradient Descent Gradient Descent Positive definite NTK Convergence to a global min. Least-squares loss Kernel ridge regression

  5. What happens inside a very wide network? - The activations of the hidden neurons become independent - The parameters and activations evolve less and less - However all layers learn: The sum of all microscopic changes yields a macroscopic effect

Recommend

Differential structure, tangent structure, and SDG Geoff Cruttwell (joint work with Robin

Differential structure, tangent structure, and SDG Geoff Cruttwell (joint work with Robin

Introduction Tangent structure Tangent spaces Representable tangent structure Conclusion Differential structure, tangent structure, and SDG Geoff Cruttwell (joint work with Robin Cockett) In honour of Robin Cocketts 60th birthday FMCS

755 views • 58 slides
General Neural Networks Compositions of linear maps and component-wise non- linearities Neural

General Neural Networks Compositions of linear maps and component-wise non- linearities Neural

Pooling and Invariance in Convolutional Neural Networks General Neural Networks Compositions of linear maps and component-wise non- linearities Neural Networks Common Non-Linearities Rectifier Linear Unit Sigmoid Hyperbolic tangent

844 views • 53 slides
When do neural networks outperform kernel methods? Song Mei Stanford University June 29, 2020

When do neural networks outperform kernel methods? Song Mei Stanford University June 29, 2020

When do neural networks outperform kernel methods? Song Mei Stanford University June 29, 2020 Joint work with Behrooz Ghorbani, Theodor Misiakiewicz, and Andrea Montanari Song Mei (Stanford University) Neural Networks and Kernel Methods June

645 views • 26 slides
Weil spaces and closed tangent structure June 2, 2018 1 / 30 Overview W 1 -actegories

Weil spaces and closed tangent structure June 2, 2018 1 / 30 Overview W 1 -actegories

Weil spaces and closed tangent structure June 2, 2018 1 / 30 Overview W 1 -actegories Tangent categories 1 W 1 -presheaf cats 2 Embedding theorem Closed and representable tangent categories 3 2 / 30 Tangent categories and Weil

559 views • 32 slides
Connections in Tangent Categories Robin Cockett University of Calgary (joint work with Geoff

Connections in Tangent Categories Robin Cockett University of Calgary (joint work with Geoff

Connections in Tangent Categories Connections in Tangent Categories Robin Cockett University of Calgary (joint work with Geoff Cruttwell) Category Theory 2016 Dalhousie Connections in Tangent Categories Tangent categories The basics

807 views • 41 slides
Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions Authors : S.

Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions Authors : S.

2018 IEEE International Workshop on Machine Learning for Signal Processing (MLSP18) Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions Authors : S. Scardapane, S. Van Vaerenbergh, D. Comminiello, S. Totaro and A.

545 views • 40 slides
A Neural Network View of Kernel Methods Shuiwang Ji Department of Computer Science &

A Neural Network View of Kernel Methods Shuiwang Ji Department of Computer Science &

A Neural Network View of Kernel Methods Shuiwang Ji Department of Computer Science & Engineering Texas A&M University 1 / 24 Linear Models 1 In a binary classification problem, we have training data { , y i } m i =1 ,

351 views • 24 slides
A Kernel Perspective for Regularizing Deep Neural Networks Julien Mairal Inria Grenoble Imaging

A Kernel Perspective for Regularizing Deep Neural Networks Julien Mairal Inria Grenoble Imaging

A Kernel Perspective for Regularizing Deep Neural Networks Julien Mairal Inria Grenoble Imaging and Machine Learning, IHP, 2019 Julien Mairal A Kernel Perspective for Regularizing NN 1/1 Publications Theoretical Foundations A. Bietti and

1.2k views • 105 slides
Tangent estimation along 3D digital curves l Postolski 1 , 2 , Marcin Janaszewski 2 , Yukiko

Tangent estimation along 3D digital curves l Postolski 1 , 2 , Marcin Janaszewski 2 , Yukiko

Introduction Discrete Straight Segments -Maximal Segment Tangent Direction Experimental Validation Tangent estimation along 3D digital curves l Postolski 1 , 2 , Marcin Janaszewski 2 , Yukiko Kenmochi 1 , Micha Jacques-Olivier Lachaud 3 1

613 views • 26 slides
Bird Call Recognition with Artificial Neural Networks, Support Vector Machines, and Kernel

Bird Call Recognition with Artificial Neural Networks, Support Vector Machines, and Kernel

M. Sc. Thesis Presentation Bird Call Recognition with Artificial Neural Networks, Support Vector Machines, and Kernel Density Estimation Derek Ross Advisors: Dr. H. Card, Electrical and Computer Eng. Dr. D. McNeill, Electrical and Computer

650 views • 64 slides
Differential Programming and Tangent Categories Work with Geoff Cruttwell and Ben MacAdam Wh

Differential Programming and Tangent Categories Work with Geoff Cruttwell and Ben MacAdam Wh

Differential Programming and Tangent Categories Work with Geoff Cruttwell and Ben MacAdam Wh What is s this talk lk about? Apply techniques from synthetic differential geometry and tangent categories to understand internal homs in

251 views • 24 slides
A Kernel Perspective for Regularizing Deep Neural Networks Alberto Bietti* Grgoire Mialon*

A Kernel Perspective for Regularizing Deep Neural Networks Alberto Bietti* Grgoire Mialon*

A Kernel Perspective for Regularizing Deep Neural Networks Alberto Bietti* Grgoire Mialon* Dexiong Chen Julien Mairal Inria ICML 2019, Long Beach Bietti, Mialon, Chen and Mairal Kernel regularization of deep nets ICML 2019, Long Beach 1

417 views • 17 slides
Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel Fernndez V, David P. Woodruff, Taisuke Yasuda Kernel Method Many machine learning tasks can be expressed as a function of the inner product

389 views • 22 slides
Tangent categories are locally Cartesian differential categories J.R.B. Cockett Department of

Tangent categories are locally Cartesian differential categories J.R.B. Cockett Department of

Tangent categories are locally Cartesian differential categories Tangent categories are locally Cartesian differential categories J.R.B. Cockett Department of Computer Science University of Calgary Alberta, Canada robin@cpsc.ucalgary.ca

825 views • 33 slides
1 Kernel methods & optimization One example of a kernel that is frequently used in practice

1 Kernel methods & optimization One example of a kernel that is frequently used in practice

Machine Learning Class Notes 9-25-12 Prof. David Sontag 1 Kernel methods & optimization One example of a kernel that is frequently used in practice and which allows for highly non-linear discriminant functions is the Gaussian kernel,

241 views • 5 slides
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei,

Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei,

Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei, Theodor Misiakiewicz, and Andrea Montanari Stanford University June 26, 2019 COLT 2019 Song Mei (Stanford University) Mean Field Dynamics for

1.12k views • 29 slides
Constructing a tangent from an external point. Let us consider the circle of center O shown below

Constructing a tangent from an external point. Let us consider the circle of center O shown below

D AY 127 T ANGENT TO A CIRCLE I NTRODUCTION So far, we understand what a tangent is. That it is a line that touches the circle. It also meets the radius at a right angle at the point where it touches the circle. We have also discussed a

421 views • 11 slides
Connections in tangent categories Geoff Cruttwell Mount Allison University (joint work with

Connections in tangent categories Geoff Cruttwell Mount Allison University (joint work with

Tangent categories Vector bundles Connections Conclusions Connections in tangent categories Geoff Cruttwell Mount Allison University (joint work with Robin Cockett) Union College Mathematics Conference Union College, October 20th, 2013

429 views • 30 slides
Product Launch: ELIO Series New Tangent product and design series Elio Design Series Design The

Product Launch: ELIO Series New Tangent product and design series Elio Design Series Design The

Product Launch: ELIO Series New Tangent product and design series Elio Design Series Design The new Tangent ELIO design series takes inspiration from the early days of radio design. The retro look combines a aluminum front with sharp edges and

446 views • 15 slides
Affine objects in a tangent category Geoff Cruttwell Mount Allison University (joint work with

Affine objects in a tangent category Geoff Cruttwell Mount Allison University (joint work with

Introduction Differential and Affine objects Tangent category of Affine objects Monads, Comonads, Dist. Laws Conclusions Affine objects in a tangent category Geoff Cruttwell Mount Allison University (joint work with Rick Blute and Rory

646 views • 23 slides
Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural IR tasks Neural IR architecture Feature Representations Neural IR query auto completion Neural IR query suggestion Neural IR document

1.48k views • 18 slides
Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel Fernndez V, David P. Woodruff, Taisuke Yasuda Overview Preliminaries Kernel ridge regression Kernel -means clustering

476 views • 25 slides
Debugging the Linux Kernel with GDB Kieran Bingham Debugging the Linux Kernel with GDB Many

Debugging the Linux Kernel with GDB Kieran Bingham Debugging the Linux Kernel with GDB Many

Debugging the Linux Kernel with GDB Kieran Bingham Debugging the Linux Kernel with GDB Many of us need to debug the Linux kernel Proprietary tools like Trace32 and DS-5 are $$$ Open source debuggers like GDB lack kernel

884 views • 40 slides
Linux Kernel Synchronization System Calls Synchronization in Kernel the kernel RCU File

Linux Kernel Synchronization System Calls Synchronization in Kernel the kernel RCU File

3/1/20 COMP 790: OS Implementation COMP 790: OS Implementation Logical Diagram Binary Memory Threads Formats Allocators User Todays Lecture Linux Kernel Synchronization System Calls Synchronization in Kernel the kernel RCU File

647 views • 8 slides

More recommend