generalisation bounds for neural networks
play

Generalisation Bounds for Neural Networks Pascale Gourdeau - PowerPoint PPT Presentation

Feb 07, 2024 •242 likes •1.05k views

Generalisation Bounds for Neural Networks Pascale Gourdeau University of Oxford 15 November 2018 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 1 / 31 Overview Introduction 1 General Strategies to

  1. Generalisation Bounds for Neural Networks Pascale Gourdeau University of Oxford 15 November 2018 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 1 / 31

  2. Overview Introduction 1 General Strategies to Obtain Generalisation Bounds 2 Survey of Generalisation Bounds for Neural Networks 3 A Compression Approach [Arora et al., 2018] 4 Conclusion, Research Directions 5 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 2 / 31

  3. Overview Introduction 1 General Strategies to Obtain Generalisation Bounds 2 Survey of Generalisation Bounds for Neural Networks 3 A Compression Approach [Arora et al., 2018] 4 Conclusion, Research Directions 5 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 3 / 31

  4. What is generalisation? The ability to perform well on unseen data. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 4 / 31

  5. What is generalisation? The ability to perform well on unseen data. Assumption: the data (both for the training and testing) comes i.i.d. from a distribution D . Usually work in a distribution-agnostic setting. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 4 / 31

  6. What are generalisation bounds? Classification setting : input space X and output space Y := { 1 , . . . , k } with a distribution D on X × Y . Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 5 / 31

  7. What are generalisation bounds? Classification setting : input space X and output space Y := { 1 , . . . , k } with a distribution D on X × Y . Goal: to learn a function f : X → Y from a sample S := { ( x i , y i ) } m i =1 ⊆ X × Y . Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 5 / 31

  8. What are generalisation bounds? Classification setting : input space X and output space Y := { 1 , . . . , k } with a distribution D on X × Y . Goal: to learn a function f : X → Y from a sample S := { ( x i , y i ) } m i =1 ⊆ X × Y . Generalisation bounds: bounding the difference between the expected and empirical losses of f with high probability over S . Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 5 / 31

  9. What are generalisation bounds? Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 6 / 31

  10. What are generalisation bounds? For neural networks, we use the expected classification loss: � � L 0 ( f ) := P ( x , y ) ∼ D f ( x ) y ≤ max y ′ � = y f ( x ) y ′ , Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 6 / 31

  11. What are generalisation bounds? For neural networks, we use the expected classification loss: � � L 0 ( f ) := P ( x , y ) ∼ D f ( x ) y ≤ max y ′ � = y f ( x ) y ′ , and the empirical margin loss: � � m � L γ ( f ) := 1 � f ( x ) y ≤ γ + max y ′ � = y f ( x ) y ′ . 1 m i =1 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 6 / 31

  12. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  13. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. E.g.: With probability 95% over the training sample, the error is at most 1%. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  14. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. E.g.: With probability 95% over the training sample, the error is at most 1%. They can also: Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  15. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. E.g.: With probability 95% over the training sample, the error is at most 1%. They can also: Provide insight on the ability of a model to generalise. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  16. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. E.g.: With probability 95% over the training sample, the error is at most 1%. They can also: Provide insight on the ability of a model to generalise. This is of particular interest for us: neural networks have many counter-intuitive properties. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  17. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. E.g.: With probability 95% over the training sample, the error is at most 1%. They can also: Provide insight on the ability of a model to generalise. This is of particular interest for us: neural networks have many counter-intuitive properties. Inspire new algorithms or regularisation techniques. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  18. Overview Introduction 1 General Strategies to Obtain Generalisation Bounds 2 Survey of Generalisation Bounds for Neural Networks 3 A Compression Approach [Arora et al., 2018] 4 Conclusion, Research Directions 5 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 8 / 31

  19. General Strategies Generalisation bounds (GB) for neural networks are usually obtained by 1 Defining a class H of functions computed by neural networks with certain properties (e.g., weight matrices with bounded norms, number of layers, etc.), Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 9 / 31

  20. General Strategies Generalisation bounds (GB) for neural networks are usually obtained by 1 Defining a class H of functions computed by neural networks with certain properties (e.g., weight matrices with bounded norms, number of layers, etc.), 2 Deriving a generalisation bound in terms of a complexity measure M ( H ) (e.g. size of H , Rademacher complexity), Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 9 / 31

  21. General Strategies Generalisation bounds (GB) for neural networks are usually obtained by 1 Defining a class H of functions computed by neural networks with certain properties (e.g., weight matrices with bounded norms, number of layers, etc.), 2 Deriving a generalisation bound in terms of a complexity measure M ( H ) (e.g. size of H , Rademacher complexity), 3 Upper bounding M ( H ) in terms of model parameters (e.g., norm of weight matrices, number of layers, etc.). Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 9 / 31

  22. General Strategies: Rademacher Complexity Definition (Rademacher complexity) Let G be a family of functions from a set Z to R . Let σ 1 , . . . , σ m be Rademacher variables: P ( σ i = 1) = P ( σ i = − 1) = 1 / 2. The empirical Rademacher complexity of G w.r.t. to a sample S = { z i } m i =1 is Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 10 / 31

  23. General Strategies: Rademacher Complexity Definition (Rademacher complexity) Let G be a family of functions from a set Z to R . Let σ 1 , . . . , σ m be Rademacher variables: P ( σ i = 1) = P ( σ i = − 1) = 1 / 2. The empirical Rademacher complexity of G w.r.t. to a sample S = { z i } m i =1 is � � m � 1 R S ( G ) = E σ sup σ i g ( z i ) . m g ∈ G i =1 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 10 / 31

  24. General Strategies: Rademacher Complexity Definition (Rademacher complexity) Let G be a family of functions from a set Z to R . Let σ 1 , . . . , σ m be Rademacher variables: P ( σ i = 1) = P ( σ i = − 1) = 1 / 2. The empirical Rademacher complexity of G w.r.t. to a sample S = { z i } m i =1 is � � m � 1 R S ( G ) = E σ sup σ i g ( z i ) . m g ∈ G i =1 Intuition: How much G correlates with random noise on S . Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 10 / 31

  25. General Strategies: Rademacher Complexity Definition (Rademacher complexity) Let G be a family of functions from a set Z to R . Let σ 1 , . . . , σ m be Rademacher variables: P ( σ i = 1) = P ( σ i = − 1) = 1 / 2. The empirical Rademacher complexity of G w.r.t. to a sample S = { z i } m i =1 is � � m � 1 R S ( G ) = E σ sup σ i g ( z i ) . m g ∈ G i =1 Intuition: How much G correlates with random noise on S . Simple examples... Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 10 / 31

  26. General Strategies: Rademacher Complexity Theorem Let G be a family of functions from Z to [0 , 1] , and let S be a sample of size m drawn from Z according to D. Let L ( g ) = E z ∼ D [ g ( z )] and � m � L ( g ) = 1 i =1 g ( z i ) . Then for any δ > 0 , with probability at least 1 − δ m over S, for all functions g ∈ G, �� � log(1 /δ ) L ( g ) ≤ � L ( g ) + 2 R S ( G ) + O . m Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 11 / 31

Recommend

Nearly-tight VC-dimension bounds for piecewise linear neural networks Nicholas J. A. Harvey,

Nearly-tight VC-dimension bounds for piecewise linear neural networks Nicholas J. A. Harvey,

Nearly-tight VC-dimension bounds for piecewise linear neural networks Nicholas J. A. Harvey, Christopher Liaw , Abbas Mehrabian University of British Columbia COLT 17 July 10, 2017 Neural networks () = max {, 0} (ReLU)

531 views • 20 slides
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao and Quanquan Gu Computer Science Department 1 / 14 Learning Over-parameterized DNNs Empirical observation on extremely wide deep neural networks

631 views • 14 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin

Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin

Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin Nov 21, 2019 1 / 32 In the previous lecture, we derived upper bounds on Rademacher averages of a set of neural networks in terms of norms of weight

640 views • 32 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Using Learning Dynamics to Understand Neural Network Generalisation Arnu Pretorius Supervisor:

Using Learning Dynamics to Understand Neural Network Generalisation Arnu Pretorius Supervisor:

Using Learning Dynamics to Understand Neural Network Generalisation Arnu Pretorius Supervisor: Dr. Steve Kroon (Computer Science ) Co-supervisor: Dr. Herman Kamper (EE Engineering + ) 20 October 2017 Maties Machine Learning, Stellenbosch

447 views • 11 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei,

Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei,

Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei, Theodor Misiakiewicz, and Andrea Montanari Stanford University June 26, 2019 COLT 2019 Song Mei (Stanford University) Mean Field Dynamics for

1.12k views • 29 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

Need for Machine . . . Neural Networks . . . How to Speed Up . . . Neural Networks: A . . . How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . . . Resulting Algorithm Faster by Taking Into How to

464 views • 17 slides
Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Paul Nuytten CPSC 607 Why Model Neural Networks? The Neuron The nervous

697 views • 4 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What well cover ... f ( x ) computer vision architectures - convolutional networks - data augmentation 1 - residual networks ... ...

1.29k views • 75 slides
Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC

Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC

Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC www.kit.edu KIT The Research University in the Helmholtz Association Neural Network Concept (very) remotely brain inspired computational system

479 views • 25 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
7. Artificial neural networks Introduction to neural networks Despite struggling to understand

7. Artificial neural networks Introduction to neural networks Despite struggling to understand

7. Artificial neural networks Introduction to neural networks Despite struggling to understand intricacies of protein, cell, and network function within the brain, neuroscientists would agree on the following simplistic description of how the

157 views • 11 slides
Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons

Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons

Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons Multilayer networks Applications of neural networks Chapter 20 2 Brains 10 11 neurons of > 20 types, 10 14 synapses, 1ms10ms cycle

804 views • 47 slides
Modeling Neural Networks Paul Nuytten CPSC 607 Outline Why model neural networks? A

Modeling Neural Networks Paul Nuytten CPSC 607 Outline Why model neural networks? A

Modeling Neural Networks Paul Nuytten CPSC 607 Outline Why model neural networks? A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Why Model Neural Networks? The nervous system is a

1.22k views • 23 slides

More recommend