global optimality in neural network training
play

Global Optimality in Neural Network Training Benjamin D. Haeffele - PowerPoint PPT Presentation

Aug 11, 2023 •638 likes •1.56k views

Global Optimality in Neural Network Training Benjamin D. Haeffele and Ren Vidal Johns Hopkins University, Center for Imaging Science. Baltimore, USA Questions in Deep Learning Architecture Design Optimization Generalization Questions in

  1. Global Optimality in Neural Network Training Benjamin D. Haeffele and René Vidal Johns Hopkins University, Center for Imaging Science. Baltimore, USA

  2. Questions in Deep Learning Architecture Design Optimization Generalization

  3. Questions in Deep Learning Are there principled ways to design networks? • How many layers? • Size of layers? • Choice of layer types? • How does architecture impact expressiveness? [1] [1 ] Cohen, et al., “On the expressive power of deep learning: A tensor analysis.” COLT. (2016)

  4. Questions in Deep Learning How to train neural networks?

  5. Questions in Deep Learning How to train neural networks? • Problem is non-convex.

  6. Questions in Deep Learning How to train neural networks? X • Problem is non-convex.

  7. Questions in Deep Learning How to train neural networks? X • Problem is non-convex. • What does the loss surface look like? [1] • Any guarantees for network training? [2] • How to guarantee optimality? • When will local descent succeed? [1] Choromanska, et al., "The loss surfaces of multilayer networks." Artificial Intelligence and Statistics. (2015) [2] Janzamin, et al., "Beating the perils of non-convexity: Guaranteed training of neural networks using tensor methods." arXiv (2015).

  8. Questions in Deep Learning Performance Guarantees? X Complex  Simple • How do networks generalize? • How should networks be regularized? • How to prevent overfitting?

  9. Interrelated Problems Architecture • Optimization can impact generalization. [1] • Architecture has a strong effect on the Generalization/ generalization of networks. [2] Regularization Optimization • Some architectures could be easier to optimize than others. [1] Neyshabur, et al., “In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning.” ICLR workshop. (2015). [2] Zhang, et al., “Understanding deep learning requires rethinking generalization.” ICLR. (2017).

  10. Today’s Talk: The Questions Architecture • Are there properties of the network architecture that allow efficient optimization? Generalization/ Regularization Optimization

  11. Today’s Talk: The Questions Architecture • Are there properties of the network architecture that allow efficient optimization? • Positive Homogeneity • Parallel Subnetwork Structure Generalization/ Regularization Optimization

  12. Today’s Talk: The Questions Architecture • Are there properties of the network architecture that allow efficient optimization? • Positive Homogeneity • Parallel Subnetwork Structure Generalization/ • Are there properties of the Regularization Optimization regularization that allow efficient optimization?

  13. Today’s Talk: The Questions Architecture • Are there properties of the network architecture that allow efficient optimization? • Positive Homogeneity • Parallel Subnetwork Structure Generalization/ • Are there properties of the Regularization Optimization regularization that allow efficient optimization? • Positive Homogeneity • Adapt network architecture to data [1] [1] Bengio, et al., “Convex neural networks .” NIPS. (2005)

  14. Today’s Talk: The Results Optimization

  15. Today’s Talk: The Results Optimization • A local minimum such that one subnetwork is all zero is a global minimum.

  16. Today’s Talk: The Results Optimization • Once the size of the network becomes large enough... • Local descent can reach a global minimum from any initialization. Today’s Framework Non-Convex Function

  17. Outline 1. Network properties that allow Architecture efficient optimization • Positive Homogeneity • Parallel Subnetwork Structure 2. Network size from regularization Generalization/ Regularization Optimization 3. Theoretical guarantees • Sufficient conditions for global optimality • Local descent can reach global minimizers

  18. Key Property 1: Positive Homogeneity • Start with a network. Network Outputs Network Weights

  19. Key Property 1: Positive Homogeneity • Scale the weights by a non-negative constant.

  20. Key Property 1: Positive Homogeneity • Scale the weights by a non-negative constant.

  21. Key Property 1: Positive Homogeneity • The network output scales by the constant to some power.

  22. Key Property 1: Positive Homogeneity • The network output scales by the constant to some power. Network Mapping

  23. Key Property 1: Positive Homogeneity • The network output scales by the constant to some power. Network Mapping - Degree of positive homogeneity

  24. Most Modern Networks Are Positively Homogeneous • Example: Rectified Linear Units (ReLUs)

  25. Most Modern Networks Are Positively Homogeneous • Example: Rectified Linear Units (ReLUs)

  26. Most Modern Networks Are Positively Homogeneous • Example: Rectified Linear Units (ReLUs)

  27. Most Modern Networks Are Positively Homogeneous • Example: Rectified Linear Units (ReLUs)

  28. Most Modern Networks Are Positively Homogeneous • Example: Rectified Linear Units (ReLUs) Doesn’t change rectification

  29. Most Modern Networks Are Positively Homogeneous • Example: Rectified Linear Units (ReLUs) Doesn’t change rectification

  30. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU

  31. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU

  32. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU

  33. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU

  34. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU

  35. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU

  36. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU

  37. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU

  38. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU

  39. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU

  40. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU

  41. Most Modern Networks Are Positively Homogeneous • Simple Network Conv Conv Max + Linear Out Input + Pool ReLU ReLU • Typically each weight layer increases degree of homogeneity by 1.

  42. Most Modern Networks Are Positively Homogeneous Some Common Positively Homogeneous Layers  Fully Connected + ReLU  Convolution + ReLU  Max Pooling  Linear Layers  Mean Pooling  Max Out  Many possibilities…

  43. Most Modern Networks Are Positively Homogeneous Some Common Positively Homogeneous Layers  Fully Connected + ReLU  Convolution + ReLU  Max Pooling X Not Sigmoids  Linear Layers  Mean Pooling  Max Out  Many possibilities…

  44. Outline 1. Network properties that allow Architecture efficient optimization • Positive Homogeneity • Parallel Subnetwork Structure 2. Network regularization Generalization/ Regularization Optimization 3. Theoretical guarantees • Sufficient conditions for global optimality • Local descent can reach global minimizers

  45. Key Property 2: Parallel Subnetworks • Subnetworks with identical architecture connected in parallel.

  46. Key Property 2: Parallel Subnetworks • Subnetworks with identical architecture connected in parallel. • Simple Example: Single hidden layer network

  47. Key Property 2: Parallel Subnetworks • Subnetworks with identical architecture connected in parallel. • Simple Example: Single hidden layer network

  48. Key Property 2: Parallel Subnetworks • Subnetworks with identical architecture connected in parallel. • Simple Example: Single hidden layer network • Subnetwork: One ReLU hidden unit

  49. Key Property 2: Parallel Subnetworks • Any positively homogeneous subnetwork can be used • Subnetwork: Multiple ReLU layers

  50. Key Property 2: Parallel Subnetworks • Example: Parallel AlexNets[1] • Subnetwork: AlexNet AlexNet AlexNet Input AlexNet Output AlexNet AlexNet [1] Krizhevsky, Sutskever, and Hinton. "Imagenet classification with deep convolutional neural networks." NIPS, 2012.

  51. Outline 1. Network properties that allow efficient Architecture optimization • Positive Homogeneity • Parallel Subnetwork Structure 2. Network regularization Generalization/ Regularization Optimization 3. Theoretical guarantees • Sufficient conditions for global optimality • Local descent can reach global minimizers

  52. Basic Regularization: Weight Decay Network Weights

  53. Basic Regularization: Weight Decay Network Weights

  54. Basic Regularization: Weight Decay Network Weights

  55. Basic Regularization: Weight Decay Network Weights

  56. Basic Regularization: Weight Decay Network Weights

Recommend

Global Neural CCG Parsing with Optimality Guarantees Kenton Lee Mike Lewis Luke Zettlemoyer

Global Neural CCG Parsing with Optimality Guarantees Kenton Lee Mike Lewis Luke Zettlemoyer

Global Neural CCG Parsing with Optimality Guarantees Kenton Lee Mike Lewis Luke Zettlemoyer University of Washington UWNLP Now at Facebook AI Research 1 This Talk Challenge : Global models (e.g. Recursive NNs) break dynamic programs

1.24k views • 70 slides
Transparent parallelization of neural network training Cyprien Noel Flickr / Yahoo - GTC 2015

Transparent parallelization of neural network training Cyprien Noel Flickr / Yahoo - GTC 2015

Transparent parallelization of neural network training Cyprien Noel Flickr / Yahoo - GTC 2015 by overthemoon Outline Neural Nets at Flickr Training Fast Parallel Distributed Q&A Tagging Photos Class Probability

562 views • 32 slides
CS4811 Neural Network Training Example Consider the following network. It has two inputs (two

CS4811 Neural Network Training Example Consider the following network. It has two inputs (two

CS4811 Neural Network Training Example Consider the following network. It has two inputs (two entries in each input vector), one hidden layer with two neurons and the output layer with a single neuron. Each neuron has a bias input to allow

216 views • 4 slides
Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics Activation Functions Stochastic Gradient Descent (SGD) Regularization (Dropout) Training Tips and Tricks Neural Network (NN)

828 views • 45 slides
IMPLEMENTATION OF A PARALLEL BATCH TRAINING ALGORITHM FOR DEEP NEURAL NETWORK YUPING LIN IFLYTEK

IMPLEMENTATION OF A PARALLEL BATCH TRAINING ALGORITHM FOR DEEP NEURAL NETWORK YUPING LIN IFLYTEK

IMPLEMENTATION OF A PARALLEL BATCH TRAINING ALGORITHM FOR DEEP NEURAL NETWORK YUPING LIN IFLYTEK LABORATORY FOR NEURAL COMPUTING FOR MACHINE LEARNING DEPARTMENT OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE YORK UNIVERSITY, TORONTO NOVEMBER

211 views • 20 slides
Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi,

Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi,

Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geofgrey Hinton Motivation We need tools to understand trained neural networks Neural network training involves interactions between an

588 views • 16 slides
Neural Network Training: Old & New Tricks Old: (80s) Stochastic Gradient Descent,

Neural Network Training: Old & New Tricks Old: (80s) Stochastic Gradient Descent,

Neural Network Training: Old & New Tricks Old: (80s) Stochastic Gradient Descent, Momentum, weight decay New: (last 5-6 years) Dropout ReLUs Batch Normalization Reminder: Overfitting, in images Classification just right

1.61k views • 132 slides
FPGA-based Training Accelerator Utilizing Sparseness of Convolutional Neural Network Hiroki

FPGA-based Training Accelerator Utilizing Sparseness of Convolutional Neural Network Hiroki

FPGA-based Training Accelerator Utilizing Sparseness of Convolutional Neural Network Hiroki Nakahara, Youki Sada, Masayuki Shimoda, Akira Jinguji, Shimpei Sato Tokyo Institute of Technology, JP FPL2019 @Barcelona Challenges in DL Training

369 views • 10 slides
Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training the network Given a training set of input-output pairs Minimize the following function w.r.t This is problem of function minimization

1.79k views • 165 slides
On The Complexity of Training a Neural Network Santosh Vempala Algorithms and Randomness Center

On The Complexity of Training a Neural Network Santosh Vempala Algorithms and Randomness Center

On The Complexity of Training a Neural Network Santosh Vempala Algorithms and Randomness Center Georgia Tech The Complexity of Learning Neural Networks Deep learning s successes are incredible Do you want to beat 9 dan Goplayers? The

873 views • 55 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
Artificial Neural Network : Training Debasis Samanta IIT Kharagpur

Artificial Neural Network : Training Debasis Samanta IIT Kharagpur

Artificial Neural Network : Training Debasis Samanta IIT Kharagpur debasis.samanta.iitkgp@gmail.com 06.04.2018 Debasis Samanta (IIT Kharagpur) Soft Computing Applications 06.04.2018 1 / 49 Learning of neural networks: Topics Concept of

608 views • 49 slides
Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks Alexander Shevchenko Marco Mondelli Neural Network Training From theoretical perspective training of neural networks is di ffi cult

494 views • 28 slides
Networks Luke Schuler Overview What is an Artificial Neural Network? History

Networks Luke Schuler Overview What is an Artificial Neural Network? History

Artificial Neural Networks Luke Schuler Overview What is an Artificial Neural Network? History Inception Abandonment Resurgence Model Training Applications What is an Artificial Neural Network? An artificial

616 views • 15 slides
FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan, Laxmikant Kishor Mokadam, Xipeng Shen, Seung-Hwan Lim, Robert Patton 1 Build an image classifier? Deep Neural Network (DNN) CPU GPU Training

957 views • 41 slides
Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural IR tasks Neural IR architecture Feature Representations Neural IR query auto completion Neural IR query suggestion Neural IR document

1.48k views • 18 slides
Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides
Principled Deep Neural Network Training through Linear Programming Daniel Bienstock 1 , Gonzalo

Principled Deep Neural Network Training through Linear Programming Daniel Bienstock 1 , Gonzalo

Principled Deep Neural Network Training through Linear Programming Daniel Bienstock 1 , Gonzalo Muoz 2 , Sebastian Pokutta 3 January 9, 2019 1 IEOR, Columbia University 2 IVADO, Polytechnique Montral 3 ISyE, Georgia Tech 1 ...Im

861 views • 55 slides
Optimality-based Domain Reductions for Global Optimization A. Caprara, M. Locatelli, M. Monaci

Optimality-based Domain Reductions for Global Optimization A. Caprara, M. Locatelli, M. Monaci

Optimality-based Domain Reductions for Global Optimization A. Caprara, M. Locatelli, M. Monaci Aussois, January 2013 M. Monaci (uniPD) Optimality-based Domain Reductions 1 / 22 Outline of the talk Settings: global optimization problems

858 views • 22 slides
AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality

AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality

AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality Conditions for Optimality Existence of Global Minimum In order to guarantee existence and uniqueness of a global min. we need to make assumptions about

862 views • 41 slides
Standalone Training of Context-Dependent Deep Neural Network Acoustic Models Chao Zhang &

Standalone Training of Context-Dependent Deep Neural Network Acoustic Models Chao Zhang &

Standalone Training of Context-Dependent Deep Neural Network Acoustic Models Chao Zhang & Phil Woodland University of Cambridge 11 November 2013 Conventional Training of CD-DNN-HMMs CD-DNN-HMMs rely on GMM-HMMs in two aspects:

170 views • 13 slides
GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training Jiezhong Qiu , Qibin Chen,

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training Jiezhong Qiu , Qibin Chen,

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training Jiezhong Qiu , Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, Jie Tang Real-world Graphs Biol Bi ologi ogical G Graph Question: Soci So cial

598 views • 20 slides
TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja, Alena Shilova Also: Olivier Beaumont Jan Huckelheim Nicola Ferrier Paul Hovland Gerard Gorman BACKGROUND BACKGROUND Typical data ow pattern

779 views • 35 slides
Training Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning

Training Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning

Training Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Training Neural Nets 1 / 40 Outline 1 The Softmax Simplex 2 Loss and Risk 3 Back-Propagation 4 Stochastic Gradient Descent 5 Regularization 6 Network Depth

675 views • 40 slides

More recommend