Training Neural Networks Some considerations Gaurav Kumar Center - PowerPoint PPT Presentation

Mar 18, 2024 •982 likes •1.19k views

Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech Processing gkumar@cs.jhu.edu Universal Approximators Neural networks can approximate [1] function. any Capacity Layers hidden layer size

Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech Processing gkumar@cs.jhu.edu
Universal Approximators • Neural networks can approximate [1] function. any • Capacity • Layers • hidden layer size • A bsence of regularization • Optimal activation functions and hyper-parameters. • Training data [1] K. Hornik, M. Stinchcombe, and H. White. 1989. Multilayer feedforward networks are universal approximators. Neural Netw. 2, 5 (July 1989) : proved this for a specific class of functions.
Universal Approximators • We will focus on two important aspects of training: • Ideal properties of parameters during training • Generalization error • Other things to consider: • Hyper-parameter optimization • Choice of model, loss functions • Learning rates (Use Adadelta or Adam) • …
Properties of Parameters • Responsive to activation functions • Numerically stable
Activation Saturation Sigmoid Relu
Initialization of weight matrices • Are you using a non-recurrent NN ? • Use the Xavier initialization • (use small values to initialize bias vectors) Glorot & Bengio (2010), He et.al (2015)
Initialization of weight matrices (Xavier, He) • Tanh • Sigmoid • Relu
Initialization of weight matrices • Are you using a recurrent NN ? • With LSTMs : Use the Saxe initialization • All weight matrices initialized to be orthonormal (Gaussian noise -> SVD) • Without LSTMS • All weight matrices initialized to identity Saxe et al, 2014,
Watch your input • A high variance in input features may cause saturation very early • Mean subtraction : Same mean across all features • Normalization : Same scale across all features
Numerical stability • Floating point precision causes values to overflow or underflow • Instead, compute
Numerical stability L = − t log( p ) − (1 − t )log(1 − p ) • Cross Entropy Loss • Probabilities close to 0 for the correct label will cause underflow • Use range clipping. All values between 0.000001 and 0.999999.
Generalization Preventing Overfitting
Regularization • L2 regularization • L1 regularization • Gradient clipping (max norm constraints)
Regularization • Perform layer-wise regularization • After computing the activated value of each layer, normalize with the L2 norm. • No regularization hyper-parameters • No waiting till back-propagation for weight penalties to flow in
Dropout Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15
Dropout
Dropout • Interpret as regularization • Interpret as training an ensemble of thinned networks

Recommend

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja, Alena Shilova Also: Olivier Beaumont Jan Huckelheim Nicola Ferrier Paul Hovland Gerard Gorman BACKGROUND BACKGROUND Typical data ow pattern

779 views • 35 slides

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training Neural Networks 2.Leveraging heterogeneous hardware Source: wikipedia Modern Neural Networks Applications: Image classification Modern Neural

513 views • 20 slides

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear models for classification Predictions are made as a sequence of simple operations matrix-vector operations non-linear activation functions

417 views • 14 slides

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views • 105 slides

DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi ,

DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi , Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, Martin Vechev github.com/eth-sri/dl2 differencing neural networks finding inputs that deactivates [Pei

992 views • 13 slides

Introduction to Neural Networks Slides from L. Lazebnik, B. Hariharan Outline Perceptrons

Introduction to Neural Networks Slides from L. Lazebnik, B. Hariharan Outline Perceptrons Perceptron update rule Multi-layer neural networks Training method Best practices for training classifiers After that:

505 views • 25 slides

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks Alexander Shevchenko Marco Mondelli Neural Network Training From theoretical perspective training of neural networks is di ffi cult

494 views • 28 slides

Neural Networks and Handwriting Recognition Background Neural Networks Neural Network Steven

Neural Networks and Handwriting Recognition Steven Sloss Math 164 Neural Networks and Handwriting Recognition Background Neural Networks Neural Network Steven Sloss Structure Training Neural Networks Math 164 Motivation Problem

888 views • 41 slides

Population Based Training of Neural Networks M. Jaderberg, V. Dalibard, S. Osindero, W.M.

Presented by: Devin Taylor Population Based Training of Neural Networks M. Jaderberg, V. Dalibard, S. Osindero, W.M. Czarnecki November 14, 2018 DeepMind, London, United Kingdom Problem Statement Problem statement Neural networks suffer from

317 views • 17 slides

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan, Laxmikant Kishor Mokadam, Xipeng Shen, Seung-Hwan Lim, Robert Patton 1 Build an image classifier? Deep Neural Network (DNN) CPU GPU Training

957 views • 41 slides

Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2

Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2 1 University of Oxford, 2 EPFL ICLR 2020 Workshop on PML4DC Motivation Compressing neural networks can save a large amount of memory and

570 views • 21 slides

What Can Neural Networks Teach us about Language? Graham Neubig a2-dlearn 11/18/2017 Supervised

What Can Neural Networks Teach us about Language? Graham Neubig a2-dlearn 11/18/2017 Supervised Training of Neural Networks for Language Training Data Training Model this is an example the cat went to the store Prediction Results

1.07k views • 62 slides

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides

Training Neural Networks with Local Error Signals Arild Nkland Lars H. Eidnes Local learning

Training Neural Networks with Local Error Signals Arild Nkland Lars H. Eidnes Local learning Typically we train neural networks by backpropagating errors from the loss function and back through the layers. Hard to explain how the

647 views • 13 slides

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides

Parameter efficient training of deep convolutional neural networks by dynamic sparse

Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization Hesham Mostafa (Intel AI) Xin Wang (Intel AI, Cerebras Systems) Easy : post-training (sparse) compression Hard : direct training of sparse

727 views • 9 slides

Convolutional Neural Networks Computer Vision Jia-Bin Huang, Virginia Tech Todays class

Convolutional Neural Networks Computer Vision Jia-Bin Huang, Virginia Tech Todays class Overview Convolutional Neural Network (CNN) Training CNN Understanding and Visualizing CNN Image Categorization: Training phase Training

1.82k views • 68 slides