Neural Network Training: Old & New Tricks Old: (80s) Stochastic - PowerPoint PPT Presentation

Jun 08, 2023 •268 likes •1.61k views

Neural Network Training: Old & New Tricks Old: (80s) Stochastic Gradient Descent, Momentum, weight decay New: (last 5-6 years) Dropout ReLUs Batch Normalization Reminder: Overfitting, in images Classification just right

Residual Network Naïve solution • If extra layers are an identity mapping, then training errors can not increase 67
Residual Modelling: Basic idea in image processing • Goal: estimate update between an original image and a changed image Preserving base information Some residual Network can treat perturbation 68
Residual Network • Plain block • Difficult to make identity mapping because of multiple non-linear layers 69
Residual Network • Residual block • If identity were optimal, easy to set weights as 0 • If optimal mapping is closer to identity, easier to find small fluctuations Appropriate for treating perturbation as keeping a base information 70
Residual Network: Deeper is better • Deeper ResNets have lower training error 71
Residual Network: Deeper is better 72
CNNs, 2017: DenseNet Densely Connected Convolutional Networks, CVPR 2017 Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger Recently proposed, better performance/parameter ratio 73
Image-to-Image 74
Graphics: Multiresolution 75
Image-to-image • So far we mapped an image image to a number or label • In graphics, output often is “richer”: • An image • A volume • A 3D mesh • … • Note: “ image ” just placeholder name here for any Eulerian data • Architectures • Fully convolutional • Encoder-Decoder • Skip connections 76
Fully-convolutional Neural Networks FCNN 77
Fully-convolutional Neural Networks FCNN 78
Fully-convolutional Neural Networks FCNN 79
Fully-convolutional Neural Networks FCNN 80
Fully-convolutional Neural Networks FCNN Flexible - works with varying input sizes 81

Recommend

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics Activation Functions Stochastic Gradient Descent (SGD) Regularization (Dropout) Training Tips and Tricks Neural Network (NN)

828 views • 45 slides

Transparent parallelization of neural network training Cyprien Noel Flickr / Yahoo - GTC 2015

Transparent parallelization of neural network training Cyprien Noel Flickr / Yahoo - GTC 2015 by overthemoon Outline Neural Nets at Flickr Training Fast Parallel Distributed Q&A Tagging Photos Class Probability

562 views • 32 slides

CS4811 Neural Network Training Example Consider the following network. It has two inputs (two

CS4811 Neural Network Training Example Consider the following network. It has two inputs (two entries in each input vector), one hidden layer with two neurons and the output layer with a single neuron. Each neuron has a bias input to allow

216 views • 4 slides

IMPLEMENTATION OF A PARALLEL BATCH TRAINING ALGORITHM FOR DEEP NEURAL NETWORK YUPING LIN IFLYTEK

IMPLEMENTATION OF A PARALLEL BATCH TRAINING ALGORITHM FOR DEEP NEURAL NETWORK YUPING LIN IFLYTEK LABORATORY FOR NEURAL COMPUTING FOR MACHINE LEARNING DEPARTMENT OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE YORK UNIVERSITY, TORONTO NOVEMBER

212 views • 20 slides

Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi,

Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geofgrey Hinton Motivation We need tools to understand trained neural networks Neural network training involves interactions between an

588 views • 16 slides

FPGA-based Training Accelerator Utilizing Sparseness of Convolutional Neural Network Hiroki

FPGA-based Training Accelerator Utilizing Sparseness of Convolutional Neural Network Hiroki Nakahara, Youki Sada, Masayuki Shimoda, Akira Jinguji, Shimpei Sato Tokyo Institute of Technology, JP FPL2019 @Barcelona Challenges in DL Training

369 views • 10 slides

Language Modeling, Efficiency/Training Tricks Graham Neubig Site

CS11-747 Neural Networks for NLP Language Modeling, Efficiency/Training Tricks Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Are These Sentences OK? Jane went to the store. store to Jane went the. Jane went store.

687 views • 54 slides

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training the network Given a training set of input-output pairs Minimize the following function w.r.t This is problem of function minimization

1.79k views • 165 slides

MIT 9.520/6.860, Fall 2018 Class 11: Neural networks tips, tricks & software Andrzej

MIT 9.520/6.860, Fall 2018 Class 11: Neural networks tips, tricks & software Andrzej Banburski Last time - Convolutional neural networks source: github.com/vdumoulin/conv arithmetic Large-scale Datasets General Purpose GPUs AlexNet

1.46k views • 111 slides

On The Complexity of Training a Neural Network Santosh Vempala Algorithms and Randomness Center

On The Complexity of Training a Neural Network Santosh Vempala Algorithms and Randomness Center Georgia Tech The Complexity of Learning Neural Networks Deep learning s successes are incredible Do you want to beat 9 dan Goplayers? The

873 views • 55 slides

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides

Why is word2vec so fast? Efficiency tricks for neural nets Taylor Berg-Kirkpatrick Site

CS11-747 Neural Networks for NLP Why is word2vec so fast? Efficiency tricks for neural nets Taylor Berg-Kirkpatrick Site https://phontron.com/class/nn4nlp2017/ Glamorous Life of an AI Scientist Perception Reality Waiting. Photo Credit:

641 views • 34 slides

Artificial Neural Network : Training Debasis Samanta IIT Kharagpur

Artificial Neural Network : Training Debasis Samanta IIT Kharagpur debasis.samanta.iitkgp@gmail.com 06.04.2018 Debasis Samanta (IIT Kharagpur) Soft Computing Applications 06.04.2018 1 / 49 Learning of neural networks: Topics Concept of

608 views • 49 slides

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks Alexander Shevchenko Marco Mondelli Neural Network Training From theoretical perspective training of neural networks is di ffi cult

494 views • 28 slides

Efficiency Tricks for Neural Nets Graham Neubig Site https://phontron.com/class/nn4nlp2020/

CS11-747 Neural Networks for NLP Efficiency Tricks for Neural Nets Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Glamorous Life of an AI Scientist Perception Reality Waiting. Photo Credit: Antoine Miech @ Twitter Why are

852 views • 37 slides

Networks Luke Schuler Overview What is an Artificial Neural Network? History

Artificial Neural Networks Luke Schuler Overview What is an Artificial Neural Network? History Inception Abandonment Resurgence Model Training Applications What is an Artificial Neural Network? An artificial

616 views • 15 slides

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan, Laxmikant Kishor Mokadam, Xipeng Shen, Seung-Hwan Lim, Robert Patton 1 Build an image classifier? Deep Neural Network (DNN) CPU GPU Training

957 views • 41 slides

Backpropagation and Gradients Agenda Motivation Backprop Tips & Tricks

Backpropagation and Gradients Agenda Motivation Backprop Tips & Tricks Matrix calculus primer Example: 2-layer Neural Network Motivation Recall: Optimization objective is minimize loss Goal: how should we tweak the

428 views • 28 slides

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural IR tasks Neural IR architecture Feature Representations Neural IR query auto completion Neural IR query suggestion Neural IR document

1.48k views • 18 slides

Walking Tips & Tricks PRESENTED BY: PAWS IN TRAINING & MAGSTER DOG TRAINING Reality

Walking Tips & Tricks PRESENTED BY: PAWS IN TRAINING & MAGSTER DOG TRAINING Reality Check No matter how much you want to, YOU will not be able to train a clients dog to walk on a loose leash or to heel. Why? For a dog to learn

584 views • 20 slides

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides

Principled Deep Neural Network Training through Linear Programming Daniel Bienstock 1 , Gonzalo

Principled Deep Neural Network Training through Linear Programming Daniel Bienstock 1 , Gonzalo Muoz 2 , Sebastian Pokutta 3 January 9, 2019 1 IEOR, Columbia University 2 IVADO, Polytechnique Montral 3 ISyE, Georgia Tech 1 ...Im

861 views • 55 slides

Standalone Training of Context-Dependent Deep Neural Network Acoustic Models Chao Zhang &

Standalone Training of Context-Dependent Deep Neural Network Acoustic Models Chao Zhang & Phil Woodland University of Cambridge 11 November 2013 Conventional Training of CD-DNN-HMMs CD-DNN-HMMs rely on GMM-HMMs in two aspects:

170 views • 13 slides

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training Jiezhong Qiu , Qibin Chen,

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training Jiezhong Qiu , Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, Jie Tang Real-world Graphs Biol Bi ologi ogical G Graph Question: Soci So cial

598 views • 20 slides