Training Neural Networks Using Features Replay Zhouyuan Huo 1 , Bin - PowerPoint PPT Presentation

Jun 06, 2023 •260 likes •359 views

Training Neural Networks Using Features Replay Zhouyuan Huo 1 , Bin Gu 1 , 2 , Heng Huang 1 , 2 1 Department of Electrical and Computer Engineering, University of Pittsburgh 2 JD.com November 28, 2018 Zhouyuan Huo 1 , Bin Gu 1 , 2 , Heng Huang 1 ,

Training Neural Networks Using Features Replay Zhouyuan Huo 1 , Bin Gu 1 , 2 , Heng Huang 1 , 2 1 Department of Electrical and Computer Engineering, University of Pittsburgh 2 JD.com November 28, 2018 Zhouyuan Huo 1 , Bin Gu 1 , 2 , Heng Huang 1 , 2 (Pitt) FR November 28, 2018 1 / 8
Motivation poster #12 Backpropagation algorithm: step 1: Forward pass. step 2: Backward pass. Problem: Backward time is about 2 times of forward time. Backward locking. Backward cannot be parallellized. Zhouyuan Huo 1 , Bin Gu 1 , 2 , Heng Huang 1 , 2 (Pitt) FR November 28, 2018 2 / 8
Problem Reformulation poster #12 New formulation: Original formulation: 2 ( w t ) � � K − 1 ∂ f ht � � min f ( h L , y ) � � δ t Lk � h t L K , y t � min k − + f � � ∂ h t w w ,δ Lk � k =1 2 s . t . h l = F l ( h l − 1 ; w l ) h t L k = F G ( k ) ( h t L k − 1 ; w t s . t . G ( k ) ) Zhouyuan Huo 1 , Bin Gu 1 , 2 , Heng Huang 1 , 2 (Pitt) FR November 28, 2018 3 / 8
Problem Reformulation (Continued) poster #12 Module 1: 2 � � ( w t ) ∂ f ht � � � δ t L 1 Module 4: min 1 − � � ∂ h t � � w ,δ L 1 � h t L 4 , y t � min f � 2 w ,δ h t L 1 = F G (1) ( h t L 0 ; w t G (1) ) s . t . h t L 4 = F G (4) ( h t L 3 ; w t s . t . G (4) ) ( w t ) ∂ f ht − 3 We approximate δ t L 1 1 = . ∂ h t − 3 L 1 Zhouyuan Huo 1 , Bin Gu 1 , 2 , Heng Huang 1 , 2 (Pitt) FR November 28, 2018 4 / 8
Features Replay poster #12 Module 1 Module 2 Module 3 Module 4 layer 1 layer 2 layer 3 layer 10 layer 11 layer 12 layer 4 layer 5 layer 6 layer 7 layer 8 layer 9 loss δ t δ t δ t 1 2 3 h t − 3 t t t t t t t t ˜ ˜ ˜ h t − 2 ˜ ˜ ˜ h t − 1 ˜ ˜ t h t h t h t h t h h h h h h h h ˜ h 0 1 2 3 3 4 5 10 11 6 6 7 8 9 9 12 h t − 2 h t − 1 h t 0 3 6 h t − 1 h t 0 3 h Forward pass Activation h t 0 Backward pass Error gradient δ Backward pass: Forward pass: ˜ L k = F G ( k ) ( h t + k − K h t ; w t G ( k ) ) (Replay) L k − 1 � � h t h t L k − 1 ; w t L k = F G ( k ) (Play) Apply chain rule using ˜ h t L k and δ t G ( k ) k in each module. Zhouyuan Huo 1 , Bin Gu 1 , 2 , Heng Huang 1 , 2 (Pitt) FR November 28, 2018 5 / 8
Convergence Guarantee poster #12 Convergence Guarantee: T − 1 γ 2 � T − 1 t f ( w 0 ) − f ( w ∗ ) 1 + LM � 2 � t =0 � � ∇ f ( w t ) � γ t E . (1) ≤ 2 2 σ T − 1 T − 1 T − 1 � t =0 � � γ t σ γ t γ t t =0 t =0 t =0 Zhouyuan Huo 1 , Bin Gu 1 , 2 , Heng Huang 1 , 2 (Pitt) FR November 28, 2018 6 / 8
Experimental Results poster #12 Faster Convergence. Lower Memory Consumption. Better Generalization Error. Zhouyuan Huo 1 , Bin Gu 1 , 2 , Heng Huang 1 , 2 (Pitt) FR November 28, 2018 7 / 8
Thanks ! Welcome to poster #12 Room 210 & 230 AB Zhouyuan Huo 1 , Bin Gu 1 , 2 , Heng Huang 1 , 2 (Pitt) FR November 28, 2018 8 / 8

Recommend

Brain-like replay for continual learning with artificial neural networks Gido M van de Ven, Hava

Brain-like replay for continual learning with artificial neural networks Gido M van de Ven, Hava T Siegelmann, Andreas S Tolias Bridging AI and Cognitive Science workshop (ICLR 2020) Catastrophic forgetting in neural networks When a

452 views • 16 slides

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja, Alena Shilova Also: Olivier Beaumont Jan Huckelheim Nicola Ferrier Paul Hovland Gerard Gorman BACKGROUND BACKGROUND Typical data ow pattern

779 views • 35 slides

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides

Neural Networks + Convolutional Neural Networks Last Class Global Features The perceptron

CS4501: Introduction to Computer Vision Neural Networks + Convolutional Neural Networks Last Class Global Features The perceptron model Neural Networks multilayer perceptron model (MLP) Backpropagation Todays Class

789 views • 34 slides

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides

Training neural networks Today's lecture Learning from small data Curriculum: Active

Training neural networks Today's lecture Learning from small data Curriculum: Active learning - How transferable are features in deep neural When you are not learning networks? Surrogat losses

1k views • 69 slides

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training Neural Networks 2.Leveraging heterogeneous hardware Source: wikipedia Modern Neural Networks Applications: Image classification Modern Neural

513 views • 20 slides

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear models for classification Predictions are made as a sequence of simple operations matrix-vector operations non-linear activation functions

417 views • 14 slides

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views • 105 slides

DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi ,

DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi , Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, Martin Vechev github.com/eth-sri/dl2 differencing neural networks finding inputs that deactivates [Pei

992 views • 13 slides

Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech

Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech Processing gkumar@cs.jhu.edu Universal Approximators Neural networks can approximate [1] function. any Capacity Layers hidden layer size

1.19k views • 17 slides

Introduction to Neural Networks Slides from L. Lazebnik, B. Hariharan Outline Perceptrons

Introduction to Neural Networks Slides from L. Lazebnik, B. Hariharan Outline Perceptrons Perceptron update rule Multi-layer neural networks Training method Best practices for training classifiers After that:

505 views • 25 slides

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks Alexander Shevchenko Marco Mondelli Neural Network Training From theoretical perspective training of neural networks is di ffi cult

494 views • 28 slides

Neural Networks and Handwriting Recognition Background Neural Networks Neural Network Steven

Neural Networks and Handwriting Recognition Steven Sloss Math 164 Neural Networks and Handwriting Recognition Background Neural Networks Neural Network Steven Sloss Structure Training Neural Networks Math 164 Motivation Problem

889 views • 41 slides

Population Based Training of Neural Networks M. Jaderberg, V. Dalibard, S. Osindero, W.M.

Presented by: Devin Taylor Population Based Training of Neural Networks M. Jaderberg, V. Dalibard, S. Osindero, W.M. Czarnecki November 14, 2018 DeepMind, London, United Kingdom Problem Statement Problem statement Neural networks suffer from

317 views • 17 slides

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan, Laxmikant Kishor Mokadam, Xipeng Shen, Seung-Hwan Lim, Robert Patton 1 Build an image classifier? Deep Neural Network (DNN) CPU GPU Training

957 views • 41 slides

Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2

Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2 1 University of Oxford, 2 EPFL ICLR 2020 Workshop on PML4DC Motivation Compressing neural networks can save a large amount of memory and

570 views • 21 slides

What Can Neural Networks Teach us about Language? Graham Neubig a2-dlearn 11/18/2017 Supervised

What Can Neural Networks Teach us about Language? Graham Neubig a2-dlearn 11/18/2017 Supervised Training of Neural Networks for Language Training Data Training Model this is an example the cat went to the store Prediction Results

1.07k views • 62 slides

Neural Networks + Backpropagation Last Class Softmax Classifier Generalization /

CS4501: Introduction to Computer Vision Neural Networks + Backpropagation Last Class Softmax Classifier Generalization / Overfitting Pytorch Todays Class Global Features The perceptron model Neural Networks

838 views • 34 slides

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides

Training Neural Networks with Local Error Signals Arild Nkland Lars H. Eidnes Local learning

Training Neural Networks with Local Error Signals Arild Nkland Lars H. Eidnes Local learning Typically we train neural networks by backpropagating errors from the loss function and back through the layers. Hard to explain how the

647 views • 13 slides