Highway Networks and Residual Networks Renjie Liao University of - PowerPoint PPT Presentation

Highway Networks and Residual Networks Renjie Liao University of Toronto Jan 26, 2016 Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 1 / 23

Neural Network A network connecting numerous neurons Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 2 / 23

Analogy Imagine a neural network as a map Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 3 / 23

Analogy Imagine a neural network as a map Imagine a neuron as a place Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 3 / 23

Analogy Imagine a neural network as a map Imagine a neuron as a place Imagine yourself as the information flow Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 3 / 23

Analogy Suppose you (information flow) wants to reach Bakery (neuron B) from City Hall (neuron A), what will you do? Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 4 / 23

Analogy Suppose you (information flow) wants to reach Bakery (neuron B) from City Hall (neuron A), what will you do? You have to follow the path of network! Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 4 / 23

Analogy Suppose you (information flow) wants to reach Bakery (neuron B) from City Hall (neuron A), what will you do? You have to follow the path of network! What if there is a highway connecting Bakery and City Hall directly? Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 4 / 23

Highway Networks Allowing direct pass (highway) between neurons in different layers. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 5 / 23

Highway Networks Original network: �� w 1 z 1 = σ n x n + b (1) n =1 Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 6 / 23

Highway Networks Original network: �� w 1 z 1 = σ n x n + b (1) n =1 Highway network: �� w 1 z 1 = T σ n x n + b + (1 − T ) x 1 (2) n =1 Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 6 / 23

Highway Networks Original network: �� w 1 z 1 = σ n x n + b (1) n =1 Highway network: �� w 1 z 1 = T σ n x n + b + (1 − T ) x 1 (2) n =1 Gating function: �� T = σ w ′ n x n + b ′ (3) n =1 Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 6 / 23

Highway Networks Remember the shape of sigmoid function. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 7 / 23

Highway Networks Remember the shape of sigmoid function. We can set bias b ′ to negative values such that gating value T → 0. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 7 / 23

Benefits of Highway networks Enable training of very deep neural networks (e.g., hundreds of layers) “Srivastava, R.K., Greff, K. and Schmidhuber, J., 2015. Highway Networks. arXiv preprint arXiv:1505.00387”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 8 / 23

Benefits of Highway networks Enable training of very deep neural networks (e.g., hundreds of layers) “Srivastava, R.K., Greff, K. and Schmidhuber, J., 2015. Highway Networks. arXiv preprint arXiv:1505.00387”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 9 / 23

Residual Networks Motivation: Does depth matter for deep learning? Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 10 / 23

Residual Networks Motivation: Does depth matter for deep learning? “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 10 / 23

Residual Networks Motivation: Does depth matter for deep learning? “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 11 / 23

Residual Networks We need new architecture to make depth matter. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 12 / 23

Residual Networks We need new architecture to make depth matter. Suppose you have a plain 2-layer network H . “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 12 / 23

Residual Networks We need new architecture to make depth matter. Suppose you have a plain 2-layer network H . We use a new building block which forces the previous 2-layer F to learn the residual H − x . Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 13 / 23

Residual Networks We need new architecture to make depth matter. Suppose you have a plain 2-layer network H . We use a new building block which forces the previous 2-layer F to learn the residual H − x . “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 13 / 23

Residual Networks What we have done? Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 14 / 23

Residual Networks What we have done? “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 14 / 23

Residual Networks Based on this building block, we can do some crazy things like... Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 15 / 23

Residual Networks Based on this building block, we can do some crazy things like... “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 15 / 23

Image Net Challenge Results “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 16 / 23

PASCAL VOC Challenge Results “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 17 / 23

More Results “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 18 / 23

Thanks! Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 23 / 23

Highway Networks and Residual Networks Renjie Liao University of - PowerPoint PPT Presentation

Highway Networks and Residual Networks Renjie Liao University of Toronto Jan 26, 2016 Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 1 / 23 Neural Network A network connecting numerous neurons Renjie Liao (UofT)

Architecture Perceptron Highway Networks Highway Networks Allows training very deep

Residual Networks (ResNet) Residual Networks (ResNet) In [1]: import d2l from mxnet import gluon,

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Invertible Residual Networks Jens Behrmann * Will Grathwohl* Ricky T. Q. Chen David Duvenaud

XDN: Towards Efficient Inference of Residual Neural Networks on Cambricon Chips 2019

AMMI Introduction to Deep Learning 6.5. Residual networks Fran cois Fleuret

Argumentative Link Prediction using Residual Networks and Multi-Objective Learning Galassi Andrea

Reduced-memory training and deployment of deep residual networks by stochastic binary quantization

Information Visualization networks without networks, couldn't have any of these: Dataset

Types of networks (social networks, computer networks, entity- relationship networks, )

Residual Flows for Invertible Generative Modeling Ricky T. Q. Chen, Jens Behrmann, David

Social and Technological Networks: Review Rik Sarkar Networks Networks/graphs are

1 Hardware architectures Point-to-point networks Many different types of networks: One

4-connected shift residual networks ICCV 2019 Neural Architects Workshop Andrew Brown, Pascal

Current Network Structure for Pediatrics Hospital Networks Country, state, regional, Academic

Computer Networks I Computer Networks I Networks A networks connection structure is known as

NETWORKS Networks There are many types of networks and network effects Physical:

R RITSUMEIKAN Introduction 1 Methodology Content 2 Experiments 3 Conclusion 4

Topics ! Use of networks ! Network structure ! Implementation of networks Computer Networks

Routing in Ad-hoc networks P R E S E N T E D B Y - L E W I S T S E N G R A C H I T A G A R W A

Very Deep Residual Networks with Maxout for Plant Identification in the Wild Mi l a n u

Types of Biological Networks Many important biological networks are defined on molecules such as

Mobility and cellular networks Mobility and cellular networks Cellular radio and PCS networks

Networks in economics Lecture 1 - Measuring networks and finance What are networks and why study