an overview of deep residual learning
play

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 - PowerPoint PPT Presentation

Feb 05, 2023 •768 likes •1.32k views

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning Microsoft Research Asia (MSRA) Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. Deep Residual Learning for Image Recognition. arXiv

  1. An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016

  2. Deep Residual Learning •Microsoft Research Asia (MSRA) •Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015. •Shaoqing Ren, Kaiming He, Ross Girshick, & Jian Sun. “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”. NIPS 2015. •ILSVRC & COCO 2015 competitions

  3. MSRA @ ILSVRC & COCO 2015 Competitions • 1st places in all five main tracks • ImageNet Classification: “ Ultra-deep ” 152-layer nets • ImageNet Detection: 16% better than 2nd • ImageNet Localization: 27% better than 2nd • COCO Detection: 11% better than 2nd • COCO Segmentation: 12% better than 2nd *improvements are relative numbers Slide Credit: He et al. (MSRA)

  8. Revolution of Depth 28.2 25.8 152 layers 16.4 11.7 22 layers 19 layers 7.3 6.7 3.57 8 layers 8 layers shallow ILSVRC'14 ILSVRC'15 ILSVRC'14 ILSVRC'13 ILSVRC'12 ILSVRC'11 ILSVRC'10 ResNet GoogleNet VGG AlexNet ImageNet Classification top-5 error (%) Slide Credit: He et al. (MSRA)

  9. 101 layers Revolution of Depth 86 Engines of visual 66 recognition 58 34 16 layers 8 layers shallow AlexNet VGG ResNet HOG, DPM (RCNN) (RCNN) (Faster RCNN)* PASCAL VOC 2007 Object Detection mAP (%) *w/ other improvements & more data Slide Credit: He et al. (MSRA)

  10. Residual learning reformulates the learning procedure and redirects the information flow in deep neural networks.

  11. Revolution of Depth AlexNet, 8 11x11 conv, 96, /4, pool/2 layers (ILSVRC 2012) 5x5 conv, 256, pool/2 3x3 conv, 384 3x3 conv, 384 3x3 conv, 256, pool/2 fc, 4096 fc, 4096 fc, 1000 Slide Credit: He et al. (MSRA)

  12. Revolution of Depth s o f t m a x 2 S oft max A c t i v a t i o n FC Av e r a ge Po o l 7 x 7 + 1 (V) AlexNet, 8 11x11 conv, 96, /4, pool/2 VGG, 19 GoogleNet, 22 3x3 conv, 64 D e pt h Co n c a t C o n v C o n v C o n v C o n v 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 5 x 5 + 1 (S ) 1 x 1 + 1 (S ) 5x5 conv, 256, pool/2 3x3 conv, 64, pool/2 layers layers layers 1 x 1 + 1 (S ) 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) D e pt h Co n c a t 3x3 conv, 384 3x3 conv, 128 C o n v C o n v C o n v C o n v 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 5 x 5 + 1 (S ) 1 x 1 + 1 (S ) (ILSVRC 2012) (ILSVRC (ILSVRC 2014) s o f t m a x 1 C o n v C o n v M a x P o o l 1 x 1 + 1 (S ) 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) S oft max A c t i v a t i o n 3x3 conv, 384 3x3 conv, 128, pool/2 Max P ool 3 x 3 + 2 (S ) FC 2014) D e pt h Co n c a t FC 3x3 conv, 256, pool/2 3x3 conv, 256 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 5 x 5 + 1 (S ) 1 x 1 + 1 (S ) 1 x 1 + 1 (S ) C o n v C o n v M a x P o o l Av e r a g e P o o l 1 x 1 + 1 (S ) 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 5 x 5 + 3 (V) fc, 4096 3x3 conv, 256 D e pt h Co n c a t C o n v C o n v C o n v C o n v 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 5 x 5 + 1 (S ) 1 x 1 + 1 (S ) fc, 4096 3x3 conv, 256 C o n v C o n v M a x P o o l 1 x 1 + 1 (S ) 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) D e pt h Co n c a t s o f t m a x 0 fc, 1000 3x3 conv, 256, pool/2 C o n v C o n v C o n v C o n v 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 5 x 5 + 1 (S ) 1 x 1 + 1 (S ) S oft max A c t i v a t i o n C o n v C o n v M a x P o o l 1 x 1 + 1 (S ) 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) FC 3x3 conv, 512 D e p t h C o n c a t F C C o n v C o n v C o n v C o n v C o n v 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 5 x 5 + 1 (S ) 1 x 1 + 1 (S ) 1 x 1 + 1 (S ) 3x3 conv, 512 C o n v C o n v M a x P o o l Av e r a g e P o o l 1 x 1 + 1 (S ) 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 5 x 5 + 3 (V) D e pt h Co n c a t 3x3 conv, 512 C o n v C o n v C o n v C o n v 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 5 x 5 + 1 (S ) 1 x 1 + 1 (S ) C o n v C o n v M a x P o o l 1 x 1 + 1 (S ) 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 3x3 conv, 512, pool/2 Max P ool 3 x 3 + 2 (S ) D e pt h Co n c a t 3x3 conv, 512 C o n v C o n v C o n v C o n v 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 5 x 5 + 1 (S ) 1 x 1 + 1 (S ) C o n v C o n v M a x P o o l 1 x 1 + 1 (S ) 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 3x3 conv, 512 D e pt h Co n c a t C o n v C o n v C o n v C o n v 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) 5 x 5 + 1 (S ) 1 x 1 + 1 (S ) 3x3 conv, 512 C o n v C o n v M a x P o o l 1 x 1 + 1 (S ) 1 x 1 + 1 (S ) 3 x 3 + 1 (S ) Max P ool 3 x 3 + 2 (S ) 3x3 conv, 512, pool/2 L o c a l R e s p N o r m C o n v 3 x 3 + 1 (S ) fc, 4096 C o n v 1 x 1 + 1 (V) L o c a l R e s p N o r m fc, 4096 Max P ool 3 x 3 + 2 (S ) C o n v 7 x 7 + 2 (S ) fc, 1000 Slide Credit: He et al. (MSRA) i n pu t

  13. 7x7 conv, 64, /2, pool /2 1x1 co nv , 64 3x3 co nv , 64 1x1 co nv , 256 1x1 co nv , 64 3x3 co nv , 64 1x1 co nv , 256 1x1 co nv , 64 3x3 co nv , 64 1x1 co nv , 256 1x1 co nv , 128, /2 3x3 co nv , 128 Revolution of Depth 1x1 co nv , 512 1x1 co nv , 128 3x3 co nv , 128 1x1 co nv , 512 1x1 co nv , 128 3x3 co nv , 128 1x1 co nv , 512 1x1 co nv , 128 3x3 co nv , 128 1x1 co nv , 512 1x1 co nv , 128 3x3 co nv , 128 1x1 co nv , 512 1x1 co nv , 128 3x3 co nv , 128 1x1 co nv , 512 1x1 co nv , 128 3x3 co nv , 128 1x1 co nv , 512 1x1 co nv , 128 3x3 co nv , 128 1x1 co nv , 512 1x1 co nv , 256, /2 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 5x5 conv , 256, pool/2 3x3 conv , AlexNet, 8 VGG, 19 ResNet, 152 1x1 co nv , 1024 384 1x1 co nv , 256 3x3 conv , 64 3x3 co nv , 256 3x3 conv , 64, pool/2 3x3 conv , 3x3 conv , 384 3x3 conv , 256, 1x1 co nv , 1024 128 pool/2 1x1 co nv , 256 3x3 conv , 128, pool/2 fc, 4096 3x3 conv , 256 fc, 4096 3x3 conv , 256 fc, 1000 3x3 conv , 256 layers layers layers 3x3 co nv , 256 3x3 conv , 256, pool/2 3x3 conv , 1x1 co nv , 1024 512 1x1 co nv , 256 3x3 conv , 512 3x3 conv , 512 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 conv , 512, pool/2 3x3 conv , 512 3x3 conv , 512 3x3 co nv , 256 1x1 co nv , 1024 (ILSVRC 2012) (ILSVRC (ILSVRC 2015) 1x1 co nv , 256 3x3 conv , 512 3x3 conv , 512, pool/2 fc, 4096 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 fc, 4096 fc, 1000 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 2014) 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 256 3x3 co nv , 256 1x1 co nv , 1024 1x1 co nv , 512, /2 3x3 co nv , 512 1x1 co nv , 2048 1x1 co nv , 512 3x3 co nv , 512 1x1 co nv , 2048 1x1 co nv , 512 3x3 co nv , 512 1x1 co nv , 2048 ave pool, fc 1000 Slide Credit: He et al. (MSRA)

Recommend

Deep Residual Learning for Portfolio Optimization: With Attention and Switching Modules Jeff

Deep Residual Learning for Portfolio Optimization: With Attention and Switching Modules Jeff

Deep Residual Learning for Portfolio Optimization: With Attention and Switching Modules Jeff Wang, Ph.D. Prepared for NYU FRE Seminar. March 7th, 2019 1 / 48 Overview Study model driven portfolio management strategies Construct

694 views • 48 slides
Deep Residual Learning for Image Recognition ILSVRC 2015 MS COCO 2015 K. He, X. Zhang, S. Ren

Deep Residual Learning for Image Recognition ILSVRC 2015 MS COCO 2015 K. He, X. Zhang, S. Ren

Deep Residual Learning for Image Recognition ILSVRC 2015 MS COCO 2015 K. He, X. Zhang, S. Ren and J. Sun WINNER Microsoft Research Article overview by Ilya Kuzovkin Computational Neuroscience Seminar University of Tartu 2016 T HE I DEA

594 views • 44 slides
Deep Residual Learning for Image Recognition Kaiming He et al. (Microsoft Research) By Zana

Deep Residual Learning for Image Recognition Kaiming He et al. (Microsoft Research) By Zana

2017-11-15 Deep Residual Learning for Image Recognition Kaiming He et al. (Microsoft Research) By Zana Rashidi (MSc student, York University) Introduction 1 2017-11-15 ILSVRC & COCO 2015 Competitions 1st place in all five main tracks :

480 views • 19 slides
Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Deep 3D Representation Learning for Visual Computing Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms Conclusion 2 Outline Overview of 3D deep learning Background 3D deep learning tasks 3D deep

1.66k views • 122 slides
Excitation Deep Learning Model Abdul Qayyum ImViA laboratory, University of Bourgogne

Excitation Deep Learning Model Abdul Qayyum ImViA laboratory, University of Bourgogne

Medi dical cal Imagin ging g with th Deep ep Learni ning ng 2020 20 Short t Paper er MID IDL L 2020 20 Segmentation of the Myocardium on Late-Gadolinium Enhanced MRI based on 2.5 D Residual Squeeze and Excitation Deep Learning

203 views • 5 slides
AMMI Introduction to Deep Learning 6.5. Residual networks Fran cois Fleuret

AMMI Introduction to Deep Learning 6.5. Residual networks Fran cois Fleuret

AMMI Introduction to Deep Learning 6.5. Residual networks Fran cois Fleuret https://fleuret.org/ammi-2018/ Fri Nov 9 22:38:28 UTC 2018 COLE POLYTECHNIQUE FDRALE DE LAUSANNE Residual networks Fran cois Fleuret AMMI

732 views • 36 slides
Reduced-memory training and deployment of deep residual networks by stochastic binary quantization

Reduced-memory training and deployment of deep residual networks by stochastic binary quantization

Reduced-memory training and deployment of deep residual networks by stochastic binary quantization Mark D. McDonnell 1 , Ruchun Wang 2 and Andr van Schaik 2 cls-lab.org 2 BENS Laboratory 1 Computational Learning Systems Laboratory MARCS

256 views • 23 slides
Solving Differential Equations through Means of Deep Learning Juliane Braunsmann February 8,

Solving Differential Equations through Means of Deep Learning Juliane Braunsmann February 8,

Solving Differential Equations through Means of Deep Learning Juliane Braunsmann February 8, 2019 Table of Contents Neural Networks 1 Introduction to machine learning applied to PDEs 2 Learning with a residual loss... 3 ... and hard

380 views • 16 slides
Deep Learning: From Theory to Algorithm Outline: 1. Overview of

Deep Learning: From Theory to Algorithm Outline: 1. Overview of

Deep Learning: From Theory to Algorithm Outline: 1. Overview of theoretical studies of deep learning 2. Optimization theory of deep neural networks 1) Gradient finds global optima 2) Gram-Gauss-Newton Algorithm

484 views • 31 slides
Pipeline Strategies and conversations behind securing a Residual Bequest Agenda 1. Why Residual?

Pipeline Strategies and conversations behind securing a Residual Bequest Agenda 1. Why Residual?

The Perpetual Pipeline Strategies and conversations behind securing a Residual Bequest Agenda 1. Why Residual? 2. Structured for Residual 3. Residual Results 4. Residual Strategy 5. Residual Relationships 6. Residual Data 7.

1.17k views • 43 slides
Course Information CS 6956: Deep Learning for NLP What we will see A general overview of

Course Information CS 6956: Deep Learning for NLP What we will see A general overview of

Course Information CS 6956: Deep Learning for NLP What we will see A general overview of underlying concepts that pervade deep learning for NLP tasks A collection of successful design ideas to handle sparse, compositional varying sized

146 views • 10 slides
The vanishing gradient problem revisited: Highway and residual connections CS 6956: Deep

The vanishing gradient problem revisited: Highway and residual connections CS 6956: Deep

The vanishing gradient problem revisited: Highway and residual connections CS 6956: Deep Learning for NLP Revisiting the vanishing gradient problem Stems from the fact that the derivative of the activation is between zero and one and

543 views • 17 slides
Think Deep Learning: Overview Ju Sun Computer Science & Engineering University of Minnesota,

Think Deep Learning: Overview Ju Sun Computer Science & Engineering University of Minnesota,

Think Deep Learning: Overview Ju Sun Computer Science & Engineering University of Minnesota, Twin Cities January 21, 2020 1 / 28 Outline Why deep learning? Why first principles? Our topics Course logistics 2 / 28 What is Deep

1.54k views • 67 slides
Deep Learning for NLP Kiran Vodrahalli Feb 11, 2015 Overview What is NLP? Natural

Deep Learning for NLP Kiran Vodrahalli Feb 11, 2015 Overview What is NLP? Natural

Deep Learning for NLP Kiran Vodrahalli Feb 11, 2015 Overview What is NLP? Natural Language Processing We try to extract meaning from text: sentiment, word sense, semantic similarity, etc. How does Deep Learning relate? NLP

2.02k views • 124 slides
Deep Learning for Predictive Maintenance Pawel Morkisz GTC 2017 Agenda Problem

Deep Learning for Predictive Maintenance Pawel Morkisz GTC 2017 Agenda Problem

Deep Learning for Predictive Maintenance Pawel Morkisz GTC 2017 Agenda Problem Introduction and notion of deep neural networks o Convolutional layers o Residual networks (ResNet) One dimensional convolutional networks in failure

276 views • 23 slides
ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY MEDIA & ENTERTAINMENT SECURITY & DEFENSE AUTONOMOUS MACHINES Image

308 views • 26 slides
Multiple resolution residual network for automatic thoracic organs-at-risk segmentation from CT

Multiple resolution residual network for automatic thoracic organs-at-risk segmentation from CT

Medical Imaging with Deep Learning 2020 Multiple resolution residual network for automatic thoracic organs-at-risk segmentation from CT Hyemin Um , Jue Jiang, Maria Thor, Andreas Rimner, Leo Luo, Joseph O. Deasy, Harini Veeraraghavan Department

305 views • 6 slides
Deep Learning Tutorial Part II Greg Shakhnarovich TTI-Chicago December 2016 Deep Learning

Deep Learning Tutorial Part II Greg Shakhnarovich TTI-Chicago December 2016 Deep Learning

Deep Learning Tutorial Part II Greg Shakhnarovich TTI-Chicago December 2016 Deep Learning Tutorial,Part II 1 Overview Goals of the tutorial Somewhat organized overview of basics, and some more advanced topics Demistify jargon Pointers

925 views • 63 slides
Deep Learning Tutorial Part I Greg Shakhnarovich TTI-Chicago December 2016 Deep Learning

Deep Learning Tutorial Part I Greg Shakhnarovich TTI-Chicago December 2016 Deep Learning

Deep Learning Tutorial Part I Greg Shakhnarovich TTI-Chicago December 2016 Deep Learning Tutorial,Part I 1 Overview Goals of the tutorial Somewhat organized overview of basics, and some more advanced topics Demistify jargon Pointers for

1.25k views • 105 slides
Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice Scaling up DL 2 What is Deep Learning? 3 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY SECURITY & DEFENSE MEDIA &

1.45k views • 39 slides
9.1 Overview 9 Deep Learning Alexander Smola Introduction to Machine Learning 10-701

9.1 Overview 9 Deep Learning Alexander Smola Introduction to Machine Learning 10-701

9.1 Overview 9 Deep Learning Alexander Smola Introduction to Machine Learning 10-701 http://alex.smola.org/teaching/10-701-15 A brief history of computers 1970s 1980s 1990s 2000s 2010s Data 10 5 10 8 10 2 10 3 10 11 RAM ? 1MB 100MB

932 views • 57 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Distributed DeepLearning at Scale Soumith Chintala Facebook AI Research Overview Deep

Distributed DeepLearning at Scale Soumith Chintala Facebook AI Research Overview Deep

Distributed DeepLearning at Scale Soumith Chintala Facebook AI Research Overview Deep Learning Research at FAIR Deep Learning on GPUs Deep Learning at scale Emerging Trends Deep Learning Research at Facebook AI Research Image

1.18k views • 50 slides
Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI Mission Helping people solve challenging problems using AI and deep learning. Developers, data scientists and engineers Self-driving cars,

1.44k views • 72 slides

More recommend