making convolutional networks shift invariant again
play

Making Convolutional Networks Shift-Invariant Again Richard Zhang - PowerPoint PPT Presentation

Apr 12, 2023 •668 likes •1.36k views

Making Convolutional Networks Shift-Invariant Again Richard Zhang Adobe Research Example classifications P(correct class) P(correct class) not Shift-Invariant Deep Networks are not P(correct class) P(correct class) not Shift-Invariant Deep

  1. Making Convolutional Networks Shift-Invariant Again Richard Zhang Adobe Research

  2. Example classifications P(correct class) P(correct class)

  3. not Shift-Invariant Deep Networks are not P(correct class) P(correct class)

  4. not Shift-Invariant Deep Networks are not P(correct class) P(correct class) Azulay and Weiss. Why y do deep convo volutional networks ks generalize ze so so poorly y to sm small image transf sformations? s? In ArXiv, 2018. Engstrom, Tsipras, Schmidt, Madry. Exp xploring the Landsc scape of Spatial Robust stness. ss. In ICML, 2019.

  5. Why is shift-invariance lost? 5

  6. Why is shift-invariance lost? “ Convo volutions are sh shift-equiva variant ” 6

  7. Why is shift-invariance lost? “ Convo volutions are sh shift-equiva variant ” “ Po Poolin ling builds up sh shift-inva variance ” 7

  8. Why is shift-invariance lost? “ Convo volutions are sh shift-equiva variant ” “ Po Poolin ling builds up sh shift-inva variance ” …but st striding ignores Nyquist sampling theorem and aliase ses 8

  9. Re-examining Max-Pooling 9

  10. Re-examining Max-Pooling max 10

  11. Re-examining Max-Pooling max 11

  12. Re-examining Max-Pooling max 12

  13. Re-examining Max-Pooling max 13

  14. Re-examining Max-Pooling max 14

  15. Re-examining Max-Pooling max 15

  16. Re-examining Max-Pooling max 16

  17. Re-examining Max-Pooling max 17

  18. Re-examining Max-Pooling 18

  19. Re-examining Max-Pooling max 19

  20. Re-examining Max-Pooling max 20

  21. Re-examining Max-Pooling max 21

  22. Re-examining Max-Pooling max 22

  23. Re-examining Max-Pooling max 23

  24. Re-examining Max-Pooling max 24

  25. Re-examining Max-Pooling 25

  26. Re-examining Max-Pooling 26

  27. Re-examining Max-Pooling 27

  28. Re-examining Max-Pooling Max-pooling breaks ks shift-equivariance 28

  29. softmax classifier Shift-equivariance in VGG 1x1 pool5 • CIFAR conv5 • VGG network pool4 • 5 max-pools conv4 pool3 • Test shift-equivariance condition • conv3 pool2 conv2 pool1 conv1 pixels 32x32

  30. softmax classifier Shift-equivariance in VGG 1x1 pool5 • CIFAR conv5 • VGG network pool4 • 5 max-pools conv4 pool3 • Test shift-equivariance condition • conv3 pool2 conv2 pool1 conv1 pixels 32x32

  31. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 conv3 Perfect shift-eq. pool2 conv2 pool1 conv1 v1 31 pixels

  32. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 conv3 Convolution is shift-equivariant Perfect shift-eq. pool2 conv2 pool1 conv1 v1 32 pixels

  33. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 conv3 Perfect shift-eq. pool2 conv2 pool1 po conv1 33 pixels

  34. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 conv3 Pooling breaks shift-equivariance Perfect shift-eq. pool2 conv2 pool1 po conv1 34 pixels

  35. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 conv3 Perfect shift-eq. pool2 po conv2 pool1 conv1 pixels

  36. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 po conv3 Perfect shift-eq. pool2 conv2 pool1 conv1 pixels

  37. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 po conv4 pool3 conv3 Perfect shift-eq. pool2 conv2 pool1 conv1 pixels

  38. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 po pool4 conv4 pool3 Nyquist theorem ignored when pooling; conv3 sing breaks shift-equivariance aliasi Perfect shift-eq. pool2 conv2 pool1 conv1 pixels

  39. Alternative downsampling methods • Blur+subsample • Antialiasi sing in signal processing; image processing; graphics • Max-pooling • Performs better in deep learning applications [Scherer 2010] 39

  40. Alternative downsampling methods • Blur+subsample • Antialiasi sing in signal processing; image processing; graphics • Max-pooling • Performs better in deep learning applications [Scherer 2010] 40

  41. Alternative downsampling methods • Blur+subsample • Antialiasi sing in signal processing; image processing; graphics • Max-pooling • Performs better in deep learning applications [Scherer 2010] Reconcile antialiasing with max-pooling 41

  42. max( x( ) Baseline (MaxPool) max( x( ) heavy vy aliasi sing

  43. max( x( ) Baseline (MaxPool) max( x( ) heavy vy aliasi sing

  44. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) heavy vy aliasi sing no aliasi sing

  45. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing

  46. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing

  47. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing max( x( ) Anti-aliased (MaxBlurPool) max( x( ) (1) Max x (dense se eva valuation) no aliasi sing

  48. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing max( x( ) Anti-aliased Blur Bl (MaxBlurPool) max( x( ) (1) Max x (dense se eva valuation) (2 (2) ) Anti Anti-aliasi sing filter filter no aliasi sing no aliasi sing

  49. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing max( x( ) Anti-aliased Blur Bl (MaxBlurPool) max( x( ) (1) Max x (dense se eva valuation) (2) (2 ) Anti-aliasi sing filter (3) Subsa sampling no aliasi sing no aliasi sing reduced aliasi sing

  50. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing max( x( ) Anti-aliased Blur Bl (MaxBlurPool) max( x( ) Evaluated together as “BlurPool” (1) Max x (dense se eva valuation) (2) (2 ) Anti-aliasi sing filter (3) Subsa sampling no aliasi sing no aliasi sing reduced aliasi sing

  51. Antialiasing any downsampling layer • Max Pool • VGG, Alexnet • Strided Convolution • Resnet, MobileNetv2 • Average Pool • DenseNet 51

  52. Antialiasing any downsampling layer • Max Pool • VGG, Alexnet • Strided Convolution • Resnet, MobileNetv2 • Average Pool • DenseNet 52

  53. Antialiasing any downsampling layer • Max Pool • VGG, Alexnet • Strided Convolution • Resnet, MobileNetv2 • Average Pool • DenseNet 53

  54. ImageNet Shift-invariance

  55. ImageNet Shift-invariance Accuracy

  56. ImageNet Shift-invariance Accuracy

  57. ImageNet Baseline Shift-invariance Accuracy

  58. ImageNet Baseline Shift-invariance Antialiased Accuracy

  59. ImageNet Baseline Shift-invariance Antialiased Accuracy

  60. ImageNet Baseline Shift-invariance Antialiased Antialiasing also improves accur accuracy acy Accuracy

  61. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 61

  62. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 62

  63. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 63

  64. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 64

  65. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 65

  66. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 66

  67. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness Antialiasing code, pretrained models https://richzhang.github.io/antialiased-cnns/ 67

Recommend

Convolutional Neural Networks Basics Praveen Krishnan Overview Paradigm Shift Simple

Convolutional Neural Networks Basics Praveen Krishnan Overview Paradigm Shift Simple

Convolutional Neural Networks Basics Praveen Krishnan Overview Paradigm Shift Simple Network Convolutional Network Layers Case Study1: Alex Net Training Generalization Visualizations Transfer Learning Case

1.02k views • 53 slides
Lecture 22: Linear Shift-Invariant (LSI) Systems and Convolution April 26, 2016. Linear

Lecture 22: Linear Shift-Invariant (LSI) Systems and Convolution April 26, 2016. Linear

Mathematical Tools for Neuroscience (NEU 314) Princeton University, Spring 2016 Jonathan Pillow Lecture 22: Linear Shift-Invariant (LSI) Systems and Convolution April 26, 2016. Linear Shift-Invariant (aka time-invariant) Systems An LSI

302 views • 4 slides
Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Transfer Learning with Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural Networks A breakthough Convolutional Neural Networks VGG-16 example Layers of Convolutional filters Bottleneck

625 views • 23 slides
Shift Invariant Spaces and BMO Morten Nielsen Department of Mathematical Sciences Aalborg

Shift Invariant Spaces and BMO Morten Nielsen Department of Mathematical Sciences Aalborg

Shift Invariant Spaces and BMO Morten Nielsen Department of Mathematical Sciences Aalborg University mnielsen@math.aau.dk Joint work with H. Siki c Fourier Talks, University of Maryland, February 20-21, 2014 M. Nielsen Shift Invariant

621 views • 19 slides
Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN)

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN)

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN) A.k.a. CNN or ConvNet Adit Deshpande, A Beginner's Guide To Understanding Convolutional Neural Networks. Digital Images Input array: an images

1.42k views • 72 slides
An Evolutionary View on Reversible Shift-invariant Transformations Luca Mariot, Stjepan Picek,

An Evolutionary View on Reversible Shift-invariant Transformations Luca Mariot, Stjepan Picek,

An Evolutionary View on Reversible Shift-invariant Transformations Luca Mariot, Stjepan Picek, Domagoj Jakobovic, Alberto Leporati l.mariot@tudelft.nl EuroGP 2020, 1517 April 2020 Outline Shift-invariant Transformations and Cellular

430 views • 25 slides
15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter

15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter

15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter (this lecture) and Ariel Procaccia Carnegie Mellon University Spring 2017 1 Outline Convolutional neural networks Applications of convolutional

1.32k views • 32 slides
LSI system Input v v1 x v1 x v2 x v2 x + + L + v3 x + v3 x v4 x + v4 x + Output

LSI system Input v v1 x v1 x v2 x v2 x + + L + v3 x + v3 x v4 x + v4 x + Output

Mathematical Tools for Neural and Cognitive Science Fall semester, 2017 Section 3: Linear Shift-invariant Systems 1 Linear shift-invariant (LSI) systems Linearity (previously discussed): linear combination in, linear combination out

497 views • 24 slides
Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional Neural Networks

648 views • 26 slides
and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast

and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast

Band-limited Training and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast Training of Convolutional Networks through FFTs Fast Convolutional Nets With fbfft: A GPU Performance Evaluation Data: x

648 views • 46 slides
Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional

444 views • 27 slides
Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12 Convolutional Networks Scale up neural networks to process very large images / video sequences Sparse connections Parameter sharing

552 views • 27 slides
Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi Convolutional Neural Networks (CNNs) Fully connected (dense) layers have no awareness of spatial information Key concept behind convolutional layers is

983 views • 41 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly used for what sort of task? a) Recognizing images b) Transcribing speech c) Translating documents d) Playing games Neural network review

431 views • 13 slides
Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs

Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs

Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs Convolutions Pooling, Padding and Stride Convolutional Neural Networks (LeNet) Deep ConvNets (AlexNet) Networks using Blocks (VGG)

1.91k views • 89 slides
Convolution matrix reversed impulse response impulse response boundaries? 8 Feedback LSI

Convolution matrix reversed impulse response impulse response boundaries? 8 Feedback LSI

1 Mathematical Tools for Neural and Cognitive Science Fall semester, 2018 Section 3: Linear Shift-invariant Systems 2 Linear shift-invariant (LSI) systems Linearity (previously discussed): linear combination in, linear combination

274 views • 16 slides
MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks

MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks

Run 3493 Event 41075, Oct. 23rd, 2015 CONVOLUTIONAL NEURAL NETWORKS IN MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks (CNNs) are a type of deep, feed-forward neural networks that have been

831 views • 43 slides
Semantic Segmentation of the sekleton in bone scintigraphy images with convolutional neural

Semantic Segmentation of the sekleton in bone scintigraphy images with convolutional neural

Semantic Segmentation of the sekleton in bone scintigraphy images with convolutional neural networks Problem Description Convolutional Neural Networks Convolutional Layer Max Pooling Transforming Classification networks to segmentation

157 views • 12 slides
Making the Shift K-9 Communicating Student Learning Why Shift? Click to add text Where are

Making the Shift K-9 Communicating Student Learning Why Shift? Click to add text Where are

Making the Shift K-9 Communicating Student Learning Why Shift? Click to add text Where are they going to next? Jennifer % Teresa % 55/60 = 92% 23/60 = 38% Formative assessment pretest 24/25 96% 16/25 64% Oral presentation

700 views • 41 slides
Convolutional Neural Networks Convolutional neural networks One of the major kinds of ANNs in use

Convolutional Neural Networks Convolutional neural networks One of the major kinds of ANNs in use

Convolutional Neural Networks<br/><br/> 5/4/19, 4(03 PM Convolutional Neural Networks<br/><br/> 5/4/19, 4(03 PM Convolutional Neural Networks Convolutional neural networks One of the major kinds of ANNs in use UMaine

412 views • 9 slides
Shift- and Transform-Invariant Representations Denoising Speech Signals Class 18. 22 Oct 2009

Shift- and Transform-Invariant Representations Denoising Speech Signals Class 18. 22 Oct 2009

11-755 Machine Learning for Signal Processing Shift- and Transform-Invariant Representations Denoising Speech Signals Class 18. 22 Oct 2009 Summary So Far PLCA: The basic mixture-multinomial model for audio (and other data) Sparse

819 views • 65 slides
Sparse Overcomplete, Shift- and Transform-Invariant Representations Class 15. 14 Oct 2009

Sparse Overcomplete, Shift- and Transform-Invariant Representations Class 15. 14 Oct 2009

11-755 Machine Learning for Signal Processing Sparse Overcomplete, Shift- and Transform-Invariant Representations Class 15. 14 Oct 2009 Recap: Mixture-multinomial model n The basic model: Each frame in the magnitude spectrogram is a histogram

953 views • 79 slides
DENSELY CONNECTED CONVOLUTIONAL NETWORKS Gao Huang*, Zhuang Liu*, Laurens van der Maaten, Kilian

DENSELY CONNECTED CONVOLUTIONAL NETWORKS Gao Huang*, Zhuang Liu*, Laurens van der Maaten, Kilian

Best paper award DENSELY CONNECTED CONVOLUTIONAL NETWORKS Gao Huang*, Zhuang Liu*, Laurens van der Maaten, Kilian Q. Weinberger Cornell University Tsinghua University Facebook AI Research CVPR 2017 CONVOLUTIONAL NETWORKS LeNet AlexNet VGG

627 views • 23 slides

More recommend