Parameter efficient training of deep convolutional neural networks - PowerPoint PPT Presentation

Mar 31, 2023 •631 likes •727 views

Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization Hesham Mostafa (Intel AI) Xin Wang (Intel AI, Cerebras Systems) Easy : post-training (sparse) compression Hard : direct training of sparse

Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization Hesham Mostafa (Intel AI) Xin Wang (Intel AI, Cerebras Systems)
Easy : post-training (sparse) compression Hard : direct training of sparse networks Compression
“Winning lottery tickets” (Frankle & Carbin 2018): post hoc identification of trainable sparse nets Compression
Dynamic sparse reparameterization (ours): training-time structural exploration
Direct training sparse nets to generalize as well as post-training compression : is this possible? - YES Directly trained sparse nets : are they “winning lottery tickets”? - NO
Dynamic sparse reparameterization prune grow 1 for each sparse parameter tensor W i do ( W i , k i ) ← prune_by_threshold ( W i , H ) ◃ k i is the number of pruned weights 2 l i ← number_of_nonzero_entries ( W i ) ◃ Number of surviving weights after pruning 3 4 end for 5 ( K, L ) ← ( � i k i , � i l i ) ◃ Total number of pruned and surviving weights 6 H ← adjust_pruning_threshold ( H, K, δ ) ◃ Adjust pruning threshold 7 for each sparse parameter tensor W i do W i ← grow_back ( W i , l i ◃ Grow l i L K ) L K zero-initialized weights at random in W i 8 9 end for
Closed gap between post-training compression and direct training of sparse nets WRN-28-2 on CIFAR10 Resnet-50 on Imagenet Global sparsity Sparsity (# Param) 0.8 (7.3M) 0.9 (5.1M) 0.0 (25.6M) 0.9 0.8 0.7 0.6 0.5 95 72.4 90.9 70.7 89.9 Thin dense [-2.5] [-1.5] [-4.2] [-2.5] 71.6 90.4 67.8 88.4 Test accuracy% 94 Static sparse [-3.3] [-2.0] [-7.1] [-4.0] 71.7 90.6 70.2 90.0 DeepR 93 (Bellec et al., 2017) [-3.2] [-1.8] [-4.7] [-2.4] 74.9 92.4 72.6 91.2 70.4 90.1 SET [0.0] [0.0] (Mocanu et al., 2018) [-2.3] [-1.2] [-4.5] [-2.3] Full dense 92 Dynamic sparse 73.3 92.4 71.6 90.5 Compressed sparse (Ours) [ -1.6 ] [ 0.0 ] [ -3.3 ] [ -1.9 ] Thin dense Static sparse DeepR SET Dynamic sparse 73.2 91.5 70.3 90.0 Compressed sparse (Zhu & Gupta, 2017) [-1.7] [-0.9] [-4.6] [-2.4] 161 306 451 596 741 Number of parameters (K)
Directly trained sparse nets are not “winning tickets”: exploration of structural degrees of freedom is crucial
Visit our poster :   Wednesday, Pacific Ballroom #248

Recommend

Parameter Hub A Rack-Scale Parameter Server for Efficient Cloud-based Distributed Deep Neural

Parameter Hub A Rack-Scale Parameter Server for Efficient Cloud-based Distributed Deep Neural Network Training Liang Luo, Jacob Nelson, Luis Ceze, Amar Phanishayee and Arvind Krishnamurthy 1 - DNN training is computationally expensive - Needs

565 views • 44 slides

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan, Laxmikant Kishor Mokadam, Xipeng Shen, Seung-Hwan Lim, Robert Patton 1 Build an image classifier? Deep Neural Network (DNN) CPU GPU Training

957 views • 41 slides

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks Wei-Lin Chiang 1 , Xuanqing Liu 2 , Si Si 3 , Yang Li 3 , Samy Bengio 3 , Cho-Jui Hsieh 23 1 National Taiwan University, 2 UCLA, 3 Google Research Graph

616 views • 23 slides

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12 Convolutional Networks Scale up neural networks to process very large images / video sequences Sparse connections Parameter sharing

552 views • 27 slides

11/21/2018 ImageNet Classification with Deep Convolutional Neural Networks Prepared by Faizaan

11/21/2018 ImageNet Classification with Deep Convolutional Neural Networks Prepared by Faizaan Naveed Won Mo (Andy) Jung Lassonde School of Engineering, York University Canada Objective Overview of Convolutional Neural Network

567 views • 17 slides

Augmentation Introduction ImageNet Classification with Deep Convolutional Neural Networks,

Day 2 Lecture 2 Augmentation Introduction ImageNet Classification with Deep Convolutional Neural Networks, Krizhevsky A., 2012 ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) 1.2 million training images , 50,000 validation images, and

537 views • 14 slides

6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt Kira Quiz coming up Next Monday (2/5) 30 minutes Topics: Optimization Basic neural networks Neural Network

619 views • 41 slides

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional Neural Networks

648 views • 26 slides

Lecture 13: Introduction to Deep Learning Deep Convolutional Neural Networks Aykut Erdem

Lecture 13: Introduction to Deep Learning Deep Convolutional Neural Networks Aykut Erdem November 2016 Hacettepe University Administrative Assignment 3 is out! It is due November 30, 2016 You will implement a 2-layer Neural

1.28k views • 114 slides

and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast

Band-limited Training and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast Training of Convolutional Networks through FFTs Fast Convolutional Nets With fbfft: A GPU Performance Evaluation Data: x

648 views • 46 slides

Convolutional Neural Networks Computer Vision Jia-Bin Huang, Virginia Tech Todays class

Convolutional Neural Networks Computer Vision Jia-Bin Huang, Virginia Tech Todays class Overview Convolutional Neural Network (CNN) Training CNN Understanding and Visualizing CNN Image Categorization: Training phase Training

1.82k views • 68 slides

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional

444 views • 27 slides

Deep Convolutional Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine

Deep Convolutional Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Deep Convolutional Neural Nets 1 / 25 Outline 1 Why Neural Networks? 2 Circuits 3 Neurons, Layers, and Networks 4 Correlation and Convolution 5

287 views • 25 slides

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly used for what sort of task? a) Recognizing images b) Transcribing speech c) Translating documents d) Playing games Neural network review

431 views • 13 slides

Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Transfer Learning with Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural Networks A breakthough Convolutional Neural Networks VGG-16 example Layers of Convolutional filters Bottleneck

625 views • 23 slides

MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks

Run 3493 Event 41075, Oct. 23rd, 2015 CONVOLUTIONAL NEURAL NETWORKS IN MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks (CNNs) are a type of deep, feed-forward neural networks that have been

831 views • 43 slides

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN)

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN) A.k.a. CNN or ConvNet Adit Deshpande, A Beginner's Guide To Understanding Convolutional Neural Networks. Digital Images Input array: an images

1.42k views • 72 slides

High-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks

High-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks Krzysztof J. Geras Joint work with Kyunghyun Cho , Linda Moy , Gene Kim , Stacey Wolfson and Artie Shen . GTC 2017 W HERE DEEP LEARNING IS USEFUL amount

772 views • 42 slides

Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs

Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs Convolutions Pooling, Padding and Stride Convolutional Neural Networks (LeNet) Deep ConvNets (AlexNet) Networks using Blocks (VGG)

1.91k views • 89 slides

Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab,

Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab, Northeastern University, Boston, US Research summary Deep convolutional neural networks for Image Restoration 1.Residual dense network. [ CVPR-2018

375 views • 33 slides

SnaPEA : Predictive Early Activation for Reducing Computation In Deep Convolutional Neural

SnaPEA : Predictive Early Activation for Reducing Computation In Deep Convolutional Neural Networks * * Vahideh Akhlaghi Amir Yazdanbakhsh Kambiz Samadi Rajesh K. Gupta Hadi Esmaeilzadeh * Equal Contribution University

565 views • 32 slides

Evaluations of Deep Convolutional Neural Networks for Automatic Identification of Malaria Infected

Evaluations of Deep Convolutional Neural Networks for Automatic Identification of Malaria Infected Cells Yuhang Dong, Zhuocheng Jiang, Hongda Shen, W. David Pan Dept. of Electrical & Computer Engineering, Univ. of Alabama in Huntsville (UAH)

370 views • 14 slides

Classification Accuracies of Malaria Infected Cells Using Deep Convolutional Neural Networks

Classification Accuracies of Malaria Infected Cells Using Deep Convolutional Neural Networks Based on Decompressed Images Yuhang Dong, Zhuocheng Jiang, Hongda Shen, W. David Pan Dept. of Electrical & Computer Engineering, Univ. of Alabama

660 views • 18 slides