ultra low bit neural network quantization
play

Ultra-low-bit Neural Network Quantization Peisong Wang Institute of - PowerPoint PPT Presentation

Feb 03, 2023 •2.44k likes •2.83k views

Ultra-low-bit Neural Network Quantization Peisong Wang Institute of Automation, Chinese Academy of Sciences 2020.06.03 Collaborators: Weixiang Xu Tianli Zhao Fanrong Li Xiangyu He Gang Li Jian Cheng Cong Leng 6/5/20 1

  1. Ultra-low-bit Neural Network Quantization Peisong Wang Institute of Automation, Chinese Academy of Sciences 2020.06.03 Collaborators: Weixiang Xu Tianli Zhao Fanrong Li Xiangyu He Gang Li Jian Cheng Cong Leng 6/5/20 1 peisong.wang@nlpr.ia.ac.cn

  2. Background: Deep Learning From: Russ Salakhutdinov 6/5/20 peisong.wang@nlpr.ia.ac.cn 2

  3. Background: Application of CNNs • 01 Classification Detection Segmentation Convolutional Neural Networks 6/5/20 peisong.wang@nlpr.ia.ac.cn 3

  4. Background: Training Train ResNet50 from several days to: • Facebook: 1 hour • Fast.ai: 18 min • Tencent: 6.6 min • Sony: 3.7 min • Google: 2.2 min • SenseTime: 1.5 min 6/5/20 peisong.wang@nlpr.ia.ac.cn 4

  5. Background: Real World Applications Ø Low inference speed Ø Large memory/storage AR/VR Ø High power consumption Self-Driving Intelligent Surveillance Car Intelligent Robot Face Unlock 6/5/20 peisong.wang@nlpr.ia.ac.cn 5

  6. Network Acceleration and Compression • Low-rank Decomposition • Sparse/Pruning • Quantization • Knowledge Distillation • …… 6/5/20 peisong.wang@nlpr.ia.ac.cn 6

  7. Fixed-point representation 𝑻 : sign M : Mantissa 𝑭 : Exponent 1 8 23 −𝟐 𝑻 ×1.M× 𝟑 𝑭 FP32 S E M 1 7 Int8 S M −𝟐 𝑻 ×M 1 3 Int4 S M 6/5/20 peisong.wang@nlpr.ia.ac.cn 7

  8. Why Fixed-point quantization? • Saving memory • Saving energy • Saving time • Saving area Mark Horowitz , Computing’s Energy Problem . ISSCC 2014. 6/5/20 peisong.wang@nlpr.ia.ac.cn 8

  9. Type of quantization 2 # 𝑤𝑏𝑚𝑣𝑓𝑡 : 000…000 ~ 111…111 𝑂 − bit Non-uniform Uniform Logarithmic 𝐷₋0 0 0 0…000 Non-uniform Quantization 𝐷₋1 1 1 0…001 𝐷 ₋ 2 2 2 0…010 Uniform Quantization 𝐷 ₋ 3 3 4 0…011 𝐷 ₋ 4 4 8 0…100 𝐷 ₋ 5 5 16 0…101 Logarithmic Quantization 𝐷 ₋ 6 6 32 0…110 . . . . . . . . . . . . Scalar quantization with/without constrains 𝐷₋(2 ! − 1) 2 ! − 1 2 " ! #" 1…111 6/5/20 peisong.wang@nlpr.ia.ac.cn 9

  10. Contents • Sparsity-inducing Binarized Neural Networks. AAAI, 2020. • Soft Threshold Ternary Networks. IJCAI, 2020. • Hardware Acceleration of CNN with One-Hot Quantization of Weights and Activations. DATE 2020. • Towards Accurate Post-training Network Quantization via Bit-Split and Stitching. ICML, 2020 6/5/20 peisong.wang@nlpr.ia.ac.cn 10

  11. Contents • Sparsity-inducing Binarized Neural Networks. AAAI, 2020. • Soft Threshold Ternary Networks. IJCAI, 2020. • Hardware Acceleration of CNN with One-Hot Quantization of Weights and Activations. DATE 2020. • Towards Accurate Post-training Network Quantization via Bit-Split and Stitching. ICML, 2020 6/5/20 peisong.wang@nlpr.ia.ac.cn 11

  12. Binary: Sparsity-inducing BNN Previous binary approach: -1 1 -1 1 -1 1 -1 1 Binary == -1/+1 -1 1 Binary = Two States ( 𝒃 𝟏 𝒃 𝟐 ) 0 0 0 1 -1 1 Which two states to use ? Peisong Wang, Xiangyu He, Gang Li, Tianli Zhao and Jian Cheng, “Sparsity-inducing Binarized Neural Networks”, AAAI, 2020. 6/5/20 peisong.wang@nlpr.ia.ac.cn 12

  13. Sparsity-inducing BNN How to accelerate BNN with 0/1 activations ? (-1, +1) Reparameterization with affine transformation ( 𝒃 𝟏 𝒃 𝟐 ) 6/5/20 peisong.wang@nlpr.ia.ac.cn 13

  14. Sparsity-inducing BNN How to determine the threshold of 0/1 binarization? v Binarization at zero-point v Normal distribution v Large quantization error v Binarization at 𝜄 v The mutual information 𝐽(𝑦; C 𝑧) of two discrete random variables x and C 𝑧 can be defined as He Z , Fan D . Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network using Truncated Gaussian Approximation. CVPR 2019. 6/5/20 peisong.wang@nlpr.ia.ac.cn 14

  15. Sparsity-inducing BNN How to determine the threshold of 0/1 binarization? Mutual Information can be formulated as the function of 𝑞 𝑦 = 0 = 𝑞 Ablation study on the selection of threshold on AlexNet 6/5/20 peisong.wang@nlpr.ia.ac.cn 15

  16. Sparsity-inducing BNN AlexNet Experiments: v Extend our methods to other network structures v Without bells and whistles Comparison with 2-bit method ResNet-18 6/5/20 peisong.wang@nlpr.ia.ac.cn 16

  17. Sparsity-inducing BNN Run-time speedup: Tianli Zhao, Xiangyu He, Jian Cheng. BitStream: Efficient Computing Architecture for Real-Time Low-Power Inference of Binary Neural Networks on CPUs. ACM MM 2018. 6/5/20 peisong.wang@nlpr.ia.ac.cn 17

  18. Contents • Sparsity-inducing Binarized Neural Networks. AAAI, 2020. • Soft Threshold Ternary Networks. IJCAI, 2020. • Hardware Acceleration of CNN with One-Hot Quantization of Weights and Activations. DATE 2020. • Towards Accurate Post-training Network Quantization via Bit-Split and Stitching. ICML, 2020 6/5/20 peisong.wang@nlpr.ia.ac.cn 18

  19. Soft Threshold Ternary Networks Previous ternary problem: From hard to soft threshold: Binary + Binary = Ternary −1 0 +1 −Δ Δ Weixiang Xu, Xiangyu He, Tianli Zhao, Qinghao Hu, Peisong Wang and Jian Cheng. “Soft Threshold Ternary Networks”, IJCAI, 2020. 6/5/20 peisong.wang@nlpr.ia.ac.cn 19

  20. Soft Threshold Ternary Networks — Ternarize both weights and activations — Without constraint of ∆ — Soft threshold 6/5/20 peisong.wang@nlpr.ia.ac.cn 20

  21. Soft Threshold Ternary Networks ImageNet Results: 6/5/20 peisong.wang@nlpr.ia.ac.cn 21

  22. Contents • Sparsity-inducing Binarized Neural Networks. AAAI, 2020. • Soft Threshold Ternary Networks. IJCAI, 2020. • Hardware Acceleration of CNN with One-Hot Quantization of Weights and Activations. DATE 2020. • Towards Accurate Post-training Network Quantization via Bit-Split and Stitching. ICML, 2020 6/5/20 peisong.wang@nlpr.ia.ac.cn 22

  23. One-hot Networks To obtain more efficient quantizer: 8-bit, both activations and weights -128~127 INT-8 S 7-hot -128~127 S 6-hot INT-7 -64~63 S S -127~126 INT-6 S -32~31 Bit-width Non-zeros … INT-5 S -16~15 -96~96 Two-hot S INT-4 S -8~7 INT-3 S One-hot S -64~64 -4~3 Gang Li, Peisong Wang, Zejian Liu, Cong Leng, Jian Cheng. Hardware Acceleration of CNN with One-Hot Quantization of Weights and Activations. DATE 2020 6/5/20 peisong.wang@nlpr.ia.ac.cn 23

  24. One-hot Networks One-hot weight (logarithmic) One-hot weight + One-hot Activation § only one non-zero bit in weights § only one non-zero bit in weights/activations § multiplication -> bit shift of activation § multiplication -> addition + encoding [2] Effectual bits: exponent bits + sign bit, § 8bit -> 3+1bit § [1] H. Tann, S. Hashemi, R. I. Bahar, S. Reda, “HardwareSoftware Codesign of Highly Accurate, Multiplier-free Deep Neural Networks”, DAC'17 [2] S. Sharify et al., “Laconic Deep Learning Inference Acceleration”, ISCA'19 6/5/20 peisong.wang@nlpr.ia.ac.cn 24

  25. One-hot Networks Baseline: 16/16 DaDianNao [1], 8/8 Laconic [2] Xilinx ZC706 Dev Board Vivado HLS 2018.2 [1] Y. Chen et al., “DaDianNao: A Machine-Learning Supercomputer,” MICRO'14 [2] S. Sharify et al., “Laconic Deep Learning Inference Acceleration,” ISCA'19 6/5/20 peisong.wang@nlpr.ia.ac.cn 25

  26. Contents • Sparsity-inducing Binarized Neural Networks. AAAI, 2020. • Soft Threshold Ternary Networks. IJCAI, 2020. • Hardware Acceleration of CNN with One-Hot Quantization of Weights and Activations. DATE 2020. • Towards Accurate Post-training Network Quantization via Bit-Split and Stitching. ICML, 2020 6/5/20 peisong.wang@nlpr.ia.ac.cn 26

  27. Bit-Split for Post-training Network Quantization Training-aware quantization Post-training Quantization Pre-trained Model Pre-trained Data-free Model BP-free Network Quantization Hyper-parameter free Easy to use Network Quantization Finetune using data/labels Peisong Wang, Qiang Chen, Xiangyu He, Jian Cheng. Towards Accurate Post-training Network Quantization via Bit-Split and Stitching. ICML2020 6/5/20 peisong.wang@nlpr.ia.ac.cn 27

  28. Bit-Split for Post-training Network Quantization Post-training quantization Min-Max Min-Max with clip Minimize the Di Distance Problem: Szymon Migacz. 8-bit Inference with TensorRT. GTC 2017 6/5/20 peisong.wang@nlpr.ia.ac.cn 28

  29. Bit-Split for Post-training Network Quantization Problem: Optimization: 6/5/20 peisong.wang@nlpr.ia.ac.cn 29

  30. Bit-Split for Post-training Network Quantization Weight Quantization: Weight and Activation Quantization: 6/5/20 peisong.wang@nlpr.ia.ac.cn 30

Recommend

Same, Same But Different Recovering Neural Network Quantization Error Through Weight

Same, Same But Different Recovering Neural Network Quantization Error Through Weight

Same, Same But Different Recovering Neural Network Quantization Error Through Weight Factorization Eldad Meller ICML 2019 Neural Network Quantization Quantization of Neural Networks is needed for efficient inference Quantization adds

482 views • 9 slides
Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural IR tasks Neural IR architecture Feature Representations Neural IR query auto completion Neural IR query suggestion Neural IR document

1.48k views • 18 slides
Learning Accurate Low-bit Deep Neural Networks with Stochastic Quantization Yinpeng Dong 1 ,

Learning Accurate Low-bit Deep Neural Networks with Stochastic Quantization Yinpeng Dong 1 ,

Learning Accurate Low-bit Deep Neural Networks with Stochastic Quantization Yinpeng Dong 1 , Renkun Ni 2 , Jianguo Li 3 , Yurong Chen 3 , Jun Zhu 1 , Hang Su 1 1 Department of CST, Tsinghua University 2 University of Virginia 3 Intel Labs China

320 views • 14 slides
Quantization for TVM Ziheng Jiang TVM Conference, Dec 12th 2018 Quantization for TVM What is

Quantization for TVM Ziheng Jiang TVM Conference, Dec 12th 2018 Quantization for TVM What is

Quantization for TVM Ziheng Jiang TVM Conference, Dec 12th 2018 Quantization for TVM What is Quantization? source: Han et al Converting weight value to low-bit integer like 8bit precision from float-point without significant accuracy drop.

421 views • 7 slides
Convolutional neural network for centrality in fixed target experiments Denis Uzhva 6 june 2019

Convolutional neural network for centrality in fixed target experiments Denis Uzhva 6 june 2019

Convolutional neural network for centrality in fixed target experiments Denis Uzhva 6 june 2019 Saint Petersburg University Laboratory of Ultra Hight Energy Physics WPCF2019 Table of contents 1. Introduction 2. Machine learning in HEP 3.

767 views • 52 slides
Quantization, after Souriau Prequantization Quantization? Group algebra Classical Franois

Quantization, after Souriau Prequantization Quantization? Group algebra Classical Franois

Quantization, after Souriau Souriau Quantization, after Souriau Prequantization Quantization? Group algebra Classical Franois Ziegler (Georgia Southern) Quantum Nilpotent Reductive Geometric Quantization: Old and New E(3) 2019 CMS

1.51k views • 123 slides
Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network Perceptron Feed forward Backpropagation Learning rate/momentum What is a neural network? Artificial Human Neurons Each

888 views • 12 slides
Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics Activation Functions Stochastic Gradient Descent (SGD) Regularization (Dropout) Training Tips and Tricks Neural Network (NN)

828 views • 45 slides
Graph Neural Network Fang Yuanqiang, 2019/05/18 Graph Neural Network Why GNN? Preliminary

Graph Neural Network Fang Yuanqiang, 2019/05/18 Graph Neural Network Why GNN? Preliminary

Graph Neural Network Fang Yuanqiang, 2019/05/18 Graph Neural Network Why GNN? Preliminary Fixed graph Vanilla Spectral Graph ConvNets ChebyNet GCN CayleyNet Multiple graphs Variable graph

817 views • 46 slides
Neural network applications ALVINN (Pomerleau, mid 1990s) Autonomous Land Vehicle in Neural

Neural network applications ALVINN (Pomerleau, mid 1990s) Autonomous Land Vehicle in Neural

Neural network applications ALVINN (Pomerleau, mid 1990s) Autonomous Land Vehicle in Neural Network To date: Sharp Straight Sharp Left Ahead Right Neural networks: what are they 30 Output Units Backpropagation: efficient

346 views • 15 slides
Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are the commonest type of neural

534 views • 32 slides
Towards Accurate Post-training Network Quantization via Bit-split and Stitching Peisong Wang ,

Towards Accurate Post-training Network Quantization via Bit-split and Stitching Peisong Wang ,

Towards Accurate Post-training Network Quantization via Bit-split and Stitching Peisong Wang , Qiang Chen, Xiangyu He, Jian Cheng Institute of Automation, Chinese Academy of Sciences 1 Outline Background Motivation Approach

188 views • 18 slides
Deep Quantization Network for Efficient Image Retrieval . . . Yue Cao, Mingsheng Long, Jianmin

Deep Quantization Network for Efficient Image Retrieval . . . Yue Cao, Mingsheng Long, Jianmin

. Deep Quantization Network for Efficient Image Retrieval . . . Yue Cao, Mingsheng Long, Jianmin Wang, Han Zhu, and Qingfu Wen School of Software Tsinghua University The Thirtieth AAAI Conference on Artificial Intelligence, 2016 . . . .

448 views • 18 slides
Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi,

Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi,

Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geofgrey Hinton Motivation We need tools to understand trained neural networks Neural network training involves interactions between an

588 views • 16 slides
Lecture 2: Deeper Neural Network 1 Objective In the second lecture, you will see How to

Lecture 2: Deeper Neural Network 1 Objective In the second lecture, you will see How to

Lecture 2: Deeper Neural Network 1 Objective In the second lecture, you will see How to deepen your neural network to learn more complex functions Several techniques to fasten your learning process The most popular artificial

423 views • 40 slides
Neural network architectures for image captioning By Emily Kern Given a set of images and

Neural network architectures for image captioning By Emily Kern Given a set of images and

Neural network architectures for image captioning By Emily Kern Given a set of images and accompanying human-generated captions, can we train a neural network to predict captions for new images? What is a neural network? A computer system

523 views • 16 slides
Unsupervised clustering with growing self-organizing neural network A comparison with non-neural

Unsupervised clustering with growing self-organizing neural network A comparison with non-neural

Unsupervised clustering with growing self-organizing neural network A comparison with non-neural approach Martin Hynar, Michal Burda, Jana Sarmanov a Department of Computer Science, V SB Technical University of Ostrava, 17.

473 views • 24 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
A Neural Network Architecture for Detec2ng Gramma2cal Errors in SMT A Neural Network Architecture

A Neural Network Architecture for Detec2ng Gramma2cal Errors in SMT A Neural Network Architecture

A Neural Network Architecture for Detec2ng Gramma2cal Errors in SMT A Neural Network Architecture for Detec2ng Gramma2cal Errors in SMT Morpho-Syntac-c features outperform word embeddings on this task Syntac-c n-grams improve the

1.18k views • 38 slides
LOW PRECISION INFERENCE ON GPU Hao Wu, NVIDIA OUTLINE Performance motivation for quantization

LOW PRECISION INFERENCE ON GPU Hao Wu, NVIDIA OUTLINE Performance motivation for quantization

LOW PRECISION INFERENCE ON GPU Hao Wu, NVIDIA OUTLINE Performance motivation for quantization Quantization details Post-training quantization accuracy Training for quantization 2 INFERENCE (sometimes called serving)

1.13k views • 60 slides
Human-Robot Interaction: Language Acquisition with Neural Network Alvin Rindra Fazrie 09.11.2015

Human-Robot Interaction: Language Acquisition with Neural Network Alvin Rindra Fazrie 09.11.2015

Human-Robot Interaction: Language Acquisition with Neural Network Alvin Rindra Fazrie 09.11.2015 1 Outline Motivation Basics and Definitions - Natural Language Processing - Neural Network Neural Network Architecture - Single

242 views • 12 slides
Neural Network II Neural Network II Week 8 1 Team Homework Assignment #10 Team Homework

Neural Network II Neural Network II Week 8 1 Team Homework Assignment #10 Team Homework

Neural Network II Neural Network II Week 8 1 Team Homework Assignment #10 Team Homework Assignment #10 Read pp. 327 334. Read pp. 327 334. Do Example 6.9. Explore neural network tools and try to use a tool for solving Example

628 views • 47 slides
using Monte Carlo Methods EMC2 Workshop Gonalo Mordido Hasso Plattner Institute @ NeurIPS

using Monte Carlo Methods EMC2 Workshop Gonalo Mordido Hasso Plattner Institute @ NeurIPS

Instant Quantization of Neural Networks using Monte Carlo Methods EMC2 Workshop Gonalo Mordido Hasso Plattner Institute @ NeurIPS 2019 Matthijs Van Keirsbilck NVIDIA Alexander Keller NVIDIA 1 Motivation and idea neural network

174 views • 6 slides
Machine Learning for NLP The Neural Network Zoo Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP The Neural Network Zoo Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP The Neural Network Zoo Aurlie Herbelot 2019 Centre for Mind/Brain Sciences University of Trento 1 The Neural Net Zoo http://www.asimovinstitute.org/neural-network-zoo/ 2 How to keep track of new architectures?

467 views • 26 slides

More recommend