xdn towards efficient inference of residual neural
play

XDN: Towards Efficient Inference of Residual Neural Networks on - PowerPoint PPT Presentation

Jun 08, 2023 •758 likes •934 views

(XiaoDianNao) XDN: Towards Efficient Inference of Residual Neural Networks on Cambricon Chips 2019 BenchCouncil International Artificial Intelligence System Challenges Guangli Li , Xueying Wang, Xiu Ma Advisor: Prof. Xiaobing Feng

  1. (XiaoDianNao) XDN: Towards Efficient Inference of Residual Neural Networks on Cambricon Chips — 2019 BenchCouncil International Artificial Intelligence System Challenges — Guangli Li , Xueying Wang, Xiu Ma Advisor: Prof. Xiaobing Feng State Key Laboratory of Computer Architecture, Institute of Computing Technology, Chinese Academy of Sciences

  2. XiaoDianNao (XDN) International Artificial Intelligence System Challenges XiaoDianNao Team Team Members: Advisor: Guangli Li is a Ph.D. student at ICT, CAS. His research interests include Programming Systems and Deep Learning. Xueying Wang is a Ph.D. student at ICT, CAS. Her research interests include Parallel Programming and Deep Learning. Prof. Xiaobing Feng Xiu Ma is a Ph.D. student at Jilin Institute of Computing Technology, University and a visiting student at Chinese Academy of Sciences ICT, CAS. Her research interests include Programming Language and Complier.

  3. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Track 2 International AI System Challenge based on Cambricon Chip 18 Sept. ~ 14 Oct. (about 4 weeks) Subject ： The implementation and optimization of convolutional neural network based image classification task on Cambricon AI Chips (MLUs) . Metrics ： Maximize the execution performance and minimize the prediction time (wall clock time) on provided test data. Artificial Intelligence Provided model: ResNet-50 (ACC: 84.39%) Chips (MLUs) Dataset: CIFAR-10 Provided Development Tools: Cambricon Caffe Experiment Platform: BenchCouncil Testbed

  4. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Contents I. Optimization Methodology II. Implementation of the XDN Engine III. Experimental Results IV. Conclusion and Future Work

  5. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Methodology input input input building block 1 conv conv+ conv 2 conv conv conv conv+ conv conv bn 3 bn scale sum scale sum ReLU ReLU input input input building block conv conv conv 1 pre-fusion of layers conv conv conv 2 fusion-guided pruning sum sum sum 3 post-fusion of layers • Aggressive Fusion Strategy • Fusion-Guided Pruning Method • Traditional Fusion Strategy

  6. XiaoDianNao (XDN) International Artificial Intelligence System Challenges XDN (XiaoDianNao) ——An Efficient Optimization and Inference Engine I III IV II V I. Pruning Optimizer and Trainer (aggressive fusion and pruning) II. Fusion Optimizer (traditional fusion) III. Auto-Tuner IV. Data Preprocessor and Executor V. Evaluator

  7. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Implementation of XDN Pruning and Fusion • Caffe Cambricon Prototxt Pruning Fusion Caffe Optimizer Optimizer Caffe (Trainer) Model Original Model

  8. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Implementation of XDN Pruning and Fusion • Caffe Cambricon Prototxt Pruning Fusion Caffe Optimizer Optimizer (Trainer) Caffe Model Optimized Model

  9. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Implementation of XDN Data Preprocessing • OpenCV Inference Preprocess CIFAR-10 Images Original Executer: On-line Data Preprocess

  10. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Implementation of XDN Data Preprocessing • Very Fast mmap() OpenCV Memory- mapped Inference Off-line Binary File Preprocess CIFAR-10 Images Off-line Data Preprocess & Efficient Memory-mapped File

  11. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Implementation of XDN Auto-Tuning of Hyper-Parameters: A Grid Search Approach • thread Executor Search Space data parallel Score Function Model parallel Evaluator Auto-Tuner … Finding a best option of hyper-parameters

  12. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Final Results p u d Total Time Optimization Accuracy e (ms) e 2263 0.8439 Original Baseline p 505 0.8439 OPT-A-1 S Filter 449 0.8462 OPT-B-1 x Pruning 4 428 0.8191 OPT-C-1 4 485 0.8441 OPT-A-2 . Pruning 7 + OPT-B-2 304 0.8455 Conv&BN Fusion 297 0.8203 OPT-C-2 Performance of ResNet-50 with XDN Engine Best Result: OPT-B-2, TIME=304ms, ACC=84.55%

  13. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Performance Analysis Optimizations of XDN Total Time (ms) Pruning Fusion Data Preprocess Auto-Tuning 2263 1291 833 746 304 Performance of OPT-B-2 Each part contributes the final performance.

  14. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Artifact Description BenchHub: http://125.39.136.212:8090/liguangli/bench19_xiaodiannao Detailed XDN Documentations (English and Chinese) • The Codes of Reproducible Experiments • Step-by-Step

  15. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Conclusion and Future Work Highlight Contributions: 1. We proposed a optimization methodology Characteristics of Cambricon Chips • Fusion-Guided Pruning Method • 2. We implemented the Efficient XDN engine Pruner, Trainer, Optimizer, Auto-Tuner, Executer… • 3. We evaluated the performance Achieves high speedup (7.44x) without accuracy loss • In the future, we plan to: test the method on large-scale datasets, such as ImageNet; • extend the XDN engine to support more DNN models; • test the method on other AI chips; • … •

  16. XDN: Towards Efficient Inference of Residual Neural Networks on Cambricon Chips Thank You liguangli@ict.ac.cn

Recommend

Efficient Layout Hotspot Detection via Binarized Residual Neural Network Yiyang Jiang 1 , Fan Yang

Efficient Layout Hotspot Detection via Binarized Residual Neural Network Yiyang Jiang 1 , Fan Yang

Efficient Layout Hotspot Detection via Binarized Residual Neural Network Yiyang Jiang 1 , Fan Yang 1 , Hengliang Zhu 1 , Bei Yu 3 , Dian Zhou 2 , Xuan Zeng 1 1 State Key Lab of ASIC & System, Microelectronics Department, Fudan

357 views • 21 slides
Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference Engine Song Han CVA group, Stanford University Apr 7, 2016 1 Intro about me and my advisor Fourth year PhD with Prof. Bill Dally at Stanford.

2.06k views • 70 slides
Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference Engine Song Han CVA group, Stanford University Jan 6, 2015 A few words about us Fourth year PhD with Prof. Bill Dally at Stanford.

966 views • 54 slides
Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab,

Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab,

Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab, Northeastern University, Boston, US Research summary Deep convolutional neural networks for Image Restoration 1.Residual dense network. [ CVPR-2018

375 views • 33 slides
Clinical Inference in the Assessment of Mental Residual Functional Capacity David J Schretlen,

Clinical Inference in the Assessment of Mental Residual Functional Capacity David J Schretlen,

Clinical Inference in the Assessment of Mental Residual Functional Capacity David J Schretlen, PhD, ABPP OIDAP Panel Meeting 10 June 2009 1 Methods of Inference 1. Pathognomonic sign approach 2. Pattern analysis 3. Level of performance or

761 views • 44 slides
Pipeline Strategies and conversations behind securing a Residual Bequest Agenda 1. Why Residual?

Pipeline Strategies and conversations behind securing a Residual Bequest Agenda 1. Why Residual?

The Perpetual Pipeline Strategies and conversations behind securing a Residual Bequest Agenda 1. Why Residual? 2. Structured for Residual 3. Residual Results 4. Residual Strategy 5. Residual Relationships 6. Residual Data 7.

1.17k views • 43 slides
Same, Same But Different Recovering Neural Network Quantization Error Through Weight

Same, Same But Different Recovering Neural Network Quantization Error Through Weight

Same, Same But Different Recovering Neural Network Quantization Error Through Weight Factorization Eldad Meller ICML 2019 Neural Network Quantization Quantization of Neural Networks is needed for efficient inference Quantization adds

482 views • 9 slides
Inference for the difference of two percentile residual life functions Alba M. Franco-Pereira

Inference for the difference of two percentile residual life functions Alba M. Franco-Pereira

Introduction The problem A simulation study A real data example References Inference for the difference of two percentile residual life functions Alba M. Franco-Pereira Department of Statistics Universidad Carlos III de Madrid August, 2010

507 views • 39 slides
GPU INFERENCE ENGINE Michael Andersch, 7 th April 2016 WHAT IS INFERENCE, ANYWAYS? Building a

GPU INFERENCE ENGINE Michael Andersch, 7 th April 2016 WHAT IS INFERENCE, ANYWAYS? Building a

April 4-7, 2016 | Silicon Valley NVIDIA GIE: HIGH-PERFORMANCE GPU INFERENCE ENGINE Michael Andersch, 7 th April 2016 WHAT IS INFERENCE, ANYWAYS? Building a deep neural network based application Step 1: Use data to train the neural network -

1.02k views • 17 slides
Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June

Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June

Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June 13, 2019 Neural language generation Cat sat on ? Probability distribution at time t given context vector h t R d , weights W R d |V|

474 views • 14 slides
Algorithms in Nature Pruning in neural networks Neural network development 1. Efficient signal

Algorithms in Nature Pruning in neural networks Neural network development 1. Efficient signal

Algorithms in Nature Pruning in neural networks Neural network development 1. Efficient signal propagation [e.g. information processing & integration] 2. Robust to noise and failures [e.g. cell or synapse failure] 3. Cost-aware design [e.g.

874 views • 33 slides
GENERIC CUTS: AN EFFICIENT ALGORITHM FOR OPTIMAL INFERENCE IN HIGHER ORDER MRF-MAP Chetan Arora,

GENERIC CUTS: AN EFFICIENT ALGORITHM FOR OPTIMAL INFERENCE IN HIGHER ORDER MRF-MAP Chetan Arora,

GENERIC CUTS: AN EFFICIENT ALGORITHM FOR OPTIMAL INFERENCE IN HIGHER ORDER MRF-MAP Chetan Arora, Subhashis Banerjee, Prem Kalra, S.N. Maheshwari Indian Institute of Technology Delhi, INDIA MRF-MAP INFERENCE MRF-MAP INFERENCE HIGHER

374 views • 19 slides
R RITSUMEIKAN Introduction 1 Methodology Content 2 Experiments 3 Conclusion 4

R RITSUMEIKAN Introduction 1 Methodology Content 2 Experiments 3 Conclusion 4

ICIP 2019 Residual Convolutional Neural Networks with Global and Local Pathways for Classification of Focal Liver A dual attention dilated residual network for liver lesion classification and localization on CT images Lesions Xiao Chen,

779 views • 19 slides
Redundancy-free Residual Dispatch Using Ordered Binary Decision Diagrams for Efficient Dispatch

Redundancy-free Residual Dispatch Using Ordered Binary Decision Diagrams for Efficient Dispatch

Formula-based Dispatch BDD-based Dispatch Assessment Redundancy-free Residual Dispatch Using Ordered Binary Decision Diagrams for Efficient Dispatch Andreas Sewe Christoph Bockisch Mira Mezini Seventh International Workshop on Foundations

408 views • 39 slides
Fast Inference for Neural Vocoders Welcome, everyone! Im excited to be here today and get the

Fast Inference for Neural Vocoders Welcome, everyone! Im excited to be here today and get the

Fast Inference for Neural Vocoders Welcome, everyone! Im excited to be here today and get the opportunity to tell you a little bit about a few incredibly interesting systems challenges my team has encountered recently. Id like to preface

595 views • 44 slides
4-connected shift residual networks ICCV 2019 Neural Architects Workshop Andrew Brown, Pascal

4-connected shift residual networks ICCV 2019 Neural Architects Workshop Andrew Brown, Pascal

4-connected shift residual networks ICCV 2019 Neural Architects Workshop Andrew Brown, Pascal Mettes, Marcel Worring, University of Amsterdam Network costs increasing! Increasing accuracy on ImageNet has come at increasing cost Popular

280 views • 15 slides
Neural Variational Inference and Learning Andriy Mnih, Karol Gregor 22 June 2014 1 / 14

Neural Variational Inference and Learning Andriy Mnih, Karol Gregor 22 June 2014 1 / 14

Neural Variational Inference and Learning Andriy Mnih, Karol Gregor 22 June 2014 1 / 14 Introduction Training directed latent variable models is difficult because inference in them is intractable. Both MCMC and traditional variational

322 views • 14 slides
Bayesian Methods for Neural Networks Readings: Bishop, Neural Networks for Pattern Recognition .

Bayesian Methods for Neural Networks Readings: Bishop, Neural Networks for Pattern Recognition .

Bayesian Methods for Neural Networks Readings: Bishop, Neural Networks for Pattern Recognition . Chapter 10. Aaron Courville Bayesian Methods for Neural Networks p.1/29 Bayesian Inference Weve seen Bayesian inference before, remember

542 views • 29 slides
Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What well cover ... f ( x ) computer vision architectures - convolutional networks - data augmentation 1 - residual networks ... ...

1.29k views • 75 slides
An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning Microsoft Research Asia (MSRA) Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. Deep Residual Learning for Image Recognition. arXiv

1.32k views • 54 slides
Computationally efficient probabilistic inference with noisy threshold models based on a CP

Computationally efficient probabilistic inference with noisy threshold models based on a CP

Computationally efficient probabilistic inference with noisy threshold models based on a CP tensor decomposition Jirka Vomlel and Petr Tichavsk y Institute of Information Theory and Automation ( UTIA) Academy of Sciences of the Czech

1.06k views • 53 slides
Invertible Residual Networks Jens Behrmann * Will Grathwohl* Ricky T. Q. Chen David Duvenaud

Invertible Residual Networks Jens Behrmann * Will Grathwohl* Ricky T. Q. Chen David Duvenaud

Invertible Residual Networks Jens Behrmann * Will Grathwohl* Ricky T. Q. Chen David Duvenaud Jrn-Henrik Jacobsen* (*equal contribution) What are Invertible Neural Networks? Non-invertible Invertible Invertible Neural Networks (INNs) are

330 views • 30 slides
Efficient Probabilistic Inference in the Quest for Physics Beyond the Standard Model Atlm

Efficient Probabilistic Inference in the Quest for Physics Beyond the Standard Model Atlm

Efficient Probabilistic Inference in the Quest for Physics Beyond the Standard Model Atlm Gne Baydin , Lukas Heinrich, Wahid Bhimji, Lei Shao, Saeid Naderiparizi, Andreas Munk, Jialin Liu, Bradley Gram-Hansen, Gilles Louppe, Lawrence

991 views • 67 slides
Efficient Inference for Mul0nomial Mixed Membership Models David Mimno

Efficient Inference for Mul0nomial Mixed Membership Models David Mimno

Efficient Inference for Mul0nomial Mixed Membership Models David Mimno UMass, Amherst Summary How fast can we make Gibbs sampling? How liGle memory can we use? Goal:

668 views • 62 slides

More recommend