Lightweight Neural Networks from PCA & LDA Based Distilled Dense - PowerPoint PPT Presentation

Oct 25, 2023 •184 likes •243 views

Lightweight Neural Networks from PCA & LDA Based Distilled Dense Neural Networks ICIP 2020 MEA. Seddik 1 , 2 , , H. Essafi 1 , A. Benzine 1 , 3 , M. Tamaazousti 1 1 CEA List, France 2 CentraleSuplec, L2S, France 3 Sorbonne University,

Lightweight Neural Networks from PCA & LDA Based Distilled Dense Neural Networks ICIP 2020 MEA. Seddik 1 , 2 , ∗ , H. Essafi 1 , A. Benzine 1 , 3 , M. Tamaazousti 1 1 CEA List, France 2 CentraleSupélec, L2S, France 3 Sorbonne University, CNRS, France ∗ http://melaseddik.github.io/ August 21, 2020 1 / 5
/ 2/5 Abstract Context: ◮ Compression of dense neural networks with the teacher-student approach. Motivation: ◮ Build lightweight neural networks that can fit into edge and IoT devices with limited resources (memory and computation). Proposed methods: ◮ We proposed two methods which rely on dimension reduction techniques (PCA and LDA). ◮ The dimension reduction is applied at each layer of the teacher net and then mapped to the layers of the student net using a multi-task loss function. 2 / 5
/ 3/5 Setting Given a Teacher Network (TN) trained on a dataset D with loss L TN � h (0) = x ∈ R p 0 (TN) : � W ( ℓ ) h ( ℓ − 1) + b ( ℓ ) � ∀ ℓ ∈ [ L ] h ( ℓ ) = f ℓ ∈ R p ℓ Construct a Student Network (SN) to train on D � h (0) = x ∈ R p 0 ˜ � b ( ℓ ) � (SN) : ∀ ℓ ∈ [ L ] ˜ h ( ℓ ) = f ℓ W ( ℓ )˜ ˜ h ( ℓ − 1) + ˜ ∈ R k ℓ Such that k ℓ ≪ p ℓ & Performance (SN) � Performance (TN) 3 / 5
/ 4/5 Proposed Methods (Net-PCAD & Net-LDAD) Given (TN) , a data matrix X and (TN) loss function L TN For each layer ℓ : 1. Extract the representations H ℓ of X from (TN) 2. Compute a projection matrix U ℓ ∈ R p ℓ × k ℓ through PCA or LDA on H ℓ Train (SN) as a multi-task 1 problem with L − 1 � ℓ h ( ℓ ) � � h ( ℓ ) , U ⊺ ˜ L SN = e − σ L TN + σ e − σ ℓ L mse + + σ ℓ � �� ℓ =1 Learning Task � �� (SN) Hidden Layers Task where σ and { σ ℓ } L − 1 ℓ =1 are learnable parameters. 1 Using the Homoscedastic loss function: A. Kendall et al. “Multitask learning using uncertainty to weigh losses for scene geometry and semantics” in Proceedings of IEEE CVPR, 2018. 4 / 5
/ 5/5 Experimental Setting & Results Layer (TN) (SN) Dense 1 p 0 × 1024 p 0 × k Dense 2 1024 × 512 k × k Dense 3 512 × 256 k × k Dense 4 256 × 10 k × 10 Table: Networks architectures. (SN) Datasets (TN) k = 50 100 200 MNIST 2 . 23s 0 . 38s 0 . 45s 0 . 65s 98% 97% 97 . 5% 97 . 8% FASHION 2 . 23s 0 . 38s 0 . 45s 0 . 65s 88% 87 . 5% 88 . 5% 88 . 5% CIFAR10 4 . 63s 0 . 75s 0 . 92s 1 . 35s 45% 50% 50 . 1% 50 . 3% Table: Networks performances. ⇒ k ℓ ≪ p ℓ & Performance (SN) � Performance (TN) 5 / 5

Recommend

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides

Evolving Neural Networks This lecture is based on Xin Yaos tutorial slides From Evolving

Evolving Neural Networks This lecture is based on Xin Yaos tutorial slides From Evolving Single Neural Networks to Evolving Ensembles presented at CEC 2007. Literature: Evolving Artificial Neural Networks , Xin Yao, Proceedings of the

281 views • 16 slides

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides

Neural Networks for Time Series Prediction 15-486/782: Artificial Neural Networks Fall 2006

Neural Networks for Time Series Prediction 15-486/782: Artificial Neural Networks Fall 2006 (based on earlier slides by Dave Touretzky and Kornel Laskowski) What is a Time Series? A sequence of vectors (or scalars) which depend on time t . In

975 views • 42 slides

Population Based Training of Neural Networks M. Jaderberg, V. Dalibard, S. Osindero, W.M.

Presented by: Devin Taylor Population Based Training of Neural Networks M. Jaderberg, V. Dalibard, S. Osindero, W.M. Czarnecki November 14, 2018 DeepMind, London, United Kingdom Problem Statement Problem statement Neural networks suffer from

317 views • 17 slides

Neural Networks: Prediction (i.e. the forward pass) Machine Learning Based on slides and

Neural Networks: Prediction (i.e. the forward pass) Machine Learning Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, 1 Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others Neural Networks What is a

470 views • 17 slides

Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker

Introduction HMM based method Deep Neural Networks (DNNs) Experiments and Results Conclusions Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker Verification Hossein Zeinali 1 , 2 , Luk s Burget 2 ,

301 views • 11 slides

Neural Networks for Time Series Prediction 15-496/782: Artificial Neural Networks Kornel

Neural Networks for Time Series Prediction 15-496/782: Artificial Neural Networks Kornel Laskowski Spring 2004 (based on slides by D. Touretzky, Spring 2002) What is a Time Series? A sequence of vectors (or scalars) which depend on time t . In

719 views • 42 slides

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The Problem of Long-Term Dependencies RNN short-term dependencies Language model trying to predict the next word based on the previous ones the clouds

729 views • 27 slides

Neural Networks as Stat Mech Systems Based on arXiv:1710.06570 [stat.ML], A

Neural Networks as Stat Mech Systems Based on arXiv:1710.06570 [stat.ML], A Correspondence Between Random Neural Networks and Statistical Field Theory Yoni Kahn, KICP/UIUC hep-ai 1/8/19 <latexit

462 views • 17 slides

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides

Dependency-based Convolutional Neural Networks for Sentence Embedding What is Hawaii

Dependency-based Convolutional Neural Networks for Sentence Embedding What is Hawaii s state flower ? ROOT Mingbo Ma Liang Huang Bing Xiang Bowen Zhou CUNY IBM T. J. Watson ACL 2015 Beijing Convolutional Neural Network

742 views • 50 slides

COMP304 Introduction to Neural Networks based on slides by: Christian Borgelt

COMP304 Introduction to Neural Networks based on slides by: Christian Borgelt http://www.borgelt.net/ Christian Borgelt Introduction to Neural Networks 1 Motivation: Why (Artificial) Neural Networks? (Neuro-)Biology / (Neuro-)Physiology

609 views • 33 slides

How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

Need for Machine . . . Neural Networks . . . How to Speed Up . . . Neural Networks: A . . . How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . . . Resulting Algorithm Faster by Taking Into How to

464 views • 17 slides

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Paul Nuytten CPSC 607 Why Model Neural Networks? The Neuron The nervous

697 views • 4 slides

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides

Neural Networks and Computation Graphs CS 6956: Deep Learning for NLP Based on slides and

Neural Networks and Computation Graphs CS 6956: Deep Learning for NLP Based on slides and material from Geoffrey Hinton, Richard Socher, Yoav Goldberg, and others. The computation graph slides are based on a tutorial Practical Neural

1.5k views • 121 slides

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides