Stochastic Deep Networks Gwendoline De Bie, Gabriel Peyr, Marco - PowerPoint PPT Presentation

Sep 12, 2023 •231 likes •303 views

Stochastic Deep Networks Gwendoline De Bie, Gabriel Peyr, Marco Cuturi Deep Architectures on Density Inputs t=1, 1000 cells t=2, 650 cells t=3, 890 cells of varying physical attributes Representing inputs as densities (discretized in practice)

Stochastic Deep Networks Gwendoline De Bie, Gabriel Peyré, Marco Cuturi
Deep Architectures on Density Inputs t=1, 1000 cells t=2, 650 cells t=3, 890 cells of varying physical attributes Representing inputs as densities (discretized in practice) ➢ How to define a ‘layer’ of a Deep Net taking such inputs ? ➢
Proposed Layer: Elementary Block (EB) random deterministic or random Discrete case: Fully connected case: → , where: where , non-linearity Deterministic output: Classical warping: → deterministic → random
Proposed Architectures Tasks Discriminative Generative Predictive Y deterministic Y random Y random X random X noise + code X random
Approximation Property ➢ Theoretically, three blocks are enough Theorem (Universal Approximation) . Let F a continuous map for the convergence in law, mapping measures supported on compact sets. Then three EBs are necessary to approximate F arbitrarily close : , there exists three continuous maps f, g, h , such that, for all random vectors X, where concatenates a uniform random vector.
Loss functions: Applications ■ Cross-entropy (classification) ■ Regularized Wasserstein (Cuturi, 2013) In practice: f fully connected ■ Sinkhorn divergence (Genevay et al, 2018) Classification Generation Dynamics Flocking MNIST as point clouds Modelnet40 as point clouds MNIST as point clouds model 99,2% accuracy, 2 EBs 83,5% accuracy, 2 EBs 2 EBs 5 EBs
Conclusion / Open Problems ■ New formalism for stochastic deep architectures ➢ Probability distributions Deterministic feature vectors ➢ ■ Robustness & approximation power ■ Perspectives Understanding block roles ➢ Investigate translation & rotation equivariance ➢ Poster: #30 Pacific Ballroom today - see you there !

Recommend

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao and Quanquan Gu Computer Science Department 1 / 14 Learning Over-parameterized DNNs Empirical observation on extremely wide deep neural networks

631 views • 14 slides

Learning Accurate Low-bit Deep Neural Networks with Stochastic Quantization Yinpeng Dong 1 ,

Learning Accurate Low-bit Deep Neural Networks with Stochastic Quantization Yinpeng Dong 1 , Renkun Ni 2 , Jianguo Li 3 , Yurong Chen 3 , Jun Zhu 1 , Hang Su 1 1 Department of CST, Tsinghua University 2 University of Virginia 3 Intel Labs China

320 views • 14 slides

Unraveling the mysteries of stochastic gradient descent on deep neural networks Pratik Chaudhari

Unraveling the mysteries of stochastic gradient descent on deep neural networks Pratik Chaudhari UCLA VISION LAB 1 The question measures disagreement of predictions with ground truth x = argmin f ( x ) Cat Dog ... x weights aka

496 views • 25 slides

Learning for Single-Shot Confidence Calibration in Deep Neural Networks through Stochastic

Learning for Single-Shot Confidence Calibration in Deep Neural Networks through Stochastic Inferences Seonguk Seo* 1 Paul Hongsuck Seo* 1,2 Bohyung Han 1 Overconfidence Issues Overconfidence to unseen examples 99.9+% sure for the

759 views • 31 slides

Reduced-memory training and deployment of deep residual networks by stochastic binary quantization

Reduced-memory training and deployment of deep residual networks by stochastic binary quantization Mark D. McDonnell 1 , Ruchun Wang 2 and Andr van Schaik 2 cls-lab.org 2 BENS Laboratory 1 Computational Learning Systems Laboratory MARCS

256 views • 23 slides

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit:

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit: Vlad Morariu Training (Deep) Neural Networks Computational graphs Improvements to gradient descent Stochastic gradient descent Momentum

814 views • 21 slides

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations deeplearning.cce2020@gmail.com Deep Networks Intuition Neural networks with multiple hidden layers - Deep networks [Hinton, 2006] Deep Networks Intuition

1.33k views • 28 slides

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides

Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for

Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels

1.45k views • 34 slides

Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for

563 views • 43 slides

E9 205 Machine Learning for Signal Processing Understanding Deep Networks 08-11-2019 Instructor

E9 205 Machine Learning for Signal Processing Understanding Deep Networks 08-11-2019 Instructor - Sriram Ganapathy (sriramg@iisc.ac.in.in) Understanding Deep Networks Understanding Deep Networks Understanding Deep Networks SVHN dataset

366 views • 23 slides

Deep networks CS 446 The ERM perspective These lectures will follow an ERM perspective on deep

Deep networks CS 446 The ERM perspective These lectures will follow an ERM perspective on deep networks: Pick a model/predictor class (network architecture). (We will spend most of our time on this!) Pick a loss/risk. (We will almost

608 views • 40 slides

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides

CHAPTER V V CHAPTER Annealing by Stochastic Annealing by Stochastic Neural Networks for

Ugur HALICI - METU EEE - ANKARA 11/18/2004 CHAPTER V V CHAPTER Annealing by Stochastic Annealing by Stochastic Neural Networks for Optimization Neural Networks for Optimization CHAPTER V : V : Annealing by Stochastic CHAPTER Annealing by

468 views • 24 slides

Architecture Perceptron Highway Networks Highway Networks Allows training very deep

Highway Networks for Visual Question Answering Aaditya Prakash PhD advisor: James Storer Brandeis University Architecture Perceptron Highway Networks Highway Networks Allows training very deep networks Srivastava et al trained

174 views • 16 slides

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics Activation Functions Stochastic Gradient Descent (SGD) Regularization (Dropout) Training Tips and Tricks Neural Network (NN)

828 views • 45 slides

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference ICTP, Trieste, Jan 29-31 2018 Deep Neural Networks When properly trained, deep NN are capable of learning the complex functional dependence of

442 views • 17 slides

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides

No Spurious Local Minima in Training Deep Quadratic Networks Abbas Kazemipour Conference on

No Spurious Local Minima in Training Deep Quadratic Networks Abbas Kazemipour Conference on Mathematical Theory of Deep Neural Networks October 31, 2019 New York City, NY Need for New Optimization Theory The mystery of deep neural networks

549 views • 17 slides

Compiling Deep Nets Scott Sanner Goal of this talk Will not evangelize deep networks /

Compiling Deep Nets Scott Sanner Goal of this talk Will not evangelize deep networks / successes Go to ICML, NIPS, Silicon Valley, read tech news Just believe But deep nets do not solve all problems Yet Lack

393 views • 37 slides

Fractional Gaussian Noise, Fractional Gaussian Noise, Subdiffusion and Stochastic and Stochastic

Fractional Gaussian Noise, Fractional Gaussian Noise, Subdiffusion and Stochastic and Stochastic Subdiffusion Networks in Biophysics Networks in Biophysics Samuel Kou Department of Statistics Harvard University Single-Molecule Experiments

741 views • 46 slides

Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

CS6501: Deep Learning for Visual Recognition Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap Regression vs Classification Generalization / Overfitting / Underfitting Regularization

633 views • 30 slides