explicitly imposing constraints in deep networks via
play

EXPLICITLY IMPOSING CONSTRAINTS IN DEEP NETWORKS VIA CONDITIONAL - PowerPoint PPT Presentation

Nov 19, 2023 •264 likes •797 views

AAAI 19 EXPLICITLY IMPOSING CONSTRAINTS IN DEEP NETWORKS VIA CONDITIONAL GRADIENTS GIVES IMPROVED GENERALIZATION AND FASTER CONVERGENCE Sathya Ravi, Tuan Dinh, Vishnu Lokhande, Vikas Singh Department of Computer Sciences University of

  1. AAAI’ 19 EXPLICITLY IMPOSING CONSTRAINTS IN DEEP NETWORKS VIA CONDITIONAL GRADIENTS GIVES IMPROVED GENERALIZATION AND FASTER CONVERGENCE Sathya Ravi, Tuan Dinh, Vishnu Lokhande, Vikas Singh Department of Computer Sciences University of Wisconsin–Madison 11/14/2018

  3. DEEP LEARNING

  4. DEEP LEARNING min ! n L ( W ) Solve W ∈

  5. <latexit sha1_base64="+DqeC3o24NjVnLkKQwoKp/pcRUk=">ACBnicbVDLSsNAFJ3UV62vqEtBovQgpREBHUhFEVw4aKCMYUmhMl0g6dPJiZiCV058ZfceNCxa3f4M6/cdJmoa0Hhjmcy/3uMnjApGN9aW5+YXGpvFxZWV1b39A3t+5EnHJMLByzmLd9JAijEbEklYy0E05Q6DNi+4OL3LfvCRc0jm7lMCFuiHoRDShGUkmevntds+vwDohkn3fzy5HnvNAYVCzD9Rf9/Sq0TDGgLPELEgVFGh5+pfTjXEakhihoTomEYi3QxSTEjo4qTCpIgPEA90lE0QiERbja+YwT3ldKFQczViyQcq787MhQKMQx9VZmvK6a9XPzP6QyOHEzGiWpJBGeDApSBmUM81Bgl3KCJRsqgjCnaleI+4gjLFV0FRWCOX3yLEOG6cN8+ao2jwv0iDHbAHasAEx6AJrkALWACDR/AMXsGb9qS9aO/ax6S0pBU92+APtM8fbsyXVg=</latexit> <latexit sha1_base64="+DqeC3o24NjVnLkKQwoKp/pcRUk=">ACBnicbVDLSsNAFJ3UV62vqEtBovQgpREBHUhFEVw4aKCMYUmhMl0g6dPJiZiCV058ZfceNCxa3f4M6/cdJmoa0Hhjmcy/3uMnjApGN9aW5+YXGpvFxZWV1b39A3t+5EnHJMLByzmLd9JAijEbEklYy0E05Q6DNi+4OL3LfvCRc0jm7lMCFuiHoRDShGUkmevntds+vwDohkn3fzy5HnvNAYVCzD9Rf9/Sq0TDGgLPELEgVFGh5+pfTjXEakhihoTomEYi3QxSTEjo4qTCpIgPEA90lE0QiERbja+YwT3ldKFQczViyQcq787MhQKMQx9VZmvK6a9XPzP6QyOHEzGiWpJBGeDApSBmUM81Bgl3KCJRsqgjCnaleI+4gjLFV0FRWCOX3yLEOG6cN8+ao2jwv0iDHbAHasAEx6AJrkALWACDR/AMXsGb9qS9aO/ax6S0pBU92+APtM8fbsyXVg=</latexit> <latexit sha1_base64="+DqeC3o24NjVnLkKQwoKp/pcRUk=">ACBnicbVDLSsNAFJ3UV62vqEtBovQgpREBHUhFEVw4aKCMYUmhMl0g6dPJiZiCV058ZfceNCxa3f4M6/cdJmoa0Hhjmcy/3uMnjApGN9aW5+YXGpvFxZWV1b39A3t+5EnHJMLByzmLd9JAijEbEklYy0E05Q6DNi+4OL3LfvCRc0jm7lMCFuiHoRDShGUkmevntds+vwDohkn3fzy5HnvNAYVCzD9Rf9/Sq0TDGgLPELEgVFGh5+pfTjXEakhihoTomEYi3QxSTEjo4qTCpIgPEA90lE0QiERbja+YwT3ldKFQczViyQcq787MhQKMQx9VZmvK6a9XPzP6QyOHEzGiWpJBGeDApSBmUM81Bgl3KCJRsqgjCnaleI+4gjLFV0FRWCOX3yLEOG6cN8+ao2jwv0iDHbAHasAEx6AJrkALWACDR/AMXsGb9qS9aO/ax6S0pBU92+APtM8fbsyXVg=</latexit> <latexit sha1_base64="+DqeC3o24NjVnLkKQwoKp/pcRUk=">ACBnicbVDLSsNAFJ3UV62vqEtBovQgpREBHUhFEVw4aKCMYUmhMl0g6dPJiZiCV058ZfceNCxa3f4M6/cdJmoa0Hhjmcy/3uMnjApGN9aW5+YXGpvFxZWV1b39A3t+5EnHJMLByzmLd9JAijEbEklYy0E05Q6DNi+4OL3LfvCRc0jm7lMCFuiHoRDShGUkmevntds+vwDohkn3fzy5HnvNAYVCzD9Rf9/Sq0TDGgLPELEgVFGh5+pfTjXEakhihoTomEYi3QxSTEjo4qTCpIgPEA90lE0QiERbja+YwT3ldKFQczViyQcq787MhQKMQx9VZmvK6a9XPzP6QyOHEzGiWpJBGeDApSBmUM81Bgl3KCJRsqgjCnaleI+4gjLFV0FRWCOX3yLEOG6cN8+ao2jwv0iDHbAHasAEx6AJrkALWACDR/AMXsGb9qS9aO/ax6S0pBU92+APtM8fbsyXVg=</latexit> DEEP LEARNING min ! n L ( W ) Solve W ∈ L ( W ) = E ξ f ( W, ξ ) ξ = (x,y) ~ 𝒠

  7. Compute an estimate of gradient

  8. <latexit sha1_base64="aGnU6jnbxPGn2A/R7LrfwJsOIM=">ACFXicbVA9SwNBEN3z2/gVtbRZDIihjsR1EIQbSwsFIwRcuGY20x0yd7esTsnhCO/wsa/YmOhYivY+W/cxBR+PRh4vDfDzLw4U9KS7394I6Nj4xOTU9Olmdm5+YXy4tKlTXMjsCZSlZqrGCwqbFGkhReZQYhiRXW485x36/forEy1RfUzbCZwLWbSmAnBSVt+pRQZtBjx/wekR8i4dIEFGoIVbAQ5KqhcVpL6J1Z29E5Ypf9Qfgf0kwJBU2xFlUfg9bqcgT1CQUWNsI/IyaBRiSQmGvFOYWMxAduMaGoxoStM1i8FaPrzmlxdupcaWJD9TvEwUk1naT2HUmQDf2t9cX/MaObX3moXUWU6oxdeidq4pbyfEW9Jg4JU1xEQRrpbubgBA4JckiUXQvD75b+ktl3drwbnO5XDo2EaU2yFrbJ1FrBdshO2BmrMcHu2AN7Ys/evfovXivX60j3nBmf2A9/YJIWdtg=</latexit> <latexit sha1_base64="aGnU6jnbxPGn2A/R7LrfwJsOIM=">ACFXicbVA9SwNBEN3z2/gVtbRZDIihjsR1EIQbSwsFIwRcuGY20x0yd7esTsnhCO/wsa/YmOhYivY+W/cxBR+PRh4vDfDzLw4U9KS7394I6Nj4xOTU9Olmdm5+YXy4tKlTXMjsCZSlZqrGCwqbFGkhReZQYhiRXW485x36/forEy1RfUzbCZwLWbSmAnBSVt+pRQZtBjx/wekR8i4dIEFGoIVbAQ5KqhcVpL6J1Z29E5Ypf9Qfgf0kwJBU2xFlUfg9bqcgT1CQUWNsI/IyaBRiSQmGvFOYWMxAduMaGoxoStM1i8FaPrzmlxdupcaWJD9TvEwUk1naT2HUmQDf2t9cX/MaObX3moXUWU6oxdeidq4pbyfEW9Jg4JU1xEQRrpbubgBA4JckiUXQvD75b+ktl3drwbnO5XDo2EaU2yFrbJ1FrBdshO2BmrMcHu2AN7Ys/evfovXivX60j3nBmf2A9/YJIWdtg=</latexit> <latexit sha1_base64="aGnU6jnbxPGn2A/R7LrfwJsOIM=">ACFXicbVA9SwNBEN3z2/gVtbRZDIihjsR1EIQbSwsFIwRcuGY20x0yd7esTsnhCO/wsa/YmOhYivY+W/cxBR+PRh4vDfDzLw4U9KS7394I6Nj4xOTU9Olmdm5+YXy4tKlTXMjsCZSlZqrGCwqbFGkhReZQYhiRXW485x36/forEy1RfUzbCZwLWbSmAnBSVt+pRQZtBjx/wekR8i4dIEFGoIVbAQ5KqhcVpL6J1Z29E5Ypf9Qfgf0kwJBU2xFlUfg9bqcgT1CQUWNsI/IyaBRiSQmGvFOYWMxAduMaGoxoStM1i8FaPrzmlxdupcaWJD9TvEwUk1naT2HUmQDf2t9cX/MaObX3moXUWU6oxdeidq4pbyfEW9Jg4JU1xEQRrpbubgBA4JckiUXQvD75b+ktl3drwbnO5XDo2EaU2yFrbJ1FrBdshO2BmrMcHu2AN7Ys/evfovXivX60j3nBmf2A9/YJIWdtg=</latexit> <latexit sha1_base64="aGnU6jnbxPGn2A/R7LrfwJsOIM=">ACFXicbVA9SwNBEN3z2/gVtbRZDIihjsR1EIQbSwsFIwRcuGY20x0yd7esTsnhCO/wsa/YmOhYivY+W/cxBR+PRh4vDfDzLw4U9KS7394I6Nj4xOTU9Olmdm5+YXy4tKlTXMjsCZSlZqrGCwqbFGkhReZQYhiRXW485x36/forEy1RfUzbCZwLWbSmAnBSVt+pRQZtBjx/wekR8i4dIEFGoIVbAQ5KqhcVpL6J1Z29E5Ypf9Qfgf0kwJBU2xFlUfg9bqcgT1CQUWNsI/IyaBRiSQmGvFOYWMxAduMaGoxoStM1i8FaPrzmlxdupcaWJD9TvEwUk1naT2HUmQDf2t9cX/MaObX3moXUWU6oxdeidq4pbyfEW9Jg4JU1xEQRrpbubgBA4JckiUXQvD75b+ktl3drwbnO5XDo2EaU2yFrbJ1FrBdshO2BmrMcHu2AN7Ys/evfovXivX60j3nBmf2A9/YJIWdtg=</latexit> Compute an estimate of gradient W t +1 = W t � η t r ˜ L t ( W t )

Recommend

Imposing Hard Constraints on Deep Networks: Promises and Limitations min w R ( w ) s.t C ( w ) =

Imposing Hard Constraints on Deep Networks: Promises and Limitations min w R ( w ) s.t C ( w ) =

Imposing Hard Constraints on Deep Networks: Promises and Limitations min w R ( w ) s.t C ( w ) = 0 P. Marquez-Neila, M. Salzmann, and P. Fua EPFL Switzerland Motivation: 3D Pose Estimation Given a CNN trained to predict the 3D locations of

460 views • 13 slides
Enforcing constraints for interpolation and extrapolation in Generative Adversarial Networks

Enforcing constraints for interpolation and extrapolation in Generative Adversarial Networks

Enforcing constraints for interpolation and extrapolation in Generative Adversarial Networks Panos Stinis (joint work with T. Hagge, A.M. Tartakovsky and E. Yeung) Pacific Northwest National Laboratory space space space Supported by Deep

255 views • 14 slides
Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations deeplearning.cce2020@gmail.com Deep Networks Intuition Neural networks with multiple hidden layers - Deep networks [Hinton, 2006] Deep Networks Intuition

1.33k views • 28 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for

Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for

Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels

563 views • 43 slides
Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for

Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for

Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels

1.45k views • 34 slides
From Traditional Neural From Traditional NN . . . Networks to Deep Learning Need to Go Beyond .

From Traditional Neural From Traditional NN . . . Networks to Deep Learning Need to Go Beyond .

Why Traditional . . . How the Need for Fast . . . Faster Differentiation: . . . Beyond Traditional NN From Traditional Neural From Traditional NN . . . Networks to Deep Learning Need to Go Beyond . . . Constraints Are . . . and Beyond

548 views • 38 slides
Deep Learning With Constraints Yatin Nandwani Work done in collaboration with Abhishek Pathak

Deep Learning With Constraints Yatin Nandwani Work done in collaboration with Abhishek Pathak

Deep Learning With Constraints Yatin Nandwani Work done in collaboration with Abhishek Pathak Under the guidance of Prof. Mausam and Prof. Parag Singla Learning with Constraints: Motivation Modern day AI == Deep Learning (DL) [Learn from

810 views • 68 slides
E9 205 Machine Learning for Signal Processing Understanding Deep Networks 08-11-2019 Instructor

E9 205 Machine Learning for Signal Processing Understanding Deep Networks 08-11-2019 Instructor

E9 205 Machine Learning for Signal Processing Understanding Deep Networks 08-11-2019 Instructor - Sriram Ganapathy (sriramg@iisc.ac.in.in) Understanding Deep Networks Understanding Deep Networks Understanding Deep Networks SVHN dataset

366 views • 23 slides
Deep networks CS 446 The ERM perspective These lectures will follow an ERM perspective on deep

Deep networks CS 446 The ERM perspective These lectures will follow an ERM perspective on deep

Deep networks CS 446 The ERM perspective These lectures will follow an ERM perspective on deep networks: Pick a model/predictor class (network architecture). (We will spend most of our time on this!) Pick a loss/risk. (We will almost

608 views • 40 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Architecture Perceptron Highway Networks Highway Networks Allows training very deep

Architecture Perceptron Highway Networks Highway Networks Allows training very deep

Highway Networks for Visual Question Answering Aaditya Prakash PhD advisor: James Storer Brandeis University Architecture Perceptron Highway Networks Highway Networks Allows training very deep networks Srivastava et al trained

174 views • 16 slides
On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides
Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference ICTP, Trieste, Jan 29-31 2018 Deep Neural Networks When properly trained, deep NN are capable of learning the complex functional dependence of

442 views • 17 slides
Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides
No Spurious Local Minima in Training Deep Quadratic Networks Abbas Kazemipour Conference on

No Spurious Local Minima in Training Deep Quadratic Networks Abbas Kazemipour Conference on

No Spurious Local Minima in Training Deep Quadratic Networks Abbas Kazemipour Conference on Mathematical Theory of Deep Neural Networks October 31, 2019 New York City, NY Need for New Optimization Theory The mystery of deep neural networks

549 views • 17 slides
Compiling Deep Nets Scott Sanner Goal of this talk Will not evangelize deep networks /

Compiling Deep Nets Scott Sanner Goal of this talk Will not evangelize deep networks /

Compiling Deep Nets Scott Sanner Goal of this talk Will not evangelize deep networks / successes Go to ICML, NIPS, Silicon Valley, read tech news Just believe But deep nets do not solve all problems Yet Lack

393 views • 37 slides
Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit:

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit:

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit: Vlad Morariu Training (Deep) Neural Networks Computational graphs Improvements to gradient descent Stochastic gradient descent Momentum

814 views • 21 slides
Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders Generative Adversarial Networks Pr. Fabien MOUTARDE Center for Robotics MINES ParisTech PSL Universit Paris Fabien.Moutarde@mines-paristech.fr

777 views • 23 slides
Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane

Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane

Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane Mallat, Ivan Dokmanic, Martin de Hoop cole Normale Suprieure www.di.ens.fr/data Deep Convolutional Networks ( x ) x ( u ) L j L 1 f M (

520 views • 47 slides
CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10

CS885 Reinforcement Learning Lecture 12: June 8, 2018 Deep Recurrent Q-Networks [GBC] Chap. 10 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Recurrent neural networks Long short term memory (LSTM) networks Deep

116 views • 11 slides
Understanding (or not) Deep Convolutional Networks Stphane Mallat cole Normale Suprieure

Understanding (or not) Deep Convolutional Networks Stphane Mallat cole Normale Suprieure

Understanding (or not) Deep Convolutional Networks Stphane Mallat cole Normale Suprieure www.di.ens.fr/data Deep Neural Networks Approximations of high-dimensional functions from examples, for classification and regression.

387 views • 38 slides
Image Classification with Deep Networks Ronan Collobert Facebook AI Research Feb 11, 2015

Image Classification with Deep Networks Ronan Collobert Facebook AI Research Feb 11, 2015

Image Classification with Deep Networks Ronan Collobert Facebook AI Research Feb 11, 2015 Overview Origins of Deep Learning Shallow vs Deep Perceptron Multi Layer Perceptrons Going Deeper Why? Issues (and fix)?

916 views • 65 slides

More recommend