deep belief networks
play

Deep Belief Networks 4-27-16 Announcement We are extending the - PowerPoint PPT Presentation

Nov 07, 2022 •556 likes •655 views

Deep Belief Networks 4-27-16 Announcement We are extending the deadline for the final project. You now have an extra week (due Monday 5/9). I still strongly recommend planning to finish your implementation this week. This will give you time to

  1. Deep Belief Networks 4-27-16

  2. Announcement We are extending the deadline for the final project. You now have an extra week (due Monday 5/9). I still strongly recommend planning to finish your implementation this week. This will give you time to run experiments, make tweaks, and try alternatives. You should also plan to finish your experiments a few days before the deadline so that you have time to write the paper.

  3. Recall: sigmoid activation function

  4. The problem with sigmoids: vanishing gradient The � terms tend to be small, and get smaller at each layer, meaning that backpropagation takes a very long time to update weights.

  5. First key idea: unsupervised weight initialization One layer at a time, train the weights to encode a compressed representation of the previous layer. This can be done with: ● Restricted Boltzman machines ● Auto-encoders After this pre-training phase (which uses only unlabeled data), train the network on labeled data with standard backprop.

  6. Second key idea: ditch the sigmoids How did we end up using sigmoid activation functions? ● Biological neurons act kind of like threshold functions. ● Sigmoids give continuous, differentiable approximations to thresholds. But maybe for artificial neural networks, there are better activation functions.

  7. Rectifier linear units

  8. Advantages of ReLUs ● The gradient never vanishes. ● Computing � is trivial. ○ The derivative is 1 or 0 everywhere. ● Activations tend to be sparse (lots of zeros). ● Unbounded range ⇒ better for regression.

  9. Third key idea: get TONS of data ● Make the training fast with GPU parallelization. ● Instead of presenting the same examples many times, go get more data. Example data sets for deep learning: ● Millions of youtube videos. ● Tens of thousands of go games.

Recommend

Overview Independence Belief Networks Conditional Independence Belief networks Chris

Overview Independence Belief Networks Conditional Independence Belief networks Chris

Overview Independence Belief Networks Conditional Independence Belief networks Chris Williams Constructing belief networks Inference in belief networks School of Informatics, University of Edinburgh Learning in belief

498 views • 6 slides
Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders Generative Adversarial Networks Pr. Fabien MOUTARDE Center for Robotics MINES ParisTech PSL Universit Paris Fabien.Moutarde@mines-paristech.fr

777 views • 23 slides
Belief Networks Some Belief Network references E. Charniak Bayesian Networks without

Belief Networks Some Belief Network references E. Charniak Bayesian Networks without

Belief Networks Some Belief Network references E. Charniak Bayesian Networks without Tears, AI Magazine Winter 1991, pp 50-63 Chris Williams, School of Informatics University of Edinburgh D. Heckerman, A Tutorial on Learning

486 views • 6 slides
outline of this tutorial motivations 1 ACISS09 tutorial on deep belief nets deep

outline of this tutorial motivations 1 ACISS09 tutorial on deep belief nets deep

outline of this tutorial motivations 1 ACISS09 tutorial on deep belief nets deep autoencoders deep belief nets sigmoid belief nets 2 Marcus Frean why are they hard to train? could layer-by-layer training work? Melbourne, 2009

509 views • 15 slides
Deep Belief Networks Presented by Joseph Nunn Psych 149/239 Computational Models of Cognition

Deep Belief Networks Presented by Joseph Nunn Psych 149/239 Computational Models of Cognition

Deep Belief Networks Presented by Joseph Nunn Psych 149/239 Computational Models of Cognition University of California, Irvine Winter 2015 1 Talk Structure Connectionist Background Material To Recognize Shapes, First Learn to Generate

642 views • 25 slides
Lecture 13 Deep Belief Networks Michael Picheny, Bhuvana Ramabhadran, Stanley F . Chen IBM T.J.

Lecture 13 Deep Belief Networks Michael Picheny, Bhuvana Ramabhadran, Stanley F . Chen IBM T.J.

Lecture 13 Deep Belief Networks Michael Picheny, Bhuvana Ramabhadran, Stanley F . Chen IBM T.J. Watson Research Center Yorktown Heights, New York, USA {picheny,bhuvana,stanchen}@us.ibm.com 12 December 2012 A spectrum of Machine Learning

1.39k views • 58 slides
Lecture 13 Deep Belief Networks Michael Picheny, Bhuvana Ramabhadran, Stanley F . Chen, Markus

Lecture 13 Deep Belief Networks Michael Picheny, Bhuvana Ramabhadran, Stanley F . Chen, Markus

Lecture 13 Deep Belief Networks Michael Picheny, Bhuvana Ramabhadran, Stanley F . Chen, Markus Nussbaum-Thom Watson Group IBM T.J. Watson Research Center Yorktown Heights, New York, USA {picheny,bhuvana,stanchen,nussbaum}@us.ibm.com 20

1.13k views • 84 slides
Inference in Belief Networks CMPUT 366: Intelligent Systems P&M 8.4 Lecture Outline

Inference in Belief Networks CMPUT 366: Intelligent Systems P&M 8.4 Lecture Outline

Inference in Belief Networks CMPUT 366: Intelligent Systems P&M 8.4 Lecture Outline 1. Recap 2. Factors 3. Variable Elimination 4. Efficiency Recap: Belief Networks Definition: A belief network (or Bayesian network )

1.27k views • 23 slides
Dynamics of Social Networks of Belief Networks Yong-Yeol (YY) Ahn, @yy http://yongyeol.com

Dynamics of Social Networks of Belief Networks Yong-Yeol (YY) Ahn, @yy http://yongyeol.com

Dynamics of Social Networks of Belief Networks Yong-Yeol (YY) Ahn, @yy http://yongyeol.com CNetS, IUNI, School of Informatics and Computing Indiana University Bloomington Many ways to think about networks of networks A network of

753 views • 74 slides
8. Other Deep Architectures CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt

8. Other Deep Architectures CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt

8. Other Deep Architectures CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt Kira and Ian Goodfellow A brief overview of other architectures Unsupervised Architectures Deep Belief Networks Autoencoders GANs

439 views • 25 slides
Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations deeplearning.cce2020@gmail.com Deep Networks Intuition Neural networks with multiple hidden layers - Deep networks [Hinton, 2006] Deep Networks Intuition

1.33k views • 28 slides
Bayesian Belief Networks Decision Theoretic Agents Introduction to Probability [Ch13]

Bayesian Belief Networks Decision Theoretic Agents Introduction to Probability [Ch13]

RN, Chapter 14 Bayesian Belief Networks Decision Theoretic Agents Introduction to Probability [Ch13] Belief networks [Ch14] Introduction [Ch14.1-14.2] Bayesian Net Inference [Ch14.4] (Bucket Elimination) Dynamic Belief

634 views • 49 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for

Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for

Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels

1.45k views • 34 slides
Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for

Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for

Deep Networks Andrea Passerini passerini@disi.unitn.it Machine Learning Deep Networks Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels

563 views • 43 slides
5.2 Learning Bayesian networks: General idea See Witten et al. 2011. Bayesian (belief) networks

5.2 Learning Bayesian networks: General idea See Witten et al. 2011. Bayesian (belief) networks

5.2 Learning Bayesian networks: General idea See Witten et al. 2011. Bayesian (belief) networks aim at representing probability distributions of features and how to combine them to make predictions about the likelihood that a particular

622 views • 22 slides
Bayesian Networks (= Belief Networks) Sven Koenig, USC Russell and Norvig, 3 rd Edition, Sections

Bayesian Networks (= Belief Networks) Sven Koenig, USC Russell and Norvig, 3 rd Edition, Sections

12/18/2019 Bayesian Networks (= Belief Networks) Sven Koenig, USC Russell and Norvig, 3 rd Edition, Sections 14.1-14.4 These slides are new and can contain mistakes and typos. Please report them to Sven (skoenig@usc.edu). 1 Rule-Based Systems

235 views • 21 slides
E9 205 Machine Learning for Signal Processing Understanding Deep Networks 08-11-2019 Instructor

E9 205 Machine Learning for Signal Processing Understanding Deep Networks 08-11-2019 Instructor

E9 205 Machine Learning for Signal Processing Understanding Deep Networks 08-11-2019 Instructor - Sriram Ganapathy (sriramg@iisc.ac.in.in) Understanding Deep Networks Understanding Deep Networks Understanding Deep Networks SVHN dataset

366 views • 23 slides
Deep networks CS 446 The ERM perspective These lectures will follow an ERM perspective on deep

Deep networks CS 446 The ERM perspective These lectures will follow an ERM perspective on deep

Deep networks CS 446 The ERM perspective These lectures will follow an ERM perspective on deep networks: Pick a model/predictor class (network architecture). (We will spend most of our time on this!) Pick a loss/risk. (We will almost

608 views • 40 slides
Belief network inference Four main approaches to determine posterior distributions in belief

Belief network inference Four main approaches to determine posterior distributions in belief

Belief network inference Four main approaches to determine posterior distributions in belief networks: Variable Elimination: exploit the structure of the network to eliminate (sum out) the non-observed, non-query variables one at a time.

330 views • 29 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Introduction to Artificial Intelligence Belief networks Chapter 15.12 Dieter Fox Based on

Introduction to Artificial Intelligence Belief networks Chapter 15.12 Dieter Fox Based on

Introduction to Artificial Intelligence Belief networks Chapter 15.12 Dieter Fox Based on AIMA Slides S. Russell and P. Norvig, 1998 c Chapter 15.12 0-0 Outline Bayesian networks: syntax and semantics Inference tasks Based

469 views • 22 slides
Architecture Perceptron Highway Networks Highway Networks Allows training very deep

Architecture Perceptron Highway Networks Highway Networks Allows training very deep

Highway Networks for Visual Question Answering Aaditya Prakash PhD advisor: James Storer Brandeis University Architecture Perceptron Highway Networks Highway Networks Allows training very deep networks Srivastava et al trained

174 views • 16 slides
On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides

More recommend