mixture density networks mixture density networks
play

MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS Charles Martin - PowerPoint PPT Presentation

Jul 05, 2023 •248 likes •1.26k views

MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS Charles Martin SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA Remember that

  1. MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS Charles Martin

  2. SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA

  3. SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA Remember that most RNNs (and most deep learning models) end with a so�max layer.

  4. SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA Remember that most RNNs (and most deep learning models) end with a so�max layer. This layer outputs a probability distribution for a set of categorical predictions.

  5. SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA Remember that most RNNs (and most deep learning models) end with a so�max layer. This layer outputs a probability distribution for a set of categorical predictions. E.g.:

  6. SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA Remember that most RNNs (and most deep learning models) end with a so�max layer. This layer outputs a probability distribution for a set of categorical predictions. E.g.: image labels,

  7. SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA Remember that most RNNs (and most deep learning models) end with a so�max layer. This layer outputs a probability distribution for a set of categorical predictions. E.g.: image labels, letters, words,

  8. SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA Remember that most RNNs (and most deep learning models) end with a so�max layer. This layer outputs a probability distribution for a set of categorical predictions. E.g.: image labels, letters, words, musical notes,

  9. SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA Remember that most RNNs (and most deep learning models) end with a so�max layer. This layer outputs a probability distribution for a set of categorical predictions. E.g.: image labels, letters, words, musical notes, robot commands,

  10. SO FAR; RNNS THAT MODEL CATEGORICAL DATA SO FAR; RNNS THAT MODEL CATEGORICAL DATA Remember that most RNNs (and most deep learning models) end with a so�max layer. This layer outputs a probability distribution for a set of categorical predictions. E.g.: image labels, letters, words, musical notes, robot commands, moves in chess.

  11. EXPRESSIVE DATA IS OFTEN CONTINUOUS EXPRESSIVE DATA IS OFTEN CONTINUOUS

  13. SO ARE BIO-SIGNALS SO ARE BIO-SIGNALS Image Credit: Wikimedia

  14. CATEGORICAL VS. CONTINUOUS MODELS CATEGORICAL VS. CONTINUOUS MODELS

  15. NORMAL (GAUSSIAN) DISTRIBUTION NORMAL (GAUSSIAN) DISTRIBUTION

  16. NORMAL (GAUSSIAN) DISTRIBUTION NORMAL (GAUSSIAN) DISTRIBUTION “Standard” probability distribution

  17. NORMAL (GAUSSIAN) DISTRIBUTION NORMAL (GAUSSIAN) DISTRIBUTION “Standard” probability distribution Has two parameters:

  18. NORMAL (GAUSSIAN) DISTRIBUTION NORMAL (GAUSSIAN) DISTRIBUTION “Standard” probability distribution Has two parameters: mean ( μ ) and

  19. NORMAL (GAUSSIAN) DISTRIBUTION NORMAL (GAUSSIAN) DISTRIBUTION “Standard” probability distribution Has two parameters: mean ( μ ) and standard deviation ( σ )

  20. NORMAL (GAUSSIAN) DISTRIBUTION NORMAL (GAUSSIAN) DISTRIBUTION “Standard” probability distribution Has two parameters: mean ( μ ) and standard deviation ( σ ) Probability Density Function:

  21. NORMAL (GAUSSIAN) DISTRIBUTION NORMAL (GAUSSIAN) DISTRIBUTION “Standard” probability distribution Has two parameters: mean ( μ ) and standard deviation ( σ ) Probability Density Function: (x− μ )2 1 N(x ∣ μ , σ 2) = e− 2 σ 2 √ 2π σ 2

  22. PROBLEM: NORMAL DISTRIBUTION MIGHT NOT FIT DATA PROBLEM: NORMAL DISTRIBUTION MIGHT NOT FIT DATA What if the data is complicated?

  23. PROBLEM: NORMAL DISTRIBUTION MIGHT NOT FIT DATA PROBLEM: NORMAL DISTRIBUTION MIGHT NOT FIT DATA What if the data is complicated? It’s easy to “fit” a normal model to any data.

  24. PROBLEM: NORMAL DISTRIBUTION MIGHT NOT FIT DATA PROBLEM: NORMAL DISTRIBUTION MIGHT NOT FIT DATA What if the data is complicated? It’s easy to “fit” a normal model to any data. Just calculate μ and σ

  25. PROBLEM: NORMAL DISTRIBUTION MIGHT NOT FIT DATA PROBLEM: NORMAL DISTRIBUTION MIGHT NOT FIT DATA What if the data is complicated? It’s easy to “fit” a normal model to any data. Just calculate μ and σ But this might not fit the data well.

  26. MIXTURE OF NORMALS MIXTURE OF NORMALS Three groups of parameters:

  27. MIXTURE OF NORMALS MIXTURE OF NORMALS Three groups of parameters: means ( μ ): location of each component

  28. MIXTURE OF NORMALS MIXTURE OF NORMALS Three groups of parameters: means ( μ ): location of each component standard deviations ( σ ): width of each component

  29. MIXTURE OF NORMALS MIXTURE OF NORMALS Three groups of parameters: means ( μ ): location of each component standard deviations ( σ ): width of each component Weight (π): height of each curve

  30. MIXTURE OF NORMALS MIXTURE OF NORMALS Three groups of parameters: means ( μ ): location of each component standard deviations ( σ ): width of each component Weight (π): height of each curve Probability Density Function:

  31. MIXTURE OF NORMALS MIXTURE OF NORMALS Three groups of parameters: means ( μ ): location of each component standard deviations ( σ ): width of each component Weight (π): height of each curve Probability Density Function: K πiN(x ∣ μ , σ 2) p(x) = ∑ i=1

  32. THIS SOLVES OUR PROBLEM: THIS SOLVES OUR PROBLEM: Returning to our modelling problem, let’s plot the PDF of a evenly-weighted mixture of the two sample normal models. We set: In this case, I knew the right parameters, but normally you would have to estimate , or learn , these somehow…

  33. THIS SOLVES OUR PROBLEM: THIS SOLVES OUR PROBLEM: Returning to our modelling problem, let’s plot the PDF of a evenly-weighted mixture of the two sample normal models. We set: K = 2 In this case, I knew the right parameters, but normally you would have to estimate , or learn , these somehow…

  34. THIS SOLVES OUR PROBLEM: THIS SOLVES OUR PROBLEM: Returning to our modelling problem, let’s plot the PDF of a evenly-weighted mixture of the two sample normal models. We set: K = 2 π = [0.5, 0.5] In this case, I knew the right parameters, but normally you would have to estimate , or learn , these somehow…

  35. THIS SOLVES OUR PROBLEM: THIS SOLVES OUR PROBLEM: Returning to our modelling problem, let’s plot the PDF of a evenly-weighted mixture of the two sample normal models. We set: K = 2 π = [0.5, 0.5] μ = [ − 5, 5] In this case, I knew the right parameters, but normally you would have to estimate , or learn , these somehow…

  36. THIS SOLVES OUR PROBLEM: THIS SOLVES OUR PROBLEM: Returning to our modelling problem, let’s plot the PDF of a evenly-weighted mixture of the two sample normal models. We set: K = 2 π = [0.5, 0.5] μ = [ − 5, 5] σ = [2, 3] In this case, I knew the right parameters, but normally you would have to estimate , or learn , these somehow…

  37. THIS SOLVES OUR PROBLEM: THIS SOLVES OUR PROBLEM: Returning to our modelling problem, let’s plot the PDF of a evenly-weighted mixture of the two sample normal models. We set: K = 2 π = [0.5, 0.5] μ = [ − 5, 5] σ = [2, 3] (bold used to indicate the vector of parameters for each component) In this case, I knew the right parameters, but normally you would have to estimate , or learn , these somehow…

  38. MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS

  39. MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS Neural networks used to model complicated real-valued data.

  40. MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS Neural networks used to model complicated real-valued data. i.e., data that might not be very “normal”

  41. MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS Neural networks used to model complicated real-valued data. i.e., data that might not be very “normal” Usual approach: use a neuron with linear activation to make predictions.

  42. MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS Neural networks used to model complicated real-valued data. i.e., data that might not be very “normal” Usual approach: use a neuron with linear activation to make predictions. Training function could be MSE (mean squared error).

  43. MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS Neural networks used to model complicated real-valued data. i.e., data that might not be very “normal” Usual approach: use a neuron with linear activation to make predictions. Training function could be MSE (mean squared error). Problem! This is equivalent to fitting to a single normal model! �

  44. MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS Neural networks used to model complicated real-valued data. i.e., data that might not be very “normal” Usual approach: use a neuron with linear activation to make predictions. Training function could be MSE (mean squared error). Problem! This is equivalent to fitting to a single normal model! � (See Bishop, C (1994) for proof and more details)

  45. MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS

  46. MIXTURE DENSITY NETWORKS MIXTURE DENSITY NETWORKS Idea: output parameters of a mixture model instead!

Recommend

Auto-conditioned Recurrent Mixture Density Networks for Learning Generalizable Robot

Auto-conditioned Recurrent Mixture Density Networks for Learning Generalizable Robot

Auto-conditioned Recurrent Mixture Density Networks for Learning Generalizable Robot Skills Hejia Zhang, Eric Heiden, Stefanos Nikolaidis, Joseph J. Lim, Gaurav S. Sukhatme Introduction 2 Introduction learn generalizable robot skills

171 views • 16 slides
3 M IXTURE DENSITY ESTIMATION In this chapter we consider mixture densities, the main building

3 M IXTURE DENSITY ESTIMATION In this chapter we consider mixture densities, the main building

3 M IXTURE DENSITY ESTIMATION In this chapter we consider mixture densities, the main building block for the dimen- sion reduction techniques described in the following chapters. In the first section we introduce mixture densities and the

598 views • 12 slides
Joint Optimisation of Tandem Systems using Gaussian Mixture Density Neural Network Discriminative

Joint Optimisation of Tandem Systems using Gaussian Mixture Density Neural Network Discriminative

Joint Optimisation of Tandem Systems using Gaussian Mixture Density Neural Network Discriminative Sequence Training Chao Zhang and Phil Woodland March 8, 2017 Cambridge University Engineering Department Introduction Tandem Systems as Mixture

926 views • 17 slides
Sub-quadratic Markov tree mixture models for probability density estimation Sourour Ammar 1 , Ph.

Sub-quadratic Markov tree mixture models for probability density estimation Sourour Ammar 1 , Ph.

Sub-quadratic mixture models Sub-quadratic Markov tree mixture models for probability density estimation Sourour Ammar 1 , Ph. Leray 1 , L. Wehenkel 2 1 Equipe COnnaissances et D ecision, LINA UMR 6241 Ecole Polytechnique de lUniversit e

940 views • 65 slides
Neural Networks - II Henrik I Christensen Robotics & Intelligent Machines @ GT Georgia

Neural Networks - II Henrik I Christensen Robotics & Intelligent Machines @ GT Georgia

Introduction Mixture Density Networks Bayesian Neural Networks Summary Neural Networks - II Henrik I Christensen Robotics & Intelligent Machines @ GT Georgia Institute of Technology, Atlanta, GA 30332-0280 hic@cc.gatech.edu Henrik I

357 views • 25 slides
Accelerating the EM Algorithm for Mixture Density Estimation Homer Walker Mathematical Sciences

Accelerating the EM Algorithm for Mixture Density Estimation Homer Walker Mathematical Sciences

Accelerating the EM Algorithm for Mixture Density Estimation Homer Walker Mathematical Sciences Department Worcester Polytechnic Instititute t r t s Pss Pr

546 views • 21 slides
EM Algorithm and Mixture Models Guojun Zhang University of Waterloo Unsupervised learning and

EM Algorithm and Mixture Models Guojun Zhang University of Waterloo Unsupervised learning and

EM Algorithm and Mixture Models Guojun Zhang University of Waterloo Unsupervised learning and clustering Learn the intrinsic representation of unlabeled data Other examples: density estimation, novelty detection Mixture model

219 views • 9 slides
Models for Probability Distributions and Density Functions 1 General Concepts Parametric:

Models for Probability Distributions and Density Functions 1 General Concepts Parametric:

Models for Probability Distributions and Density Functions 1 General Concepts Parametric: E.g., Gaussian, Gamma, Binomial Non-Parametric: E.g., kernel estimates Intermediate models: Mixture Models 2 Gaussian Mixture Model

382 views • 22 slides
A Formal Approach to Decipher a Mixture of Genetic and Metabolic Networks Fabien Corblin, Eric

A Formal Approach to Decipher a Mixture of Genetic and Metabolic Networks Fabien Corblin, Eric

A Formal Approach to Decipher a Mixture of Genetic and Metabolic Networks Fabien Corblin, Eric Fanchon, Laurent Trilling Workshop Toward Systems Biology 31 mai 2011 General problems exploration of regulatory biological networks qualitative

556 views • 20 slides
Parametric Models Part II: Expectation-Maximization and Mixture Density Estimation Selim Aksoy

Parametric Models Part II: Expectation-Maximization and Mixture Density Estimation Selim Aksoy

Parametric Models Part II: Expectation-Maximization and Mixture Density Estimation Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Spring 2019 CS 551, Spring 2019 2019, Selim Aksoy

356 views • 32 slides
APPLIED MACHINE LEARNING Probability Density Functions Gaussian Mixture Models 1 APPLIED

APPLIED MACHINE LEARNING Probability Density Functions Gaussian Mixture Models 1 APPLIED

APPLIED MACHINE LEARNING APPLIED MACHINE LEARNING Probability Density Functions Gaussian Mixture Models 1 APPLIED MACHINE LEARNING Discrete Probabilities Consider two variables x and y taking discrete values over the intervals [1.....N ] and

689 views • 54 slides
Bernoulli Mixture Models Victor Medina Researcher at SBIF DataCamp Mixture Models in R The

Bernoulli Mixture Models Victor Medina Researcher at SBIF DataCamp Mixture Models in R The

DataCamp Mixture Models in R MIXTURE MODELS IN R Bernoulli Mixture Models Victor Medina Researcher at SBIF DataCamp Mixture Models in R The handwritten digits dataset DataCamp Mixture Models in R Continuous versus discrete variables

440 views • 41 slides
Structure of mixture models Victor Medina Researcher at SBIF DataCamp Mixture Models in R

Structure of mixture models Victor Medina Researcher at SBIF DataCamp Mixture Models in R

DataCamp Mixture Models in R MIXTURE MODELS IN R Structure of mixture models Victor Medina Researcher at SBIF DataCamp Mixture Models in R Description of mixture models 1. Which is the suitable probability distribution? Get familiar with

1.07k views • 36 slides
Lecture 20 Lecture 20 Nov 12 th 2008 Clustering with Mixture of Gaussians Clustering with Mixture

Lecture 20 Lecture 20 Nov 12 th 2008 Clustering with Mixture of Gaussians Clustering with Mixture

Lecture 20 Lecture 20 Nov 12 th 2008 Clustering with Mixture of Gaussians Clustering with Mixture of Gaussians Underlying model for data: assumes a generative process: Each cluster follows a Gaussian distribution First choose which

302 views • 19 slides
5G integrated Fiber-Wireless networks exploiting existing photonic technologies for high-density

5G integrated Fiber-Wireless networks exploiting existing photonic technologies for high-density

A RISTOTLE U NIVERSITY OF T HESSALONIKI 5G integrated Fiber-Wireless networks exploiting existing photonic technologies for high-density SDN-programmable network architectures George Kalfas , Sotirios Papaioannou & Nikos Pleros Dptm of

538 views • 12 slides
Classification of High Dimensional Data By Two-way Mixture Models Jia Li Statistics Department

Classification of High Dimensional Data By Two-way Mixture Models Jia Li Statistics Department

Classification of High Dimensional Data By Two-way Mixture Models Jia Li Statistics Department The Pennsylvania State University 1 Outline Goals Two-way mixture model approach Background: mixture discriminant analysis Model

418 views • 23 slides
Gaussian Mixture Models & EM CE-717: Machine Learning Sharif University of Technology M.

Gaussian Mixture Models & EM CE-717: Machine Learning Sharif University of Technology M.

Gaussian Mixture Models & EM CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Mixture Models: definition Mixture models: Linear supper-position of mixtures or components | =

1.04k views • 39 slides
Local Density Estimation for Contention Window Adaptation in Vehicular Networks Razvan Stanica,

Local Density Estimation for Contention Window Adaptation in Vehicular Networks Razvan Stanica,

Local Density Estimation for Contention Window Adaptation in Vehicular Networks Razvan Stanica, Emmanuel Chaput, Andr-Luc Beylot University of Toulouse Institut de Recherche en Informatique de Toulouse 22 nd Annual IEEE International Symposium

316 views • 20 slides
Density Functional Theory meets Bayesian Neural Networks: A New Paradigm in the Study of

Density Functional Theory meets Bayesian Neural Networks: A New Paradigm in the Study of

Density Functional Theory meets Bayesian Neural Networks: A New Paradigm in the Study of Neutron Stars Jorge Piekarewicz - Florida State University Highlights of 2015 Welcome to JPhysG's 2015 highlights! Before you get to the articles,

502 views • 18 slides
Density and trace results in generalized fractal networks Serge Nicaise Universit de

Density and trace results in generalized fractal networks Serge Nicaise Universit de

Density and trace results in generalized fractal networks Serge Nicaise Universit de Valenciennes et du Hainaut-Cambresis Laboratoire de Mathematiques et leurs Applications de Valenciennes, LAMAV joint work with Adrien Semin Serge Nicaise

618 views • 25 slides
Relative Density Chapters 3.5 Relative Density 1 2/5/2015 Minimum Density Pluviate soil from

Relative Density Chapters 3.5 Relative Density 1 2/5/2015 Minimum Density Pluviate soil from

2/5/2015 Relative Density Chapters 3.5 Relative Density 1 2/5/2015 Minimum Density Pluviate soil from height of 25 mm Maximum Density Vibrate for 8 minutes 2 2/5/2015 Relative Density Correlations Clean sand (0 F c 5%) Sand w/

373 views • 13 slides
Density-based vs. Proximity-based Anycast Routing for Mobile Networks Vincent Lenders, Martin May

Density-based vs. Proximity-based Anycast Routing for Mobile Networks Vincent Lenders, Martin May

Density-based vs. Proximity-based Anycast Routing for Mobile Networks Vincent Lenders, Martin May and Bernhard Plattner Department of Information Technology and Electrical Engineering Swiss Federal Institute of Technology (ETH) Z urich,

465 views • 13 slides
Solutions Unit 6 1 Solutions Homogenous Mixture (Solution) two or more substances mixed

Solutions Unit 6 1 Solutions Homogenous Mixture (Solution) two or more substances mixed

Solutions Unit 6 1 Solutions Homogenous Mixture (Solution) two or more substances mixed together to have a uniform composition, its components are not distinguishable from one another Heterogenous Mixture Homogenous Mixture (Not a

523 views • 29 slides
CSC321 Lecture 18: Mixture Modeling Roger Grosse Roger Grosse CSC321 Lecture 18: Mixture

CSC321 Lecture 18: Mixture Modeling Roger Grosse Roger Grosse CSC321 Lecture 18: Mixture

CSC321 Lecture 18: Mixture Modeling Roger Grosse Roger Grosse CSC321 Lecture 18: Mixture Modeling 1 / 27 Overview Some examples of situations where youd use unupservised learning You want to understand how a scientific field has changed

613 views • 27 slides

More recommend