EM Algorithm and Mixture Models Guojun Zhang University of Waterloo - PowerPoint PPT Presentation

Feb 04, 2023 •111 likes •218 views

EM Algorithm and Mixture Models Guojun Zhang University of Waterloo Unsupervised learning and clustering Learn the intrinsic representation of unlabeled data Other examples: density estimation, novelty detection Mixture model

EM Algorithm and Mixture Models Guojun Zhang University of Waterloo
Unsupervised learning and clustering • Learn the intrinsic representation of unlabeled data • Other examples: density estimation, novelty detection
Mixture model • Continuous: mixture of Gaussians • Discrete: mixture of Bernoullis
Gaussian Bernoulli: flipping a coin
Optimization algorithms • Loss function: negative log likelihood • Expectation-Maximization (DLR 1977):
Optimization algorithms • Loss function: negative log likelihood • Gradient descent:
k-cluster region • What if just some clusters are used? Has the algorithm learned the ground truth? How bad are these regions?
Potential project • To study how EM and GD (or any other algorithm) behave in learning mixture models • Can they avoid some bad local minima, such as the k-cluster regions? • Some Results/Guesses: 1) EM does but GD does not (on BMMs) 2) EM escapes exponentially faster than GD (on GMMs) • Ultimate goal: to understand their convergence property and the limit of each algorithm; to propose better algorithms • Need strong mathematical background: linear algebra, advanced calculus, probability theory and statistics, continuous optimization, (maybe) dynamical systems…
References • Christopher Bishop, “Pattern Recognition and Machine Learning” (2006). • Guojun Zhang, Pascal Poupart and George Trimponias, “Comparing EM with GD in Mixtures of Two Components,” to appear in UAI 2019. • Dempster, Arthur P ., Nan M. Laird and Donald B. Rubin. “Maximum likelihood from incomplete data via the EM algorithm.” Journal of the Royal Statistical Society: Series B (1977).

Recommend

The EM Algorithm The EM algorithm Mixture models Why EM works EM variants Learning

Preview The EM Algorithm The EM algorithm Mixture models Why EM works EM variants Learning with Missing Data The EM Algorithm Goal: Learn parameters of Bayes net with known structure Initialize parameters ignoring missing

363 views • 3 slides

Bag-of-components: an online algorithm for batch learning of mixture models Olivier Schwander

Information Geometry for mixtures Co-Mixture Models Bag of components Bag-of-components: an online algorithm for batch learning of mixture models Olivier Schwander Frank Nielsen Universit Pierre et Marie Curie, Paris, France cole

551 views • 20 slides

z A single Gaussian might be a poor fit . . . . . Simplest form is 2 layer . . . ... .

Overview Hidden Variable Models 1: Mixture Models Hidden variable models Mixture models Chris Williams Mixtures of Gaussians Aside: Kullback-Leibler divergence School of Informatics, University of Edinburgh The EM algorithm October 2008

395 views • 6 slides

EM Algorithm 11-09-2019 For Mixture Gaussian Models Instructor - Sriram Ganapathy

E9 205 Machine Learning for Signal Processing EM Algorithm 11-09-2019 For Mixture Gaussian Models Instructor - Sriram Ganapathy (sriramg@iisc.ac.in) Teaching Assistant - Prachi Singh (prachisingh@iisc.ac.in). EM Algorithm For GMMs EM

368 views • 18 slides

Bernoulli Mixture Models Victor Medina Researcher at SBIF DataCamp Mixture Models in R The

DataCamp Mixture Models in R MIXTURE MODELS IN R Bernoulli Mixture Models Victor Medina Researcher at SBIF DataCamp Mixture Models in R The handwritten digits dataset DataCamp Mixture Models in R Continuous versus discrete variables

440 views • 41 slides

Structure of mixture models Victor Medina Researcher at SBIF DataCamp Mixture Models in R

DataCamp Mixture Models in R MIXTURE MODELS IN R Structure of mixture models Victor Medina Researcher at SBIF DataCamp Mixture Models in R Description of mixture models 1. Which is the suitable probability distribution? Get familiar with

1.07k views • 36 slides

EM Algorithm 09-09-2019 For Mixture Gaussian Models Instructor - Sriram Ganapathy

E9 205 Machine Learning for Signal Processing EM Algorithm 09-09-2019 For Mixture Gaussian Models Instructor - Sriram Ganapathy (sriramg@iisc.ac.in) Teaching Assistant - Prachi Singh (prachisingh@iisc.ac.in). Basics of Information Theory

606 views • 23 slides

Gaussian Mixture Models & EM CE-717: Machine Learning Sharif University of Technology M.

Gaussian Mixture Models & EM CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Mixture Models: definition Mixture models: Linear supper-position of mixtures or components | =

1.04k views • 39 slides

Mixture models of truncated data for estimating the number of species. Li-Thiao-T e S

Context Mixture models Estimation Application to Metagenomics Mixture models of truncated data for estimating the number of species. Li-Thiao-T e S ebastien , Jean-Jacques Daudin, St ephane Robin Equipe Statistique et G enome, UMR

146 views • 13 slides

AND MACHINE LEARNING CHAPTER 10: MIXTURE MODELS AND EM Mixture Models - Define a joint

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 10: MIXTURE MODELS AND EM Mixture Models - Define a joint distribution over observed and latent variables - The corresponding distribution of the observed variables alone is obtained by

878 views • 62 slides

Classification of High Dimensional Data By Two-way Mixture Models Jia Li Statistics Department

Classification of High Dimensional Data By Two-way Mixture Models Jia Li Statistics Department The Pennsylvania State University 1 Outline Goals Two-way mixture model approach Background: mixture discriminant analysis Model

418 views • 23 slides

CSci 8980: Advanced Topics in Graphical Models Mixture Models, EM, Exponential Families

Variants Auxiliary Functions Mixture of Gaussians Exponential Family Mixtures of Exponential Families Bayes Net CSci 8980: Advanced Topics in Graphical Models Mixture Models, EM, Exponential Families Instructor: Arindam Banerjee September

1.04k views • 83 slides

A Method of Moments for Mixture Models and Hidden Markov Models Anima Anandkumar @ Daniel Hsu #

A Method of Moments for Mixture Models and Hidden Markov Models Anima Anandkumar @ Daniel Hsu # Sham M. Kakade # @ University of California, Irvine # Microsoft Research, New England Outline 1. Latent class models and parameter estimation 2.

1.44k views • 82 slides

Sub-quadratic Markov tree mixture models for probability density estimation Sourour Ammar 1 , Ph.

Sub-quadratic mixture models Sub-quadratic Markov tree mixture models for probability density estimation Sourour Ammar 1 , Ph. Leray 1 , L. Wehenkel 2 1 Equipe COnnaissances et D ecision, LINA UMR 6241 Ecole Polytechnique de lUniversit e

940 views • 65 slides

Models for Replicated Discrimination Tests: A Synthesis of Latent Class Mixture Models and

Models for Replicated Discrimination Tests Models for Replicated Discrimination Tests: A Synthesis of Latent Class Mixture Models and Generalized Linear Mixed Models Rune Haubo Bojesen Christensen & Per Bruun Brockhoff DTU Informatics

868 views • 21 slides

Deep Gaussian Mixture Models Cinzia Viroli (University of Bologna, Italy) joint with Geoff

Deep Gaussian Mixture Models Cinzia Viroli (University of Bologna, Italy) joint with Geoff McLachlan (University of Queensland, Australia) JOCLAD 2018, Lisbona, April 5th, 2018 Outline Deep Learning Mixture Models Deep Gaussian Mixture

915 views • 66 slides

Vine copula mixture models and clustering for non-Gaussian data Statistical Methods in Machine

Vine copula mixture models and clustering for non-Gaussian data Statistical Methods in Machine Learning Prof. Claudia Czado Ozge Sahin <ozge.sahin@tum.de> Bernoulli-IMS One World Symposium August 2020 Finite mixture models k

467 views • 14 slides

Efficient algorithms for estimating multi-view mixture models Daniel Hsu Microsoft Research, New

Efficient algorithms for estimating multi-view mixture models Daniel Hsu Microsoft Research, New England Outline Multi-view mixture models Multi-view method-of-moments Some applications and open questions Concluding remarks Part 1.

1.08k views • 85 slides

Clustering: Models and Algorithms Shikui Tu 2019-03-07 1 Outline Gaussian Mixture Models

Clustering: Models and Algorithms Shikui Tu 2019-03-07 1 Outline Gaussian Mixture Models (GMM) Expectation-Maximization (EM) for maximum likelihood Model selection, Bayesian learning 2 From distance to probability distance

383 views • 37 slides

Some algorithms to fit some reliability mixture models under censoring Laurent Bordes Didier

Some algorithms to fit some reliability mixture models under censoring Laurent Bordes Didier Chauveau University of Pau University of Orl eans COMPSTAT August 22-27, 2010 Laurent Bordes () Fitting some reliability mixture models 27

621 views • 35 slides

Auxiliary Mixture Sampling for Age-Period-Cohort Models Andrea Riebler & Leonhard Held

Auxiliary Mixture Sampling for Age-Period-Cohort Models Andrea Riebler & Leonhard Held University of Zurich Reisensburg, September 2007 Introduction Age-Period-Cohort Models Auxiliary Mixture Sampling Summary and Outlook Outline

367 views • 33 slides

Optimal transport for Gaussian mixture models Yongxin Chen, Tryphon T. Georgiou and Allen

Optimal transport for Gaussian mixture models Yongxin Chen, Tryphon T. Georgiou and Allen Tannenbaum Presented by: Zach Lucas Intro and Motivation A mixture model is a probabilistic model describing properties of populations with

357 views • 35 slides

Mixture Models Simulation-based Estimation Michel Bierlaire michel.bierlaire@epfl.ch

Mixture Models Simulation-based Estimation Michel Bierlaire michel.bierlaire@epfl.ch Transport and Mobility Laboratory Mixture Models Simulation-based Estimation p. 1/72 Outline Mixtures Capturing correlation

951 views • 74 slides

A model selection algorithm for mixture experiments including process variables Hugo Maruri and

A model selection algorithm for mixture experiments including process variables Hugo Maruri and Eva Riccomagno Department of Statistics, London School of Economics and Dipartimento di Matematica, Universit` a di Genova mODa 8, Almagro, Spain

353 views • 18 slides