sub quadratic markov tree mixture models for probability
play

Sub-quadratic Markov tree mixture models for probability density - PowerPoint PPT Presentation

Jul 13, 2023 •278 likes •940 views

Sub-quadratic mixture models Sub-quadratic Markov tree mixture models for probability density estimation Sourour Ammar 1 , Ph. Leray 1 , L. Wehenkel 2 1 Equipe COnnaissances et D ecision, LINA UMR 6241 Ecole Polytechnique de lUniversit e

  1. Sub-quadratic mixture models Sub-quadratic Markov tree mixture models for probability density estimation Sourour Ammar 1 , Ph. Leray 1 , L. Wehenkel 2 1 Equipe COnnaissances et D´ ecision, LINA UMR 6241 Ecole Polytechnique de l’Universit´ e de Nantes 2 Department of EECS and GIGA-Research, University of Li` ege COMPSTAT’2010 - Paris - 22-27 August 2010 S. Ammar et al. Sub-quadratic mixture models (1/22)

  2. Sub-quadratic mixture models A simple idea Proposition Develop density estimation techniques that could scale to very high-dimensional spaces, by exploiting the Perturb & Combine idea with probabilistic graphical models. Outline Background Our proposal Some results Conclusions and Further works S. Ammar et al. Sub-quadratic mixture models (2/22)

  3. Sub-quadratic mixture models Background Perturb and combine principle P&C principle in supervised learning Principle : (Bagging, Random forests, Extremely randomized trees) How can we apply this idea to density estimation with Bayesian networks (BN)? S. Ammar et al. Sub-quadratic mixture models (3/22)

  4. Sub-quadratic mixture models Background Perturb and combine principle P&C principle in supervised learning Principle : (Bagging, Random forests, Extremely randomized trees) Learning algorithm (research proc.) How can we apply this idea to density estimation with Bayesian networks (BN)? S. Ammar et al. Sub-quadratic mixture models (3/22)

  5. Sub-quadratic mixture models Background Perturb and combine principle P&C principle in supervised learning Principle : (Bagging, Random forests, Extremely randomized trees) Learning Perturb Weak algorithm algorithm (randomization) (research proc.) How can we apply this idea to density estimation with Bayesian networks (BN)? S. Ammar et al. Sub-quadratic mixture models (3/22)

  6. � � � Sub-quadratic mixture models Background Perturb and combine principle P&C principle in supervised learning Principle : (Bagging, Random forests, Extremely randomized trees) Learning Perturb Weak algorithm algorithm (randomization) (research proc.) Weak Weak Weak algorithm algorithm algorithm 1 m 2 How can we apply this idea to density estimation with Bayesian networks (BN)? S. Ammar et al. Sub-quadratic mixture models (3/22)

  7. � � � � � � Sub-quadratic mixture models Background Perturb and combine principle P&C principle in supervised learning Principle : (Bagging, Random forests, Extremely randomized trees) Learning Perturb Weak algorithm algorithm (randomization) (research proc.) Weak Weak Weak algorithm algorithm algorithm 1 m 2 prediction 1 prediction 2 prediction m How can we apply this idea to density estimation with Bayesian networks (BN)? S. Ammar et al. Sub-quadratic mixture models (3/22)

  8. � � � � � � Sub-quadratic mixture models Background Perturb and combine principle P&C principle in supervised learning Principle : (Bagging, Random forests, Extremely randomized trees) Learning Perturb Weak algorithm algorithm (randomization) (research proc.) Weak Weak Weak algorithm algorithm algorithm 1 m 2 prediction 1 prediction 2 prediction m Combine (Weighting schema) Final prediction How can we apply this idea to density estimation with Bayesian networks (BN)? S. Ammar et al. Sub-quadratic mixture models (3/22)

  9. � � � � � � Sub-quadratic mixture models Background Perturb and combine principle P&C principle in supervised learning Principle : (Bagging, Random forests, Extremely randomized trees) Learning Perturb Weak algorithm algorithm (randomization) (research proc.) Weak Weak Weak algorithm algorithm algorithm 1 m 2 prediction 1 prediction 2 prediction m Combine (Weighting schema) Final prediction How can we apply this idea to density estimation with Bayesian networks (BN)? S. Ammar et al. Sub-quadratic mixture models (3/22)

  10. Sub-quadratic mixture models Background Density estimation with BN Density estimation with BN A B D E C S. Ammar et al. Sub-quadratic mixture models (4/22)

  11. Sub-quadratic mixture models Background Density estimation with BN Density estimation with BN ˜ S A B D E C S. Ammar et al. Sub-quadratic mixture models (4/22)

  12. Sub-quadratic mixture models Background Density estimation with BN Density estimation with BN ˜ S A B C D E 0 0 1 0 0 0 1 1 1 1 1 0 0 0 0 A B 1 1 1 1 1 0 0 0 0 1 D E C 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 S. Ammar et al. Sub-quadratic mixture models (4/22)

  13. Sub-quadratic mixture models Background Density estimation with BN Density estimation with BN ˜ S A B C D E 0 0 1 0 0 B D 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 C 0 0 0 0 1 E 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 A 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 S. Ammar et al. Sub-quadratic mixture models (4/22)

  14. Sub-quadratic mixture models Background Density estimation with BN Density estimation with BN ˜ S A B C D E 0 0 1 0 0 B D 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 C 0 0 0 0 1 E 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 A 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 S. Ammar et al. Sub-quadratic mixture models (4/22)

  15. Sub-quadratic mixture models Background Density estimation with BN Density estimation with BN ˜ S A B C D E 0 0 1 0 0 B D 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 C 0 0 0 0 1 E 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 A 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 S. Ammar et al. Sub-quadratic mixture models (4/22)

  16. � � � Sub-quadratic mixture models Background Density estimation with BN Density estimation with BN ˜ S A B C D E 0 0 1 0 0 B D 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 C 0 0 0 0 1 E 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 A 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 S. Ammar et al. Sub-quadratic mixture models (4/22)

  17. Sub-quadratic mixture models Background Density estimation with BN Bayesian averaging Instead of searching for an optimal model (structure + parameters): Assume prior probabilities over the space of structures Determine posterior probabilities of each model given a dataset approach the target distribution by avereaging the different models wighted by their posterior probabilities Caveats : Exact Bayesian averaging over large space of models is not ‘scalable’ ⇒ requires to strongly constrain the space of structures S. Ammar et al. Sub-quadratic mixture models (5/22)

  18. Sub-quadratic mixture models Outline Background Our proposal Some results Conclusions and Further works S. Ammar et al. Sub-quadratic mixture models (6/22)

  19. Sub-quadratic mixture models Proposal Strategy Use simple spaces of graphical structures ˜ S (e.g. chains, trees, poly-trees etc.) Do not assume that target distribution is representable by one of these structures Rather, assume that target distribution may be approximated well by a mixture of a reasonable number of ( S , θ ∗ ) pairs, S ∈ ˜ S S. Ammar et al. Sub-quadratic mixture models (7/22)

  20. Sub-quadratic mixture models Proposal Generic algorithm principle ˜ S A B C D E 0 0 1 0 0 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 0 0 0 0 1 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 S. Ammar et al. Sub-quadratic mixture models (8/22)

  21. � � � � Sub-quadratic mixture models Proposal Generic algorithm principle ˜ S A B C D E 0 0 1 0 0 0 1 1 1 1 T 1 T 2 T m 1 0 0 0 0 1 1 1 1 1 0 0 0 0 1 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 S. Ammar et al. Sub-quadratic mixture models (8/22)

  22. � � � � � � Sub-quadratic mixture models Proposal Generic algorithm principle ˜ S A B C D E 0 0 1 0 0 0 1 1 1 1 T 1 T 2 T m 1 0 0 0 0 1 1 1 1 1 0 0 0 0 1 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 θ ∗ θ ∗ θ ∗ 1 2 m S. Ammar et al. Sub-quadratic mixture models (8/22)

  23. � � � � � � Sub-quadratic mixture models Proposal Generic algorithm principle ˜ S A B C D E 0 0 1 0 0 0 1 1 1 1 T 1 T 2 T m 1 0 0 0 0 1 1 1 1 1 0 0 0 0 1 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 θ ∗ θ ∗ θ ∗ 1 2 m µ 1 µ 2 µ m S. Ammar et al. Sub-quadratic mixture models (8/22)

  24. � � � � � � Sub-quadratic mixture models Proposal Generic algorithm principle ˜ S A B C D E 0 0 1 0 0 0 1 1 1 1 T 1 T 2 T m 1 0 0 0 0 1 1 1 1 1 0 0 0 0 1 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 θ ∗ θ ∗ θ ∗ 1 2 m µ 1 µ 2 µ m Final model ( θ ∗ i , µ i ) i =1 .. m S. Ammar et al. Sub-quadratic mixture models (8/22)

  25. � � � � � � Sub-quadratic mixture models Proposal Degrees of freedom ˜ S What space ˜ S ? A B C D E 0 0 1 0 0 0 1 1 1 1 T 1 T 2 T m 1 0 0 0 0 1 1 1 1 1 0 0 0 0 1 1 0 1 0 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 θ ∗ θ ∗ θ ∗ 1 2 m µ 1 µ 2 µ m Final model ( θ ∗ i , µ i ) i =1 .. m S. Ammar et al. Sub-quadratic mixture models (9/22)

Recommend

Structure of mixture models Victor Medina Researcher at SBIF DataCamp Mixture Models in R

Structure of mixture models Victor Medina Researcher at SBIF DataCamp Mixture Models in R

DataCamp Mixture Models in R MIXTURE MODELS IN R Structure of mixture models Victor Medina Researcher at SBIF DataCamp Mixture Models in R Description of mixture models 1. Which is the suitable probability distribution? Get familiar with

1.07k views • 36 slides
A Method of Moments for Mixture Models and Hidden Markov Models Anima Anandkumar @ Daniel Hsu #

A Method of Moments for Mixture Models and Hidden Markov Models Anima Anandkumar @ Daniel Hsu #

A Method of Moments for Mixture Models and Hidden Markov Models Anima Anandkumar @ Daniel Hsu # Sham M. Kakade # @ University of California, Irvine # Microsoft Research, New England Outline 1. Latent class models and parameter estimation 2.

1.44k views • 82 slides
Bernoulli Mixture Models Victor Medina Researcher at SBIF DataCamp Mixture Models in R The

Bernoulli Mixture Models Victor Medina Researcher at SBIF DataCamp Mixture Models in R The

DataCamp Mixture Models in R MIXTURE MODELS IN R Bernoulli Mixture Models Victor Medina Researcher at SBIF DataCamp Mixture Models in R The handwritten digits dataset DataCamp Mixture Models in R Continuous versus discrete variables

440 views • 41 slides
Models for Probability Distributions and Density Functions 1 General Concepts Parametric:

Models for Probability Distributions and Density Functions 1 General Concepts Parametric:

Models for Probability Distributions and Density Functions 1 General Concepts Parametric: E.g., Gaussian, Gamma, Binomial Non-Parametric: E.g., kernel estimates Intermediate models: Mixture Models 2 Gaussian Mixture Model

382 views • 22 slides
Markov Systems, Markov Decision Processes, and Dynamic Programming Andrew W. Moore Note to

Markov Systems, Markov Decision Processes, and Dynamic Programming Andrew W. Moore Note to

Artificial Intelligence 15-381 April 5, 2007 Sequential Decision Problems & Markov Decision Processes Recap of last lecture Reasoning over time - Markov Processes - Hidden Markov Models - modeling state transitions - probability of

321 views • 16 slides
Probability Recap CS 188: Artificial Intelligence Hidden Markov Models Conditional probability

Probability Recap CS 188: Artificial Intelligence Hidden Markov Models Conditional probability

Probability Recap CS 188: Artificial Intelligence Hidden Markov Models Conditional probability Product rule Chain rule X, Y independent if and only if: X and Y are conditionally independent given Z if and only if: Instructors:

231 views • 11 slides
Markov processes (Markov chains) Construct a Bayes net from these variables: parents? Markov

Markov processes (Markov chains) Construct a Bayes net from these variables: parents? Markov

Markov processes (Markov chains) Construct a Bayes net from these variables: parents? Markov assumption: X t depends on bounded subset of X 0: t 1 Temporal probability models First-order Markov process: P ( X t | X 0: t 1 ) = P ( X t | X t

359 views • 7 slides
CSE P 590 A Markov Models and Hidden Markov Models

CSE P 590 A Markov Models and Hidden Markov Models

CSE P 590 A Markov Models and Hidden Markov Models http://upload.wikimedia.org/wikipedia/commons/b/ba/Calico_cat Dosage Compensation Reminder: Proteins Read DNA and X-Inactivation E.g.: 2 copies (mom/dad) of each

1.28k views • 21 slides
Clustering: Models and Algorithms Shikui Tu 2019-03-07 1 Outline Gaussian Mixture Models

Clustering: Models and Algorithms Shikui Tu 2019-03-07 1 Outline Gaussian Mixture Models

Clustering: Models and Algorithms Shikui Tu 2019-03-07 1 Outline Gaussian Mixture Models (GMM) Expectation-Maximization (EM) for maximum likelihood Model selection, Bayesian learning 2 From distance to probability distance

383 views • 37 slides
CSE 527 " Markov Models and Hidden Markov Models !

CSE 527 " Markov Models and Hidden Markov Models !

CSE 527 " Markov Models and Hidden Markov Models ! http://upload.wikimedia.org/wikipedia/commons/b/ba/Calico_cat ! Dosage Compensation Reminder: Proteins Read DNA ! and X-Inactivation ! E.g.: ! 2 copies (mom/dad) of each chromosome

770 views • 17 slides
CSE P 527 Markov Models and Hidden Markov Models 1 2

CSE P 527 Markov Models and Hidden Markov Models 1 2

CSE P 527 Markov Models and Hidden Markov Models 1 2 http://upload.wikimedia.org/wikipedia/commons/b/ba/Calico_cat Dosage Compensation and X-Inactivation 2 copies (mom/dad) of each chromosome 1-23 Mostly, both copies of each gene are

970 views • 66 slides
CSE 427 Markov Models and Hidden Markov Models 2

CSE 427 Markov Models and Hidden Markov Models 2

CSE 427 Markov Models and Hidden Markov Models 2 http://upload.wikimedia.org/wikipedia/commons/b/ba/Calico_cat Dosage Compensation and X-Inactivation 2 copies (mom/dad) of each chromosome 1-23 Mostly, both copies of each gene are

1.07k views • 60 slides
Introduction to Markov Models Estimating the probability of phrases of words, sentences, etc .

Introduction to Markov Models Estimating the probability of phrases of words, sentences, etc .

Introduction to Markov Models Estimating the probability of phrases of words, sentences, etc . But first: A few preliminaries CIS 391 - Intro to AI 2 What counts as a word? A tricky question. CIS 391 - Intro to AI 3 How to find

530 views • 31 slides
Gaussian Mixture Models & EM CE-717: Machine Learning Sharif University of Technology M.

Gaussian Mixture Models & EM CE-717: Machine Learning Sharif University of Technology M.

Gaussian Mixture Models & EM CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Mixture Models: definition Mixture models: Linear supper-position of mixtures or components | =

1.04k views • 39 slides
Markov Logic Markov Logic Probability First-Order Logic Propositional Logic Markov Logic

Markov Logic Markov Logic Probability First-Order Logic Propositional Logic Markov Logic

Putting the Pieces Together Markov Logic Markov Logic Probability First-Order Logic Propositional Logic Markov Logic Definition A Markov Logic Network (MLN) is a set of A logical KB is a set of hard constraints pairs (F, w) where on

350 views • 6 slides
Probability Recap CS 4100: Artificial Intelligence Hidden Markov Models Co Conditional

Probability Recap CS 4100: Artificial Intelligence Hidden Markov Models Co Conditional

Probability Recap CS 4100: Artificial Intelligence Hidden Markov Models Co Conditional probability Pr Product rule Ch Chain rule X, , Y in independent if if and only ly if if: X an and Y ar are co e conditional ally

267 views • 6 slides
Probability*and*Statistics* ! ! for*Computer*Science** All!models!are!wrong,!but!some!

Probability*and*Statistics* ! ! for*Computer*Science** All!models!are!wrong,!but!some!

Probability*and*Statistics* ! ! for*Computer*Science** All!models!are!wrong,!but!some! models!are!useful555!George!Box! !! Credit:!wikipedia! Hongye!Liu,!Teaching!Assistant!Prof,!CS361,!UIUC,!04.30.2020! Contents* Markov!chain!

808 views • 43 slides
1 X 1 X 2 X 3 Ghostbusters HMM Chain Rule and HMMs E 1 E 2 E 3 P(X 1 ) = uniform 1/9 1/9

1 X 1 X 2 X 3 Ghostbusters HMM Chain Rule and HMMs E 1 E 2 E 3 P(X 1 ) = uniform 1/9 1/9

Hidden Markov Models 1 Hidden Markov Models Markov chains not so useful for most agents Need observations to update your beliefs Hidden Markov models (HMMs) Underlying Markov chain over states X You observe outputs (effects) at

617 views • 7 slides
Markov chains and Hidden Markov Models 9000 Markov chains and HMMs We will discuss: Markov

Markov chains and Hidden Markov Models 9000 Markov chains and HMMs We will discuss: Markov

Markov chains and Hidden Markov Models 9000 Markov chains and HMMs We will discuss: Markov chains Hidden Markov Models (HMMs) Algorithms: Viterbi, forward, backward, posterior decoding Profile HMMs Baum-Welch algorithm 9001

1.16k views • 87 slides
CSE 527 Lectures 12-13 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2

CSE 527 Lectures 12-13 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2

CSE 527 Lectures 12-13 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2 adjacent nts, same strand (not Watson-Crick pair; p mnemonic for the phosphodiester bond of the DNA backbone) C of CpG is often (70-80%)

690 views • 55 slides
CSE 527 Lectures 11-12 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2

CSE 527 Lectures 11-12 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2

CSE 527 Lectures 11-12 Markov Models and Hidden Markov Models DNA Methylation CH 3 CpG - 2 adjacent nts, same strand (not Watson-Crick pair; p mnemonic for the phosphodiester bond of the DNA backbone) C of CpG is often (70-80%)

780 views • 55 slides
HMM Review Lecture Outline 1. Markov models 2. Hidden Markov

HMM Review Lecture Outline 1. Markov models 2. Hidden Markov

HMM Review Lecture Outline 1. Markov models 2. Hidden Markov models 3. Viterbi algorithm MARKOV MODELS One View of Text Sequence of symbols (bytes,

489 views • 46 slides
Markov Chains DS GA 1002 Probability and Statistics for Data Science

Markov Chains DS GA 1002 Probability and Statistics for Data Science

Markov Chains DS GA 1002 Probability and Statistics for Data Science http://www.cims.nyu.edu/~cfgranda/pages/DSGA1002_fall17 Carlos Fernandez-Granda Definition Recurrence Periodicity Convergence Markov-chain Monte Carlo Markov property The

1.33k views • 112 slides
Recitation 10/8 Mixture Models, PCA Slides borrowed from Prof. Seyoung Kim, Ryan Tibshirani.

Recitation 10/8 Mixture Models, PCA Slides borrowed from Prof. Seyoung Kim, Ryan Tibshirani.

Recitation 10/8 Mixture Models, PCA Slides borrowed from Prof. Seyoung Kim, Ryan Tibshirani. Thanks! Law of Total Probability Completely Observed Data Bishop Page 431 Since z uses a 1-of-K representation, we have What if we do not know

433 views • 17 slides

More recommend