learning gaussian tree models analysis of error exponents
play

Learning Gaussian Tree Models: Analysis of Error Exponents and - PowerPoint PPT Presentation

Sep 29, 2022 •454 likes •1.07k views

Learning Gaussian Tree Models: Analysis of Error Exponents and Extremal Structures Vincent Tan Animashree Anandkumar, Alan Willsky Stochastic Systems Group, Laboratory for Information and Decision Systems, Massachusetts Institute of Technology

  1. Learning Gaussian Tree Models: Analysis of Error Exponents and Extremal Structures Vincent Tan Animashree Anandkumar, Alan Willsky Stochastic Systems Group, Laboratory for Information and Decision Systems, Massachusetts Institute of Technology Allerton Conference (Sep 30, 2009) 1/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 1 / 20

  2. Motivation Given a set of i.i.d. samples drawn from p , a Gaussian tree model. 2/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 2 / 20

  3. Motivation Given a set of i.i.d. samples drawn from p , a Gaussian tree model. Inferring structure of Phylogenetic Trees from observed data. Carlson et al. 2008, PLoS Comp. Bio. 2/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 2 / 20

  4. More motivation What is the exact rate of decay of the probability of error? 3/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 3 / 20

  5. More motivation What is the exact rate of decay of the probability of error? How do the structure and parameters of the model influence the error exponent (rate of decay)? 3/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 3 / 20

  6. More motivation What is the exact rate of decay of the probability of error? How do the structure and parameters of the model influence the error exponent (rate of decay)? What are extremal tree distributions for learning? 3/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 3 / 20

  7. More motivation What is the exact rate of decay of the probability of error? How do the structure and parameters of the model influence the error exponent (rate of decay)? What are extremal tree distributions for learning? Consistency is well established (Chow and Wagner 1973). Error Exponent is a quantitative measure of the “goodness” of learning. 3/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 3 / 20

  8. Main Contributions Provide the exact Rate of Decay for a given p . 1 4/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 4 / 20

  9. Main Contributions Provide the exact Rate of Decay for a given p . 1 Rate of decay ≈ SNR for learning. 2 4/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 4 / 20

  10. Main Contributions Provide the exact Rate of Decay for a given p . 1 Rate of decay ≈ SNR for learning. 2 Characterized the extremal trees structures for learning, i.e., stars 3 and Markov chains. Stars have the slowest rate. Chains have the fastest rate. 4/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 4 / 20

  11. Notation and Background p = N ( 0 , Σ ) : d -dimensional Gaussian tree model. 5/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 5 / 20

  12. Notation and Background p = N ( 0 , Σ ) : d -dimensional Gaussian tree model. Samples x n = { x 1 , x 2 , . . . , x n } drawn i.i.d. from p . 5/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 5 / 20

  13. Notation and Background p = N ( 0 , Σ ) : d -dimensional Gaussian tree model. Samples x n = { x 1 , x 2 , . . . , x n } drawn i.i.d. from p . p : Markov on T p = ( V , E p ) , a tree. p : Factorizes according to T p . 5/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 5 / 20

  14. Notation and Background p = N ( 0 , Σ ) : d -dimensional Gaussian tree model. Samples x n = { x 1 , x 2 , . . . , x n } drawn i.i.d. from p . p : Markov on T p = ( V , E p ) , a tree. p : Factorizes according to T p . p ( x ) = p 1 ( x 1 ) p 1 , 2 ( x 1 , x 2 ) p 1 , 3 ( x 1 , x 3 ) p 1 , 4 ( x 1 , x 4 ) , p 1 ( x 1 ) p 1 ( x 1 ) p 1 ( x 1 ) 5/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 5 / 20

  15. Notation and Background p = N ( 0 , Σ ) : d -dimensional Gaussian tree model. Samples x n = { x 1 , x 2 , . . . , x n } drawn i.i.d. from p . p : Markov on T p = ( V , E p ) , a tree. p : Factorizes according to T p .   ♠ ♣ ♣ ♣   p ( x ) = p 1 ( x 1 ) p 1 , 2 ( x 1 , x 2 ) p 1 , 3 ( x 1 , x 3 ) p 1 , 4 ( x 1 , x 4 ) ♣ ♠ 0 0 Σ − 1 =   ,   ♣ ♠ p 1 ( x 1 ) p 1 ( x 1 ) p 1 ( x 1 ) 0 0 ♣ ♠ 0 0 5/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 5 / 20

  16. Max-Likelihood Learning of Tree Distributions (Chow-Liu) p x n as the empirical distribution of x n , i.e., Denote � p = � p ( x ) := N ( x ; 0 , � � Σ ) where � Σ is the empirical covariance matrix of x n . 6/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 6 / 20

  17. Max-Likelihood Learning of Tree Distributions (Chow-Liu) p x n as the empirical distribution of x n , i.e., Denote � p = � p ( x ) := N ( x ; 0 , � � Σ ) where � Σ is the empirical covariance matrix of x n . � p e : Empirical on edge e . 6/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 6 / 20

  18. Max-Likelihood Learning of Tree Distributions (Chow-Liu) p x n as the empirical distribution of x n , i.e., Denote � p = � p ( x ) := N ( x ; 0 , � � Σ ) where � Σ is the empirical covariance matrix of x n . � p e : Empirical on edge e . Reduces to a max-weight spanning tree problem (Chow-Liu 1968) � � p e ) := � E CL ( x n ) = argmax I ( � p e ) . I ( � I ( X i ; X j ) . E q : q ∈ Trees e ∈E q 6/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 6 / 20

  19. Max-Likelihood Learning of Tree Distributions 7/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 7 / 20

  20. Max-Likelihood Learning of Tree Distributions True MIs { I ( p e ) } Max-weight spanning tree E p 7/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 7 / 20

  21. Max-Likelihood Learning of Tree Distributions True MIs { I ( p e ) } Max-weight spanning tree E p 7/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 7 / 20

  22. Max-Likelihood Learning of Tree Distributions True MIs { I ( p e ) } Max-weight spanning tree E p Max-weight spanning tree � p e ) } from x n E CL ( x n ) � = E p Empirical MIs { I ( � 7/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 7 / 20

  23. Problem Statement The estimated edge set is � E CL ( x n ) 8/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 8 / 20

  24. Problem Statement The estimated edge set is � E CL ( x n ) and the error event is � � � E CL ( x n ) � = E p . 8/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 8 / 20

  25. Problem Statement The estimated edge set is � E CL ( x n ) and the error event is � � � E CL ( x n ) � = E p . 8/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 8 / 20

  26. Problem Statement The estimated edge set is � E CL ( x n ) and the error event is � � � E CL ( x n ) � = E p . Find and analyze the error exponent K p : �� �� n →∞ − 1 � E CL ( x n ) � = E p K p := lim n log P . 8/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 8 / 20

  27. Problem Statement The estimated edge set is � E CL ( x n ) and the error event is � � � E CL ( x n ) � = E p . Find and analyze the error exponent K p : �� �� n →∞ − 1 � E CL ( x n ) � = E p K p := lim n log P . Alternatively, �� �� . � E CL ( x n ) � = E p = exp ( − nK p ) . P 8/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 8 / 20

  28. The Crossover Rate I 9/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 9 / 20

  29. The Crossover Rate I � V � Two pairs of nodes e , e ′ ∈ with distribution p e , e ′ , s.t. 2 I ( p e ) > I ( p e ′ ) . 9/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 9 / 20

  30. The Crossover Rate I � V � Two pairs of nodes e , e ′ ∈ with distribution p e , e ′ , s.t. 2 I ( p e ) > I ( p e ′ ) . Consider the crossover event: { I ( � p e ) ≤ I ( � p e ′ ) } . 9/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 9 / 20

  31. The Crossover Rate I � V � Two pairs of nodes e , e ′ ∈ with distribution p e , e ′ , s.t. 2 I ( p e ) > I ( p e ′ ) . Consider the crossover event: { I ( � p e ) ≤ I ( � p e ′ ) } . Definition: Crossover Rate n →∞ − 1 J e , e ′ := n log P ( { I ( � p e ) ≤ I ( � p e ′ ) } ) . lim 9/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 9 / 20

  32. The Crossover Rate I � V � Two pairs of nodes e , e ′ ∈ with distribution p e , e ′ , s.t. 2 I ( p e ) > I ( p e ′ ) . Consider the crossover event: { I ( � p e ) ≤ I ( � p e ′ ) } . Definition: Crossover Rate n →∞ − 1 J e , e ′ := n log P ( { I ( � p e ) ≤ I ( � p e ′ ) } ) . lim This event may potentially lead to an error in structure learning. Why? 9/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 9 / 20

  33. The Crossover Rate II Theorem The crossover rate is � � J e , e ′ = D ( q || p e , e ′ ) : I ( q e ′ ) = I ( q e ) inf . q ∈ Gaussians 10/20 Vincent Tan (MIT) Learning Gaussian Tree Models Allerton Conference 10 / 20

Recommend

Generalization Error MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON Elie Kawerk

Generalization Error MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON Elie Kawerk

Generalization Error MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON Elie Kawerk Data Scientist Supervised Learning - Under the Hood Supervised Learning: y = f ( x ) , f is unknown. MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON

601 views • 37 slides
Error Exponents, Weight and Magnetization Enumerators in LDPC Error Correcting Codes

Error Exponents, Weight and Magnetization Enumerators in LDPC Error Correcting Codes

Error Exponents, Weight and Magnetization Enumerators in LDPC Error Correcting Codes http://www.ncrg.aston.ac.uk/ D. Saad J. van Mourik, N. Skantzos, Y. Kabashima Aston University & TITECH 11 July 2003 Hayashibara Forum 11 July 2003

301 views • 19 slides
Probabilistic Graphical Models Probabilistic Graphical Models Gaussian Network Models Fall 2019

Probabilistic Graphical Models Probabilistic Graphical Models Gaussian Network Models Fall 2019

Probabilistic Graphical Models Probabilistic Graphical Models Gaussian Network Models Fall 2019 Siamak Ravanbakhsh Learning objectives Learning objectives multivariate Gaussian density: different parametrizations marginalization and

540 views • 39 slides
MLE 04-09-2019 For Gaussian and Mixture Gaussian Models Instructor - Sriram Ganapathy

MLE 04-09-2019 For Gaussian and Mixture Gaussian Models Instructor - Sriram Ganapathy

E9 205 Machine Learning for Signal Processing MLE 04-09-2019 For Gaussian and Mixture Gaussian Models Instructor - Sriram Ganapathy (sriramg@iisc.ac.in) Teaching Assistant - Prachi Singh (prachisingh@iisc.ac.in). Finding the parameters of the

390 views • 14 slides
Factor analysis & Exact inference for Gaussian networks Probabilistic Graphical Models

Factor analysis & Exact inference for Gaussian networks Probabilistic Graphical Models

Factor analysis & Exact inference for Gaussian networks Probabilistic Graphical Models Sharif University of Technology Spring 2017 Soleymani Multivariate Gaussian distribution 2 /2 1/2 exp{ 1 1 2

808 views • 43 slides
2 - 2 Laws of Exponents The laws of exponents are listed below:

2 - 2 Laws of Exponents The laws of exponents are listed below:

Exponents and Radicals 2 -1 Exponents If a is any real number and n is a positive integer, then the n th power of a is: exponent a n a n = a a a a base n factors The

296 views • 6 slides
Gaussian Mixture Models & EM CE-717: Machine Learning Sharif University of Technology M.

Gaussian Mixture Models & EM CE-717: Machine Learning Sharif University of Technology M.

Gaussian Mixture Models & EM CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Mixture Models: definition Mixture models: Linear supper-position of mixtures or components | =

1.04k views • 39 slides
Information Theory Lecture 9 Error Exponents The part on discrete channels of R.

Information Theory Lecture 9 Error Exponents The part on discrete channels of R.

Information Theory Lecture 9 Error Exponents The part on discrete channels of R. Gallager, A Simple Derivation of the Coding Theorem and Some Applications, IEEE Trans. on Inform. Theory , Jan. 1965 In addition some concepts

196 views • 15 slides
Error Exponents for Composite Hypothesis Testing of Markov Forest Distributions Vincent Tan,

Error Exponents for Composite Hypothesis Testing of Markov Forest Distributions Vincent Tan,

Error Exponents for Composite Hypothesis Testing of Markov Forest Distributions Vincent Tan, Anima Anandkumar, Alan S. Willsky Stochastic Systems Group, Laboratory for Information and Decision Systems, Massachusetts Institute of Technology

1.25k views • 58 slides
Learning unknown forces in nonlinear models with Gaussian processes and autoregressive flows Wil

Learning unknown forces in nonlinear models with Gaussian processes and autoregressive flows Wil

Learning unknown forces in nonlinear models with Gaussian processes and autoregressive flows Wil O C Ward w.ward@sheffield.ac.uk Department of Physics and Astronomy, The University of Sheffield GPSS Workshop: Structurally Constrained Gaussian

923 views • 89 slides
The Sender-Excited Secret Key Agreement Model: Capacity and Error Exponents Tzu-Han Chou, Vincent

The Sender-Excited Secret Key Agreement Model: Capacity and Error Exponents Tzu-Han Chou, Vincent

The Sender-Excited Secret Key Agreement Model: Capacity and Error Exponents Tzu-Han Chou, Vincent Y. F. Tan , Stark C. Draper Department of Electrical and Computer Engineering, University of Wisconsin-Madison Allerton (Sep 2011) Vincent Tan

915 views • 72 slides
Structured Gaussian Graphical Models Ashish Katiyar, Jessica Hoffmann, Constantine Caramanis The

Structured Gaussian Graphical Models Ashish Katiyar, Jessica Hoffmann, Constantine Caramanis The

Robust Estimation of Tree Structured Gaussian Graphical Models Ashish Katiyar, Jessica Hoffmann, Constantine Caramanis The University of Texas at Austin Undirected Probabilistic Graphical Models Represent conditional independence relations

404 views • 17 slides
Non-Parametric Models Review of last class: Decision Tree Learning dealing with the

Non-Parametric Models Review of last class: Decision Tree Learning dealing with the

Non-Parametric Models Review of last class: Decision Tree Learning dealing with the overlearning problem: pruning ensemble learning boosting Agenda Nearest neighbor models Finding nearest neighbors with kd trees

493 views • 27 slides
ECS 231 Lecture on Approximation and Error Analysis 1 / 9 Approximation and error analysis 1.

ECS 231 Lecture on Approximation and Error Analysis 1 / 9 Approximation and error analysis 1.

ECS 231 Lecture on Approximation and Error Analysis 1 / 9 Approximation and error analysis 1. Approximation and error ( not mistake! ) are the facts of life. 2. Sources of errors: measurement and data uncertainty modeling truncation

245 views • 9 slides
Runtime Error Analysis - A Machine Learning Perspective Praful Mangalath University of Colorado,

Runtime Error Analysis - A Machine Learning Perspective Praful Mangalath University of Colorado,

Runtime Error Analysis - A Machine Learning Perspective Praful Mangalath University of Colorado, Boulder Center for Computational Language and EducAtion Research (CLEAR) April 29th, 2009 Project Summary Runtime Error Analysis 5535

323 views • 18 slides
Compound channel with feedback: Opportunistic capacity and error exponents Aditya Mahajan and

Compound channel with feedback: Opportunistic capacity and error exponents Aditya Mahajan and

Compound channel with feedback: Opportunistic capacity and error exponents Aditya Mahajan and Sekhar Tatikonda Yale University March 17, 2010 CISS Compound channel Channel model | , =

791 views • 57 slides
& Exact inference for Gaussian networks Probabilistic Graphical Models Sharif University of

& Exact inference for Gaussian networks Probabilistic Graphical Models Sharif University of

Factor analysis & Exact inference for Gaussian networks Probabilistic Graphical Models Sharif University of Technology Spring 2016 Soleymani Multivariate Gaussian distribution 2 /2 1/2 exp{ 1 1 2

1.13k views • 43 slides
Probabilistic & Unsupervised Learning Beyond linear-Gaussian and Mixture models Maneesh

Probabilistic & Unsupervised Learning Beyond linear-Gaussian and Mixture models Maneesh

Probabilistic & Unsupervised Learning Beyond linear-Gaussian and Mixture models Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College London Term 1,

1.41k views • 124 slides
Probabilistic & Unsupervised Learning Beyond linear-Gaussian and Mixture models Maneesh

Probabilistic & Unsupervised Learning Beyond linear-Gaussian and Mixture models Maneesh

Probabilistic & Unsupervised Learning Beyond linear-Gaussian and Mixture models Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College London Term 1,

1.46k views • 120 slides
Probabilistic & Unsupervised Learning Beyond linear-Gaussian models and Mixtures Maneesh

Probabilistic & Unsupervised Learning Beyond linear-Gaussian models and Mixtures Maneesh

Probabilistic & Unsupervised Learning Beyond linear-Gaussian models and Mixtures Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College London Term 1,

1.64k views • 141 slides
Clustering: Models and Algorithms Shikui Tu 2019-03-07 1 Outline Gaussian Mixture Models

Clustering: Models and Algorithms Shikui Tu 2019-03-07 1 Outline Gaussian Mixture Models

Clustering: Models and Algorithms Shikui Tu 2019-03-07 1 Outline Gaussian Mixture Models (GMM) Expectation-Maximization (EM) for maximum likelihood Model selection, Bayesian learning 2 From distance to probability distance

383 views • 37 slides
Decision Tree Learning: Part 1 CS 760@UW-Madison Zoo of machine learning models Note: only a

Decision Tree Learning: Part 1 CS 760@UW-Madison Zoo of machine learning models Note: only a

Decision Tree Learning: Part 1 CS 760@UW-Madison Zoo of machine learning models Note: only a subset of ML methods Figure from scikit-learn.org Even a subarea has its own collection Figure from asimovinstitute.org The lectures organized

534 views • 34 slides
An Analysis of Graph Cut Size for Transductive Learning Steve Hanneke Machine Learning

An Analysis of Graph Cut Size for Transductive Learning Steve Hanneke Machine Learning

An Analysis of Graph Cut Size for Transductive Learning Steve Hanneke Machine Learning Department Carnegie Mellon University 1 Outline Transductive Learning with Graphs Error Bounds for Transductive Learning Error Bounds Based

691 views • 25 slides
The Learning Tree Workshop: The Learning Tree Workshop: Regulating Sensation Series on Learning

The Learning Tree Workshop: The Learning Tree Workshop: Regulating Sensation Series on Learning

The Learning Tree Workshop: The Learning Tree Workshop: Regulating Sensation Series on Learning Differences, Learning Challenges, and Learning Strengths Transcript by Stanley I. Greenspan M.D. Narration by Brian MacDonald Sensory Modulation

453 views • 16 slides

More recommend