variational inference spin glasses and tap free energy
play

Variational inference, spin glasses, and TAP free energy Song Mei - PowerPoint PPT Presentation

Sep 12, 2023 •271 likes •995 views

Variational inference, spin glasses, and TAP free energy Song Mei Stanford University September 19, 2018 Joint work with Zhou Fan and Andrea Montanari Song Mei (Stanford University) TAP free energy September 19, 2018 1 / 29 General

  1. Variational inference, spin glasses, and TAP free energy Song Mei Stanford University September 19, 2018 Joint work with Zhou Fan and Andrea Montanari Song Mei (Stanford University) TAP free energy September 19, 2018 1 / 29

  2. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  3. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  4. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  5. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  6. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  7. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  8. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  9. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  10. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  11. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  12. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  13. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  14. Bayes estimation in Z ✷ synchronization ◮ Settings: Y ❂ ✭ ✕❂♥ ✮ xx T ✰ W ✿ x ✘ ❯♥✐❢✭ Z ♥ ✷ ✮ ❀ ◮ Estimate X ❂ xx T with loss: ❵ ✭ X ❀ ❝ X ✮ ❂ ✭✶ ❂♥ ✷ ✮ ❦ X � ❝ X ❦ ✷ ❋ ✿ ◮ For ✕ ❁ ✶ , estimation is impossible. ◮ For ✕ ❃ ✶ , estimation is possible and efficient, e.g., spectral estimator (Baik, Ben Arous, Peche phase transition). ◮ The optimal estimator is the Bayes estimator (also minimax estimator): X ❇❛②❡s ❂ E ❬ xx T ❥ Y ❪ ✿ ❝ Song Mei (Stanford University) TAP free energy September 19, 2018 4 / 29

  15. Bayes estimation in Z ✷ synchronization ◮ Settings: Y ❂ ✭ ✕❂♥ ✮ xx T ✰ W ✿ x ✘ ❯♥✐❢✭ Z ♥ ✷ ✮ ❀ ◮ Estimate X ❂ xx T with loss: ❵ ✭ X ❀ ❝ X ✮ ❂ ✭✶ ❂♥ ✷ ✮ ❦ X � ❝ X ❦ ✷ ❋ ✿ ◮ For ✕ ❁ ✶ , estimation is impossible. ◮ For ✕ ❃ ✶ , estimation is possible and efficient, e.g., spectral estimator (Baik, Ben Arous, Peche phase transition). ◮ The optimal estimator is the Bayes estimator (also minimax estimator): X ❇❛②❡s ❂ E ❬ xx T ❥ Y ❪ ✿ ❝ Song Mei (Stanford University) TAP free energy September 19, 2018 4 / 29

  16. Bayes estimation in Z ✷ synchronization ◮ Settings: Y ❂ ✭ ✕❂♥ ✮ xx T ✰ W ✿ x ✘ ❯♥✐❢✭ Z ♥ ✷ ✮ ❀ ◮ Estimate X ❂ xx T with loss: ❵ ✭ X ❀ ❝ X ✮ ❂ ✭✶ ❂♥ ✷ ✮ ❦ X � ❝ X ❦ ✷ ❋ ✿ ◮ For ✕ ❁ ✶ , estimation is impossible. ◮ For ✕ ❃ ✶ , estimation is possible and efficient, e.g., spectral estimator (Baik, Ben Arous, Peche phase transition). ◮ The optimal estimator is the Bayes estimator (also minimax estimator): X ❇❛②❡s ❂ E ❬ xx T ❥ Y ❪ ✿ ❝ Song Mei (Stanford University) TAP free energy September 19, 2018 4 / 29

Recommend

Variational Inference for GPs: Presenters Group1: Stochastic variational inference. Slides 2 - 28

Variational Inference for GPs: Presenters Group1: Stochastic variational inference. Slides 2 - 28

Variational Inference for GPs: Presenters Group1: Stochastic variational inference. Slides 2 - 28 Chaoqi Wang Sana Tonekaboni Will Grathwohl Group2: Variational inference for GPs. Slides 29 - 57 Trefor Evans Kingsley Chang Shems Saleh James

1.32k views • 98 slides
Spin glasses, concepts and analysis of susceptibility P.C.W. Holdsworth Ecole Normale Suprieure

Spin glasses, concepts and analysis of susceptibility P.C.W. Holdsworth Ecole Normale Suprieure

Spin glasses, concepts and analysis of susceptibility P.C.W. Holdsworth Ecole Normale Suprieure de Lyon 1. A (brief) review of spin glasses 2. Slow dynamics, aging phenomena 3. An illustration from a simple model My sources - Binder and Young

787 views • 53 slides
Recent advances on mean-field spin glasses Wei-Kuo Chen University of Minnesota Joint work with

Recent advances on mean-field spin glasses Wei-Kuo Chen University of Minnesota Joint work with

Recent advances on mean-field spin glasses Wei-Kuo Chen University of Minnesota Joint work with A. Auffinger, M. Handschy, D. Gamarnik, G. Lerman, D. Panchenko, M. Rahman, A. Sen, Q. Zeng July, 2018 1 / 17 What are spin glasses? 2 / 17

465 views • 34 slides
CS480/680 Machine Learning Lecture 11: February 11 th , 2020 Variational Inference Zahra

CS480/680 Machine Learning Lecture 11: February 11 th , 2020 Variational Inference Zahra

CS480/680 Machine Learning Lecture 11: February 11 th , 2020 Variational Inference Zahra Sheikhbahaee VARIATIONAL ALGORITHMS FOR APPROXIMATE BAYESIAN INFERENCE (Beal 2003, chapter 2) Variational Inference: A Review for Statisticians (Blei et al.

713 views • 28 slides
Spin glasses and Adiabatic Quantum Computing A.P. Young Talk at the Workshop on Theory and

Spin glasses and Adiabatic Quantum Computing A.P. Young Talk at the Workshop on Theory and

Spin glasses and Adiabatic Quantum Computing A.P. Young Talk at the Workshop on Theory and Practice of Adiabatic Quantum Computers and Quantum Simulation, ICTP, Trieste, August 22-26, 2016 Spin Glasses The problems studied on current quantum

537 views • 24 slides
Rejection Sampling Variational Inference Karan Grewal CSC2547 / STA4273 Overview Variational

Rejection Sampling Variational Inference Karan Grewal CSC2547 / STA4273 Overview Variational

Rejection Sampling Variational Inference Karan Grewal CSC2547 / STA4273 Overview Variational Inference Interested in computing posterior , but it is often intractable parametrize a variational family of distributions to

623 views • 12 slides
Deep Variational Inference FLARE Reading Group Presentation Wesley Tansey 9/28/2016 What is

Deep Variational Inference FLARE Reading Group Presentation Wesley Tansey 9/28/2016 What is

Deep Variational Inference FLARE Reading Group Presentation Wesley Tansey 9/28/2016 What is Variational Inference? What is Variational Inference? p*(x) Want to estimate some distribution, p*(x) What is Variational Inference?

684 views • 52 slides
Lecture Variational 13 Inference Panini Kaushal Scribes : - Margulies Smedeuranh Niklas

Lecture Variational 13 Inference Panini Kaushal Scribes : - Margulies Smedeuranh Niklas

Lecture Variational 13 Inference Panini Kaushal Scribes : - Margulies Smedeuranh Niklas Variational Inference Approximate by posterior Idea maximizing , bound lower Ptt variational fly ply ? ) a , , ) " 9) * =

709 views • 15 slides
Variational Inference for Bayes vMF Mixture Hanxiao Liu September 23, 2014 1 / 14 Variational

Variational Inference for Bayes vMF Mixture Hanxiao Liu September 23, 2014 1 / 14 Variational

Variational Inference for Bayes vMF Mixture Hanxiao Liu September 23, 2014 1 / 14 Variational Inference Review Lower bound the likelihood L ( ; X ) = E q log p ( X | ) log p ( X , Z | ) q ( Z ) = E q + E q log q (

310 views • 14 slides
On the replica approach for statistical mechanics of random system Giorgio Parisi Spin

On the replica approach for statistical mechanics of random system Giorgio Parisi Spin

On the replica approach for statistical mechanics of random system Giorgio Parisi Spin glasses in the infinite range limit (the Sherrington Kirkpatrick model). The computation of the free energy in the SK model (mean field

440 views • 19 slides
Spin Glasses and Information Processing Pavithran S Iyer Guide: Prof. V.V Sreedhar Chennai

Spin Glasses and Information Processing Pavithran S Iyer Guide: Prof. V.V Sreedhar Chennai

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin Glasses and Information Processing Pavithran S Iyer Guide: Prof. V.V Sreedhar Chennai Mathematical Institute April

748 views • 55 slides
Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family

Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family

Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations Wu Lin (UBC) June 11, 2019 Joint work with Mohammad Emtiyaz Khan (AIP, RIKEN) and Mark Schmidt (UBC) 1 / 7 Variational Inference (VI) VI

119 views • 7 slides
Variational Inference CMSC 691 UMBC Goal: Posterior Inference Hyperparameters Unknown

Variational Inference CMSC 691 UMBC Goal: Posterior Inference Hyperparameters Unknown

Approximate Inference: Variational Inference CMSC 691 UMBC Goal: Posterior Inference Hyperparameters Unknown parameters Data: p ( | ) Likelihood model: p( | ) (Some) Learning Techniques MAP/MLE: Point

1.09k views • 81 slides
You, the product! Me, my object. My glasses How I see/look without them My glasses = my normal

You, the product! Me, my object. My glasses How I see/look without them My glasses = my normal

You, the product! Me, my object. My glasses How I see/look without them My glasses = my normal vision. My glasses = beautiful finish of my appearance. My glasses = a big part of me . DISABILITY. ATTACHMENT. IDENTITY? Glasses or

677 views • 19 slides
Neural Variational Inference and Learning Andriy Mnih, Karol Gregor 22 June 2014 1 / 14

Neural Variational Inference and Learning Andriy Mnih, Karol Gregor 22 June 2014 1 / 14

Neural Variational Inference and Learning Andriy Mnih, Karol Gregor 22 June 2014 1 / 14 Introduction Training directed latent variable models is difficult because inference in them is intractable. Both MCMC and traditional variational

322 views • 14 slides
Lecture 14: Inference in Dirichlet Processes (Blei & Jordan, Variational inference for

Lecture 14: Inference in Dirichlet Processes (Blei & Jordan, Variational inference for

CS598JHM: Advanced NLP (Spring 2013) http://courses.engr.illinois.edu/cs598jhm/ Lecture 14: Inference in Dirichlet Processes (Blei & Jordan, Variational inference for Dirichlet Process Mixture models, Bayesian Analysis 2006) Julia

354 views • 21 slides
Spin glasses and proteins: an old tool for an old problem ? Olivier Rivoire L aboratoire I

Spin glasses and proteins: an old tool for an old problem ? Olivier Rivoire L aboratoire I

Spin glasses and proteins: an old tool for an old problem ? Olivier Rivoire L aboratoire I nterdisciplinaire de Phy sique Grenoble, France Cargse - 2014 Two old problems (1) (2) SEQUENCE STRUCTURE FUNCTION (1) Folding How do proteins

590 views • 17 slides
The free energy of the quantum Heisenberg ferromagnet: validity of the spin wave approximation

The free energy of the quantum Heisenberg ferromagnet: validity of the spin wave approximation

The free energy of the quantum Heisenberg ferromagnet: validity of the spin wave approximation Alessandro Giuliani Based on joint work with M. Correggi and R. Seiringer Warwick, March 19, 2014 Outline 1 Introduction: continuous symmetry

1.25k views • 95 slides
Glasses-Free 3D Photo Photography has gone through a long evolution, Today 3D photo is

Glasses-Free 3D Photo Photography has gone through a long evolution, Today 3D photo is

Glasses-Free 3D Photo Photography has gone through a long evolution, Today 3D photo is available from painters portraits and black and white prints to everyone! to a form of art in its own right. Glasses-free 3D photography was once

203 views • 7 slides
Low-lying excitations of quantum spin-glasses Koh Yang Wei International workshop on numerical

Low-lying excitations of quantum spin-glasses Koh Yang Wei International workshop on numerical

Low-lying excitations of quantum spin-glasses Koh Yang Wei International workshop on numerical methods and simulations for materials design and strongly correlated quantum matters March 24-25, 2017. Kobe, Japan. A Personal Motivation

841 views • 25 slides
Variational Inference and Learning Michael Gutmann Probabilistic Modelling and Reasoning

Variational Inference and Learning Michael Gutmann Probabilistic Modelling and Reasoning

Variational Inference and Learning Michael Gutmann Probabilistic Modelling and Reasoning (INFR11134) School of Informatics, University of Edinburgh Spring semester 2018 Recap Learning and inference often involves intractable integrals

786 views • 36 slides
161-17 Stochastic Variational Lecture Inference I Kaushal Panchi Scribes : Jay DeYoung

161-17 Stochastic Variational Lecture Inference I Kaushal Panchi Scribes : Jay DeYoung

161-17 Stochastic Variational Lecture Inference I Kaushal Panchi Scribes : Jay DeYoung Recap Vaniatianal Inference : Approximate Goal Posterior : 17,13 ) pH )p( 137 Generative Model 13 ) pc plx ,z : = , Approx

347 views • 18 slides
Meta-Amoruized Variational Inference and Learning Kristy Choi CS236: December 4th, 2019

Meta-Amoruized Variational Inference and Learning Kristy Choi CS236: December 4th, 2019

Meta-Amoruized Variational Inference and Learning Kristy Choi CS236: December 4th, 2019 Probabilistic Inference Probabilistic inference is a paruicular way of viewing the world: + = Observations Updated (posterior) Prior belief Typically

452 views • 28 slides
Variational Mean Field Variational Mean Field for Graphical Models for Graphical Models

Variational Mean Field Variational Mean Field for Graphical Models for Graphical Models

Variational Mean Field Variational Mean Field for Graphical Models for Graphical Models CS/CNS/EE 155 Baback Moghaddam Machine Learning Group baback @ jpl.nasa.gov Approximate Inference Approximate Inference Consider general UGs ( i.e.,

382 views • 37 slides

More recommend