latent lstm allocation
play

Latent LSTM Allocation Manzil Zaheer, Amr Ahmed and Alexander J - PowerPoint PPT Presentation

Oct 03, 2022 •375 likes •606 views

Latent LSTM Allocation Manzil Zaheer, Amr Ahmed and Alexander J Smola Presented by Akshay Budhkar & Krishnapriya Vishnubhotla March 3, 2018 Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya

  1. Latent LSTM Allocation Manzil Zaheer, Amr Ahmed and Alexander J Smola Presented by Akshay Budhkar & Krishnapriya Vishnubhotla March 3, 2018 Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 1 / 22

  2. Outline Introduction 1 Latent Dirichlet Allocation LSTMs Latent LSTM Allocation 2 Algorithm Inference Different Models Results 3 Conclusion 4 Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 2 / 22

  3. Latent Dirichlet Allocation Probabilistic graphical model Not sequential, but easily interpretable. Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 3 / 22

  4. LSTMs Good for modeling sequential data, preserves temporal aspect Too many parameters Hard to interpret Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 4 / 22

  5. Latent LSTM Allocation (LLA) - Algorithm Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 5 / 22

  6. Graphical model for LLA Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 6 / 22

  7. Marginal probability of observing a document is � p ( w d | LSTM , φ ) = p ( w d , z d | LSTM , φ ) z d (1) � � = p ( w d , t | z d , t ; φ ) p ( z d , t | z d , 1: t − 1 ; LSTM ) z d t Uses a K × H dense matrix and a V × K sparse matrix. Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 7 / 22

  8. Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 8 / 22

  9. Inference Stochastic Expectation Maximization is used to compute the posterior. The Evidence Lower Bound (ELBO) can be written as: � log p ( w d | LSTM , φ ) d (2) q ( z ) log p ( z d ; LSTM ) � t p ( w d , t | z d , t ; φ ) � � ≥ q ( z d ) z d d Conditional probability of topic at time step t is: p ( z d , t = k | w d , t , z d , 1: t − 1 | LSTM , φ ) (3) ∝ p ( z d , t = k | z d , 1: t ; LSTM ) p ( w d , t | z d , t = k ; φ ) And p ( w d , t | z d , t = k ; φ ) = φ w , k = n w , k + β (4) n k + V β Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 9 / 22

  10. Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 10 / 22

  11. Mathematical Intuition LDA � log p ( w ) = log p ( w t | model ) t (5) � � = log p ( w t | z t ) p ( z t | doc ) t z t LSTM � log p ( w ) = log p ( w t | w t − 1 , w t − 2 , . . . , w 1 ) (6) t LLA � � log p ( w ) = log p ( w t | z t ) p ( z t | z t − 1 , z t − 2 , . . . , z 1 ) (7) z 1: T t Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 11 / 22

  12. Different Models Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 12 / 22

  13. Perplexity vs. Number of topics (Wikipedia) Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 13 / 22

  14. Perplexity vs. Number of topics (User Search) Cannot use Char LLA, since URLs lack morphological structure Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 14 / 22

  15. LDA Ablation Study Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 15 / 22

  16. Interpreting Cleaner Topics Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 16 / 22

  17. Interpreting Factored Topics Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 17 / 22

  18. LSTM Topic Embedding (Wikipedia) Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 18 / 22

  19. Convergence Speed Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 19 / 22

  20. Effect of Joint vs. Independent Training Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 20 / 22

  21. Final Thoughts Pros Provides a knob for interpretability and accuracy Less number of parameters for a reasonable perplexity Cleaner factored topics Cons Did not compare to something like hierarchical LDA Can’t use Char LLA for every problem Perplexity is not a good measure of text generation accuracy Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 21 / 22

  22. Bibliography Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent dirichlet allocation. Journal of machine Learning research , 3(Jan):993–1022. Galley, M., Brockett, C., Sordoni, A., Ji, Y., Auli, M., Quirk, C., Mitchell, M., Gao, J., and Dolan, B. (2015). deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. arXiv preprint arXiv:1506.06863 . Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation , 9(8):1735–1780. Zaheer, M., Ahmed, A., and Smola, A. J. (2017). Latent lstm allocation: Joint clustering and non-linear dynamic modeling of sequence data. In International Conference on Machine Learning , pages 3967–3976. Manzil Zaheer, Amr Ahmed and Alexander J Smola (Presented by Akshay Budhkar & Krishnapriya Vishnubhotla) Latent LSTM Allocation March 3, 2018 22 / 22

Recommend

Sunday Homework 3 : an Diniohlet Allocation Model Latent Generative : Generative model

Sunday Homework 3 : an Diniohlet Allocation Model Latent Generative : Generative model

Dirichet Allocation Latent Lecture ( Part 11 2) : Jordan Yuan Andrea Scribes : , , Midterm This Wednesday : Due Sunday Homework 3 : an Diniohlet Allocation Model Latent Generative : Generative model : Pu Diricwetlw

166 views • 13 slides
Latent Dirichlet allocation IN TR OD U C TION TO TE XT AN ALYSIS IN R Marc Dotson Assistant

Latent Dirichlet allocation IN TR OD U C TION TO TE XT AN ALYSIS IN R Marc Dotson Assistant

Latent Dirichlet allocation IN TR OD U C TION TO TE XT AN ALYSIS IN R Marc Dotson Assistant Professor of Marketing Uns u per v ised learning Some more nat u ral lang u age processing ( NLP ) v ocab u lar y: Latent Dirichlet allocation ( LDA )

400 views • 28 slides
Latent Dirichlet Allocation Alberto Bie+ Trop dinformation

Latent Dirichlet Allocation Alberto Bie+ Trop dinformation

Latent Dirichlet Allocation Alberto Bie+ Trop dinformation Topic modeling Dcouvrir la structure thma5que cache dans chaque document dune archive

2.13k views • 28 slides
Linked Latent Dirichlet Allocation in Web Spam Filtering o 1 o 1 Istv an B r D avid

Linked Latent Dirichlet Allocation in Web Spam Filtering o 1 o 1 Istv an B r D avid

Linked Latent Dirichlet Allocation in Web Spam Filtering o 1 o 1 Istv an B r D avid Sikl osi J acint Szab ur 1 Andr as A. Bencz 1 Data Mining and Web Search Group Computer and Automation Institute Hungarian Academy

627 views • 14 slides
A Framework for Incorporating General Domain Knowledge into Latent Dirichlet Allocation using

A Framework for Incorporating General Domain Knowledge into Latent Dirichlet Allocation using

A Framework for Incorporating General Domain Knowledge into Latent Dirichlet Allocation using First-Order Logic David Andrzejewski 1 Xiaojin Zhu 2 Mark Craven 3 , 2 Benjamin Recht 2 1 Center for Applied Scientific Computing 2 Department of

1.13k views • 96 slides
14 Allocation Dirichlet Latent Lecture : Taheri Sara Scribes : Chu 4am Exam Man Tue

14 Allocation Dirichlet Latent Lecture : Taheri Sara Scribes : Chu 4am Exam Man Tue

14 Allocation Dirichlet Latent Lecture : Taheri Sara Scribes : Chu 4am Exam Man Tue 12 : Midterm Exam focus Open Format book understanding on : , IEM ) Everything 8 Lecture Content to up ' . ( VBEM 9810 lectures

440 views • 21 slides
Dirichet Allocation Latent Lecture 11 : Jordan Yuan Andrea Scribes : , , Exam Next

Dirichet Allocation Latent Lecture 11 : Jordan Yuan Andrea Scribes : , , Exam Next

Dirichet Allocation Latent Lecture 11 : Jordan Yuan Andrea Scribes : , , Exam Next Feb ) ( week Wed 28 : on ) ( Due Homework Man Fri next was : Midterm Exam focus Open Format book understanding on : , 1 Gandhy

353 views • 19 slides
Attention Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Encoder-decoder Models

Attention Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Encoder-decoder Models

CS11-747 Neural Networks for NLP Attention Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Encoder-decoder Models (Sutskever et al. 2014) Encoder kono eiga ga kirai </s> LSTM LSTM LSTM LSTM LSTM I hate this movie

823 views • 35 slides
Attention Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Encoder-decoder Models

Attention Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Encoder-decoder Models

CS11-747 Neural Networks for NLP Attention Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Encoder-decoder Models (Sutskever et al. 2014) Encoder kono eiga ga kirai </s> LSTM LSTM LSTM LSTM LSTM I hate this movie

614 views • 35 slides
E-LSTM: Efficient Inference of Sparse LSTM on Embedded Heterogeneous System Runbin Shi 1 Junjie

E-LSTM: Efficient Inference of Sparse LSTM on Embedded Heterogeneous System Runbin Shi 1 Junjie

E-LSTM: Efficient Inference of Sparse LSTM on Embedded Heterogeneous System Runbin Shi 1 Junjie Liu 1 Shuo Wang 2 Yun Liang 2 Hayden So 1 1 Department of Electrical and Electronic Engineering The University of Hong Kong 2 Center for

317 views • 30 slides
Measuring Topic Quality in Latent Dirichlet Allocation Sergey Nikolenko Sergei Koltsov Olessia

Measuring Topic Quality in Latent Dirichlet Allocation Sergey Nikolenko Sergei Koltsov Olessia

Topic modeling Measuring topic quality Measuring Topic Quality in Latent Dirichlet Allocation Sergey Nikolenko Sergei Koltsov Olessia Koltsova Steklov Institute of Mathematics at St. Petersburg Laboratory for Internet Studies, National

597 views • 26 slides
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting LSTM

Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting LSTM

Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting LSTM : VALSE 2016/03/23 Content Quick Review of Recurrent Neural Network

738 views • 27 slides
1 Latent variable models In the next section we will discuss latent variable models for

1 Latent variable models In the next section we will discuss latent variable models for

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 16 notes: Latent variable models and EM Tues, 4.10 1 Latent variable models In the next section we will discuss latent

271 views • 4 slides
C unobserved construct (e.g. Disordered v. Non- Disordered) Latent classes are mutually

C unobserved construct (e.g. Disordered v. Non- Disordered) Latent classes are mutually

University of Ulster at Magee, Friday 15 th June 2012 Latent Transition Analysis Overview of latent class and Dr Oliver Perra latent transition models Institute of Child Care Research Queens University Belfast email : o.perra@qub.ac.uk

812 views • 38 slides
LSTM: A Search Space Odyssey Klaus Greff, Rupesh K. Srivastava, Jan Koutn k, Bas R.

LSTM: A Search Space Odyssey Klaus Greff, Rupesh K. Srivastava, Jan Koutn k, Bas R.

LSTM: A Search Space Odyssey Klaus Greff, Rupesh K. Srivastava, Jan Koutn k, Bas R. Steunebrink, Ju rgen Schmidhuber, 2015. Presenter: Yijun Tian, Zhenyu Liu Abstract In this paper, the authors analyze performance of LSTM and its

758 views • 10 slides
AMMI Introduction to Deep Learning 11.2. LSTM and GRU Fran cois Fleuret

AMMI Introduction to Deep Learning 11.2. LSTM and GRU Fran cois Fleuret

AMMI Introduction to Deep Learning 11.2. LSTM and GRU Fran cois Fleuret https://fleuret.org/ammi-2018/ November 12, 2018 COLE POLYTECHNIQUE FDRALE DE LAUSANNE The Long-Short Term Memory unit (LSTM) by Hochreiter and Schmidhuber

310 views • 30 slides
Framewise Phoneme Classification with Bidirectional LSTM Networks Alex Graves and Jurgen

Framewise Phoneme Classification with Bidirectional LSTM Networks Alex Graves and Jurgen

Framewise Phoneme Classification with Bidirectional LSTM Networks Alex Graves and Jurgen Schmidhuber IJCNN 2005 conference proceedings Chase Duncan Overview 1. Bidirectional LSTM overview 2. What sort of tasks are they appropriate for? 3.

336 views • 15 slides
Part III: Latent Tree Models Le Song ICML 2012 Tutorial on Spectral Algorithms for Latent

Part III: Latent Tree Models Le Song ICML 2012 Tutorial on Spectral Algorithms for Latent

Spectral Algorithms for Latent Variable Models Part III: Latent Tree Models Le Song ICML 2012 Tutorial on Spectral Algorithms for Latent Variable Models, Edinburgh, UK Joint work with Mariya Ishteva, Ankur Parikh, Eric Xing, Byron Boots , Geoff

548 views • 40 slides
LSTM Neural Reordering Model for Statistical Machine Translation Yiming Cui, Shijin Wang,

LSTM Neural Reordering Model for Statistical Machine Translation Yiming Cui, Shijin Wang,

LSTM Neural Reordering Model for Statistical Machine Translation Yiming Cui, Shijin Wang, Jianfeng Li iFLYTEK Research June 14, 2016 O UTLINE Lexicalized Reordering Model LSTM Neural Reordering Model Experiments & Analyses

423 views • 20 slides
Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model CS330

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model CS330

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model CS330 Student Presentation Table of Contents Motivation & problem Method overview Experiments Takeaways Discussion (strengths

653 views • 24 slides
1 Calling Conventions Architecture Review: Caller- and Callee-Saved Registers Partition

1 Calling Conventions Architecture Review: Caller- and Callee-Saved Registers Partition

Prelude Register Allocation II The Sears Tower in Chicago was the worlds Last time tallest building from 1974 until 1998. Register allocation Global allocation via graph coloring Today More register allocation Fun facts

229 views • 5 slides
Demystifying Relational Latent Representations Sebastijan Dumani, Hendrik Blockeel DTAI, KU

Demystifying Relational Latent Representations Sebastijan Dumani, Hendrik Blockeel DTAI, KU

Demystifying Relational Latent Representations Sebastijan Dumani, Hendrik Blockeel DTAI, KU Leuven September 6, ILP 2017 1 Outline 2/24 1 Introduction 2 Understanding latent features 3 Properties of latent spaces Demystifying

281 views • 24 slides
Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representation Eliyahu

Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representation Eliyahu

Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representation Eliyahu Kiperwasser & Yoav Goldberg 2016 Presented by: Yaoyang Zhang Outline Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature

344 views • 22 slides
Latent Variable Models CS3750 Xiaoting Li 1 Out utli line Latent Variable Models

Latent Variable Models CS3750 Xiaoting Li 1 Out utli line Latent Variable Models

2/4/2020 Latent Variable Models CS3750 Xiaoting Li 1 Out utli line Latent Variable Models Expectation Maximization Algorithm (EM) Factor Analysis Probabilistic Principal Component Analysis Model Formulation Maximum

654 views • 18 slides

More recommend