decoding in latent conditional models
play

Decoding in Latent Conditional Models: A Practically Fast Solution - PowerPoint PPT Presentation

Sep 16, 2022 •237 likes •741 views

Latent dynamics workshop 2010 Decoding in Latent Conditional Models: A Practically Fast Solution for an NP-hard Problem Xu Sun ( ) University of Tokyo 2010.06.16 Outline Introduction Related Work & Motivations Our

  1. Latent dynamics workshop 2010 Decoding in Latent Conditional Models: A Practically Fast Solution for an NP-hard Problem Xu Sun ( 孫 栩 ) University of Tokyo 2010.06.16

  2. Outline • Introduction • Related Work & Motivations • Our proposals • Experiments • Conclusions 2

  3. Latent dynamics • Latent-structures (latent dynamics here) are important in information processing – Natural language processing – Data mining – Vision recognition • Modeling latent dynamics: Latent-dynamic conditional random fields (LDCRF) 3

  4. Latent dynamics • Latent-structures (latent dynamics here) are important in information processing Parsing: Learn refined grammars with latent info S NP VP . PRP VBD NP . He heard DT NN the voice 4

  5. Latent dynamics • Latent-structures (latent dynamics here) are important in information processing Parsing: Learn refined grammars with latent info S -x NP -x VP -x . -x PRP -x VBD -x NP -x . He heard DT -x NN -x the voice 5

  6. More common cases: linear-chain latent dynamics • The previous example is a tree-structure • More common cases could be linear-chain latent dynamics – Named entity recognition – Phrase segmentation – Word segmentation seg seg seg noSeg These are her flowers. Phrase segmentation [Sun+ COLING 08] 6

  7. A solution without latent annotation: Latent-dynamic CRFs A solution: Latent-dynamic conditional random fields (LDCRFs) [Morency+ CVPR 07] * No need to annotate latent info seg seg seg noSeg These are her flowers. Phrase segmentation [Sun+ COLING 08] 7

  8. Current problem & our target A solution: Latent-dynamic conditional random fields (LDCRFs) [Morency+ CVPR 07] * No need to annotate latent info Our target: Current problem: An almost exact Inference (decoding) is an inference method NP-hard problem. with fast speed. 8

  9. Outline • Introduction • Related Work & Motivations • Our proposals • Experiments • Conclusions 9

  10. Traditional methods • Traditional sequential labeling models – Hidden Markov Model (HMM) [Rabiner IEEE 89] – Maximum Entropy Model (MEM) [Ratnaparkhi EMNLP 96] – Conditional Random Fields (CRF) [Lafferty+ ICML 01] – Collins Perceptron [Collins EMNLP 02] Arguably the most accurate one. • Problem: not able to model latent structures  We will use it as one of the baseline. 10

  11. Conditional random field (CRF) [Lafferty+ ICML 01] y 1 y 2 y 3 y 4 y n x 1 x 2 x 3 x 4 x n   1       ( | , ) exp ( , ) P y x F y x  k k   Z ( , ) x k Problem: CRF does not model latent info 11

  12. Latent-Dynamic CRFs [Morency+ CVPR 07] y 1 y 2 y 3 y 4 y n Latent-dynamic h 1 h 2 h 3 h 4 h n CRFs x 1 x 2 x 3 x 4 x n y 1 y 2 y 3 y 4 y n Conditional random fields x 1 x 2 x 3 x 4 x n 12

  13. Latent-Dynamic CRFs [Morency+ CVPR 07] y 1 y 2 y 3 y 4 y n Latent-dynamic h 1 h 2 h 3 h 4 h n CRFs x 1 x 2 x 3 x 4 x n We can think (informally) it as y 1 y 2 y 3 y 4 y n “CRF + unsup . learning on latent info” Conditional random fields x 1 x 2 x 3 x 4 x n 13

  14. Latent-Dynamic CRFs [Morency+ CVPR 07]   1           exp ( , ) ( | , ) ( | , ) F h x P y x P h x  k k Z   ( , ) x     : H H h h h : h k j j y y j j Good performance reports * Outperforming HMM, MEMM, SVM, CRF, etc. * Syntactic parsing [Petrov+ NIPS 08] * Syntactic chunking [Sun+ COLING 08] * Vision object recognition [Morency+ CVPR 07; Quattoni+ PAMI 08] 14

  15. Outline • Introduction • Related Work & Motivations • Our proposals • Experiments • Conclusions 15

  16. Inference problem Recent fast solutions are only y 1 y 2 y 3 y 4 y n approximation methods: *Best Hidden Path [Matsuzaki+ ACL 05] h 1 h 2 h 3 h 4 h n *Best Marginal Path [Morency+ CVPR 07] x 1 x 2 x 3 x 4 x n • Prob: Exact inference (find the sequence with max probability) is NP-hard! – no fast solution existing 16

  17. Related work 1: Best hidden path (BHP) [Matsuzaki+ ACL 05] Seg-0 Seg-1 Seg-2 noSeg-0 noSeg-1 noSeg-2 These are her flowers . 17

  18. Related work 1: Best hidden path (BHP) [Matsuzaki+ ACL 05] Seg-0 Seg-1 Seg-2 noSeg-0 noSeg-1 noSeg-2 Result: These are her flowers . Seg Seg Seg NoSeg Seg 18

  19. Related work 2: Best marginal path (BMP) [Morency+ CVPR 07] Seg-0 Seg-1 Seg-2 noSeg-0 noSeg-1 noSeg-2 These are her flowers . 19

  20. Related work 2: Best marginal path (BMP) [Morency+ CVPR 07] Seg-0 0.1 0.1 0.4 0.0 0.1 0.6 0.1 0.3 0.1 0.1 Seg-1 Seg-2 0.2 0.5 0.0 0.1 0.5 noSeg-0 0.1 0.1 0.2 0.1 0.2 0.0 0.2 0.0 0.7 0.0 noSeg-1 noSeg-2 0.0 0.0 0.1 0.0 0.1 Result: These are her flowers . Seg Seg Seg NoSeg Seg 20

  21. Our target 1) Exact inference y 1 y 2 y 3 y 4 y n 2) Comparable speed to existing approximation methods h 1 h 2 h 3 h 4 h n x 1 x 2 x 3 x 4 x n • Prob: Exact inference (find the sequence with Challenge/Difficulty: Exact & practically-fast solution max probability) is NP-hard! on an NP-hard problem – no fast solution existing 21

  22. Outline • Introduction • Related Work & Motivations • Our proposals • Experiments • Conclusions 22

  23. Essential ideas [Sun+ EACL 09] • Fast & exact inference from a key observation – A key observation on prob. Distribution – Dynamic top-n search – Fast decision on optimal result from top-n candidates 23

  24. Key observation • Natural problems (e.g., NLP problems) are not completely ambiguous • Normally, Only a few result candidate are highly probable • Therefore, probability distribution on latent models could be sharp 24

  25. Key observation • Probability distribution on latent models is sharp These are her flowers . seg noSeg seg seg seg P = 0.2 0.8 seg seg seg noSeg seg P = 0.3 prob seg seg seg seg seg P = 0.2 seg seg noSeg noSeg seg P = 0.1 P = … seg noSeg seg noSeg seg P = … … … … … … 25

  26. Key observation • Probability distribution on latent models is • Challenge: the number of probable candidates are unknown & changing sharp • Need a method which can automatically These are her flowers . adapt itself on different cases seg noSeg seg seg seg P = 0.2 seg seg seg noSeg seg P = 0.3 seg seg seg seg seg P = 0.2 compare seg seg noSeg noSeg seg P = 0.1 P = … seg noSeg seg noSeg seg P(unknown) P = … … … … … … ≤ 0.2 26

  27. A demo on lattice Seg-0 Seg-1 Seg-2 noSeg-0 noSeg-1 noSeg-2 These are her flowers . 27

  28. (1) Admissible heuristics for A* search Seg-0 Seg-1 Seg-2 noSeg-0 noSeg-1 noSeg-2 These are her flowers . 28

  29. (1) Admissible heuristics for A* search Seg-0 h00 h10 h20 h30 h40 h01 h11 h21 h31 h41 Seg-1 Seg-2 h02 h12 h22 h32 h42 noSeg-0 h03 h13 h23 h33 h43 h04 h14 h24 h34 h44 noSeg-1 noSeg-2 h05 h15 h25 h35 h45 These are her flowers . Viterbi algo. (Right to left) 29

  30. (1) Admissible heuristics for A* search Seg-0 h00 h10 h20 h30 h40 h01 h11 h21 h31 h41 Seg-1 Seg-2 h02 h12 h22 h32 h42 noSeg-0 h03 h13 h23 h33 h43 h04 h14 h24 h34 h44 noSeg-1 noSeg-2 h05 h15 h25 h35 h45 These are her flowers . 30

  31. (2) Find 1st latent path h1: A* search Seg-0 h00 h10 h20 h30 h40 h01 h11 h21 h31 h41 Seg-1 Seg-2 h02 h12 h22 h32 h42 noSeg-0 h03 h13 h23 h33 h43 h04 h14 h24 h34 h44 noSeg-1 noSeg-2 h05 h15 h25 h35 h45 These are her flowers . 31

  32. (3) Get y1 & P(y1): Forward-Backward algo. Seg-0 h00 h10 h20 h30 h40 h01 h11 h21 h31 h41 Seg-1 Seg-2 h02 h12 h22 h32 h42 noSeg-0 h03 h13 h23 h33 h43 h04 h14 h24 h34 h44 noSeg-1 noSeg-2 h05 h15 h25 h35 h45 These are her flowers . 32

  33. (3) Get y1 & P(y1): Forward-Backward algo. Seg-0 h00 h10 h20 h30 h40 h01 h11 h21 h31 h41 Seg-1 Seg-2 h02 h12 h22 h32 h42 noSeg-0 h03 h13 h23 h33 h43 h04 h14 h24 h34 h44 noSeg-1 P(seg, noSeg, seg, seg, seg) = 0.2 noSeg-2 P(y*) = 0.2 h05 h15 h25 h35 h45 P(unknown) = 1 - 0.2 = 0.8 These are her flowers . P(y*) > P(unknown) ? 33

  34. (4) Find 2nd latent path h2: A* search Seg-0 h00 h10 h20 h30 h40 h01 h11 h21 h31 h41 Seg-1 Seg-2 h02 h12 h22 h32 h42 noSeg-0 h03 h13 h23 h33 h43 h04 h14 h24 h34 h44 noSeg-1 noSeg-2 h05 h15 h25 h35 h45 These are her flowers . 34

  35. (5) Get y2 & P(y2): Forward-backward algo. Seg-0 h00 h10 h20 h30 h40 h01 h11 h21 h31 h41 Seg-1 Seg-2 h02 h12 h22 h32 h42 noSeg-0 h03 h13 h23 h33 h43 h04 h14 h24 h34 h44 noSeg-1 noSeg-2 h05 h15 h25 h35 h45 These are her flowers . 35

Recommend

1 Latent variable models In the next section we will discuss latent variable models for

1 Latent variable models In the next section we will discuss latent variable models for

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 16 notes: Latent variable models and EM Tues, 4.10 1 Latent variable models In the next section we will discuss latent

271 views • 4 slides
Applications: Network Models Network Models I Latent factors reflect disease model Single

Applications: Network Models Network Models I Latent factors reflect disease model Single

Applications: Network Models Network Models I Latent factors reflect disease model Single underlying cause Requires conditional independence Symptoms/behaviors (likely) causally-related Network models provide an alternative

483 views • 20 slides
Latent Models: Sequence Models Beyond HMMs and Machine Translation Alignment CMSC 473/673 UMBC

Latent Models: Sequence Models Beyond HMMs and Machine Translation Alignment CMSC 473/673 UMBC

Latent Models: Sequence Models Beyond HMMs and Machine Translation Alignment CMSC 473/673 UMBC Outline Review: EM for HMMs Machine Translation Alignment Limited Sequence Models Maximum Entropy Markov Models Conditional Random Fields

1.2k views • 101 slides
Latent Models: Sequence Models Beyond HMMs and Machine Translation Alignment CMSC 473/673 UMBC

Latent Models: Sequence Models Beyond HMMs and Machine Translation Alignment CMSC 473/673 UMBC

Latent Models: Sequence Models Beyond HMMs and Machine Translation Alignment CMSC 473/673 UMBC Outline Review: EM for HMMs Machine Translation Alignment Limited Sequence Models Maximum Entropy Markov Models Conditional Random Fields

1.37k views • 100 slides
Latent Variable Models CS3750 Xiaoting Li 1 Out utli line Latent Variable Models

Latent Variable Models CS3750 Xiaoting Li 1 Out utli line Latent Variable Models

2/4/2020 Latent Variable Models CS3750 Xiaoting Li 1 Out utli line Latent Variable Models Expectation Maximization Algorithm (EM) Factor Analysis Probabilistic Principal Component Analysis Model Formulation Maximum

654 views • 18 slides
Part III: Latent Tree Models Le Song ICML 2012 Tutorial on Spectral Algorithms for Latent

Part III: Latent Tree Models Le Song ICML 2012 Tutorial on Spectral Algorithms for Latent

Spectral Algorithms for Latent Variable Models Part III: Latent Tree Models Le Song ICML 2012 Tutorial on Spectral Algorithms for Latent Variable Models, Edinburgh, UK Joint work with Mariya Ishteva, Ankur Parikh, Eric Xing, Byron Boots , Geoff

548 views • 40 slides
Conditional Heteroscedasticity (CH) So far, our models are for the conditional mean . For

Conditional Heteroscedasticity (CH) So far, our models are for the conditional mean . For

Conditional Heteroscedasticity (CH) So far, our models are for the conditional mean . For instance, the Gaussian AR(1) model + t y t = y t 1 may be written: , 2 Conditionally on y t 1 , y

411 views • 19 slides
Observation Decoding with Sensor Models: Recognition Tasks via Classical Planning Diego Aineto,

Observation Decoding with Sensor Models: Recognition Tasks via Classical Planning Diego Aineto,

Observation Decoding with Sensor Models: Recognition Tasks via Classical Planning Diego Aineto, Sergio Jimenez, Eva Onaindia October 16, 2020 Universitat Polit` ecnica de Val` encia 1 What is decoding? What is decoding? Decoding : finding

848 views • 26 slides
Examples and Implementations [Bayesian approach to Latent Class Models: Definition, Simulation,

Examples and Implementations [Bayesian approach to Latent Class Models: Definition, Simulation,

BIOSTAT 830 GRAPHICAL MODELS Problem Set 4 Case Study: Latent Class Models Note: 1. Due 11:59PM, December 21, 2016. 2. Electronic submission to your instructors email. 3. You are VERY MUCH encouraged to form teams to discuss proofs and

545 views • 3 slides
Learning Overcomplete Latent Variable Models through Tensor Methods Anima Anandkumar UC Irvine

Learning Overcomplete Latent Variable Models through Tensor Methods Anima Anandkumar UC Irvine

Learning Overcomplete Latent Variable Models through Tensor Methods Anima Anandkumar UC Irvine Joint work with Majid Janzamin Rong Ge UC Irvine Microsoft Research Latent Variable Probabilistic Models Latent (hidden) variable h R k ,

769 views • 47 slides
C unobserved construct (e.g. Disordered v. Non- Disordered) Latent classes are mutually

C unobserved construct (e.g. Disordered v. Non- Disordered) Latent classes are mutually

University of Ulster at Magee, Friday 15 th June 2012 Latent Transition Analysis Overview of latent class and Dr Oliver Perra latent transition models Institute of Child Care Research Queens University Belfast email : o.perra@qub.ac.uk

812 views • 38 slides
Provably Efficient RL via Latent State Decoding Akshay Alekh John Simon S. Du Krishnamurthy

Provably Efficient RL via Latent State Decoding Akshay Alekh John Simon S. Du Krishnamurthy

Provably Efficient RL via Latent State Decoding Akshay Alekh John Simon S. Du Krishnamurthy Nan Jiang Miro Dudk Agarwal Langford RL theory vs practice RL theory vs practice Theory Simple tabular environments No generalization RL

763 views • 26 slides
Chapter 4: Advanced IR Models 4.1 Probabilistic IR 4.2 Statistical Language Models (LMs) 4.3

Chapter 4: Advanced IR Models 4.1 Probabilistic IR 4.2 Statistical Language Models (LMs) 4.3

Chapter 4: Advanced IR Models 4.1 Probabilistic IR 4.2 Statistical Language Models (LMs) 4.3 Latent-Concept Models 4.3.1 Foundations from Linear Algebra 4.3.2 Latent Semantic Indexing (LSI) 4.3.3 Probabilistic Aspect Model (pLSI) 4-1 IRDM

330 views • 32 slides
Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges

Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges

Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges in Unsupervised Learning Learn a latent variable model without labeled examples. E.g. topic models, hidden Markov models, Gaussian mixtures,

763 views • 61 slides
Outline Latent Variable Generative Models Cooperative Vector Quantizer Model Model

Outline Latent Variable Generative Models Cooperative Vector Quantizer Model Model

2/4/2020 CS 3750 Advanced Machine Learning Latent Variable Generative Models II Ahmad Diab AHD23@cs.pitt.edu Feb 4, 2020 Based on slides of Professor Milos Hauskrecht Outline Latent Variable Generative Models Cooperative Vector

435 views • 20 slides
Models for Replicated Discrimination Tests: A Synthesis of Latent Class Mixture Models and

Models for Replicated Discrimination Tests: A Synthesis of Latent Class Mixture Models and

Models for Replicated Discrimination Tests Models for Replicated Discrimination Tests: A Synthesis of Latent Class Mixture Models and Generalized Linear Mixed Models Rune Haubo Bojesen Christensen & Per Bruun Brockhoff DTU Informatics

868 views • 21 slides
Predicting Structures: Conditional Models and Local Classifiers CS 6355: Structured Prediction 1

Predicting Structures: Conditional Models and Local Classifiers CS 6355: Structured Prediction 1

Predicting Structures: Conditional Models and Local Classifiers CS 6355: Structured Prediction 1 Outline Sequence models Hidden Markov models Inference with HMM Learning Conditional Models and Local Classifiers Global

1.1k views • 73 slides
Case Study: Approximate Bayesian Inference for Latent Gaussian Models by Using Integrated Nested

Case Study: Approximate Bayesian Inference for Latent Gaussian Models by Using Integrated Nested

Case Study: Approximate Bayesian Inference for Latent Gaussian Models by Using Integrated Nested Laplace Approximations BIOSTAT830: Graphical Models December 08, 2016 Introduction - INLA Inference for latent Gaussian Markov random field

605 views • 35 slides
Latent Class Models: The Latent Class Logit Model Accouting for unobserved heterogeneity:

Latent Class Models: The Latent Class Logit Model Accouting for unobserved heterogeneity:

Latent Class Models: The Latent Class Logit Model Accouting for unobserved heterogeneity: Accouting for unobserved heterogeneity: Random Parameters Parametric assumption: must specify the functional form of the mixing distribution (for

190 views • 8 slides
Models for Retrieval Models for Retrieval 1. HMM/N-gram-based 2. Latent Semantic Indexing (LSI)

Models for Retrieval Models for Retrieval 1. HMM/N-gram-based 2. Latent Semantic Indexing (LSI)

Models for Retrieval Models for Retrieval 1. HMM/N-gram-based 2. Latent Semantic Indexing (LSI) 3. Probabilistic Latent Semantic Analysis (PLSA) Berlin Chen 2003 References: 1. Berlin Chen et al., An HMM/N-gram-based Linguistic Processing

723 views • 47 slides
Maxent Models, Conditional Introduction Estimation, and Optimization In recent years there

Maxent Models, Conditional Introduction Estimation, and Optimization In recent years there

Maxent Models, Conditional Introduction Estimation, and Optimization In recent years there has been extensive use Without of conditional or discriminative probabilistic Magic models in NLP, IR, and Speech That is, Because: With Math!

323 views • 20 slides
Piecewise Bounds for Estimating Bernoulli- Logistic Latent Gaussian Models Mohammad Emtiyaz Khan

Piecewise Bounds for Estimating Bernoulli- Logistic Latent Gaussian Models Mohammad Emtiyaz Khan

Piecewise Bounds for Estimating Bernoulli- Logistic Latent Gaussian Models Mohammad Emtiyaz Khan Joint work with Benjamin Marlin, and Kevin Murphy University of British Columbia June 29, 2011 Piecewise Bounds for Binary Latent Gaussian Models

694 views • 34 slides
Pengtao Xie Joint work with Yuntian Deng and Eric Xing Carnegie Mellon University 1 Latent

Pengtao Xie Joint work with Yuntian Deng and Eric Xing Carnegie Mellon University 1 Latent

Mutual Angular Regularization of Latent Variable Models: Theory, Algorithm and Applications Pengtao Xie Joint work with Yuntian Deng and Eric Xing Carnegie Mellon University 1 Latent Variable Models (LVMs) Machine Learning Latent Variable

1.04k views • 47 slides
Decoding in SMT Nitin Madnani February 8, 2006 The Decoding Problem Search Inputs:

Decoding in SMT Nitin Madnani February 8, 2006 The Decoding Problem Search Inputs:

Decoding in SMT Nitin Madnani February 8, 2006 The Decoding Problem Search Inputs: Input string Bunch of statistical models A function to assign score to any translation Output: Best scoring translation

660 views • 34 slides

More recommend