dynamic programming for linear time incremental parsing
play

Dynamic Programming for Linear-Time Incremental Parsing Liang Huang - PowerPoint PPT Presentation

Dec 21, 2023 •316 likes •1.21k views

Dynamic Programming for Linear-Time Incremental Parsing Liang Huang Information Sciences Institute University of Southern California (Joint work with Kenji Sagae, USC/ICT) JHU CLSP Seminar September 14, 2010 Remembering Fred Jelinek

  1. Dynamic Programming for Linear-Time Incremental Parsing Liang Huang Information Sciences Institute University of Southern California (Joint work with Kenji Sagae, USC/ICT) JHU CLSP Seminar September 14, 2010

  2. Remembering Fred Jelinek (1932-2010) Prof. Jelinek hosted my visit and this talk on his last day. He was very supportive of this work, which is related to his work on structured language models, and I dedicate my work to his memory.

  3. Ambiguity and Incrementality • NLP is (almost) all about ambiguity resolution • human-beings resolve ambiguity incrementally DP for Incremental Parsing 3

  4. Ambiguity and Incrementality • NLP is (almost) all about ambiguity resolution • human-beings resolve ambiguity incrementally One morning in Africa, I shot an elephant in my pajamas; DP for Incremental Parsing 3

  5. Ambiguity and Incrementality • NLP is (almost) all about ambiguity resolution • human-beings resolve ambiguity incrementally One morning in Africa, I shot an elephant in my pajamas; how he got into my pajamas I’ll never know. DP for Incremental Parsing 3

  6. Ambiguity and Incrementality • NLP is (almost) all about ambiguity resolution • human-beings resolve ambiguity incrementally One morning in Africa, I shot an elephant in my pajamas; how he got into my pajamas I’ll never know. DP for Incremental Parsing 3

  7. Ambiguity and Incrementality • NLP is (almost) all about ambiguity resolution • human-beings resolve ambiguity incrementally One morning in Africa, I shot an elephant in my pajamas; how he got into my pajamas I’ll never know. DP for Incremental Parsing 3

  8. Ambiguities in Translation CS 562 - Intro 4

  9. Ambiguities in Translation CS 562 - Intro 4

  10. Ambiguities in Translation Google translate: carefully slide CS 562 - Intro 4

  11. Ambiguities in Translation Google translate: carefully slide CS 562 - Intro 4

  12. Ambiguities in Translation Google translate: carefully slide CS 562 - Intro 4

  13. If you are stolen... CS 562 - Intro 5

  14. If you are stolen... Google translate: Once the theft to the police CS 562 - Intro 5

  15. or even... CS 562 - Intro 6

  16. or even... clear evidence that NLP is used in real life! CS 562 - Intro 6

  17. Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing 7

  18. Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing 7

  19. Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing 7

  20. Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing 7

  21. But full DP is too slow... I feed cats nearby in the garden ... • full DP (like CKY) is too slow (cubic-time) • while human parsing is fast & incremental (linear-time) DP for Incremental Parsing 8

  22. But full DP is too slow... I feed cats nearby in the garden ... • full DP (like CKY) is too slow (cubic-time) • while human parsing is fast & incremental (linear-time) • how about incremental parsing then? • yes, but only with greedy search (accuracy suffers) • explores tiny fraction of trees (even w/ beam search) DP for Incremental Parsing 8

  23. But full DP is too slow... I feed cats nearby in the garden ... • full DP (like CKY) is too slow (cubic-time) • while human parsing is fast & incremental (linear-time) • how about incremental parsing then? • yes, but only with greedy search (accuracy suffers) • explores tiny fraction of trees (even w/ beam search) • can we combine the merits of both approaches? • a fast, incremental parser with dynamic programming? • explores exponentially many trees in linear-time? DP for Incremental Parsing 8

  24. Linear-Time Incremental DP incremental parsing greedy (e.g. shift-reduce) search ☹ (Nivre 04; Collins/Roark 04; ...) this work: full DP principled fast shift-reduce parsing (e.g. CKY) search with dynamic programming ☺ (Eisner 96; Collins 99; ...) ☹ ☺ fast slow (linear-time) (cubic-time) DP for Incremental Parsing 9

  25. Big Picture natural programming languages languages ☺ ☹ human psycholinguistics ☹ NLP ☺ computer compiler theory (LR, LALR, ...) DP for Incremental Parsing 10

  26. Big Picture natural programming languages languages ☺ ☹ human psycholinguistics ☹ NLP ☺ computer compiler theory (LR, LALR, ...) DP for Incremental Parsing 10

  27. Preview of the Results • very fast linear-time dynamic programming parser • best reported dependency accuracy on PTB/CTB • explores exponentially many trees (and outputs forest) 1.4 parsing time (secs) 1.2 1 0.8 0.6 0.4 0.2 0 0 10 20 30 40 50 60 70 sentence length DP for Incremental Parsing 11

  28. Preview of the Results • very fast linear-time dynamic programming parser • best reported dependency accuracy on PTB/CTB • explores exponentially many trees (and outputs forest) 1.4 y k e parsing time (secs) a l i e 1.2 n k r r a 1 e h B C MST 0.8 0.6 0.4 this work 0.2 0 0 10 20 30 40 50 60 70 sentence length DP for Incremental Parsing 11

  29. Preview of the Results • very fast linear-time dynamic programming parser • best reported dependency accuracy on PTB/CTB • explores exponentially many trees (and outputs forest) 1.4 y 10 10 DP: exponential k e parsing time (secs) a l i e 1.2 n k 10 8 r r a 1 e h B 10 6 C MST 0.8 0.6 10 4 0.4 10 2 this work 0.2 non-DP beam search 10 0 0 0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70 sentence length sentence length DP for Incremental Parsing 11

  30. Outline • Motivation • Incremental (Shift-Reduce) Parsing • Dynamic Programming for Incremental Parsing • Experiments DP for Incremental Parsing 12

  31. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - feed cats nearby ... cats nearby in ... cats nearby in ... nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing 13

  32. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift cats nearby in ... cats nearby in ... nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing 14

  33. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... cats nearby in ... nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing 15

  34. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing 16

  35. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I 4 shift feed cats nearby in the ... I nearby in the ... in the garden ... DP for Incremental Parsing 17

  36. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I 4 shift feed cats nearby in the ... I 5a r-reduce feed nearby in the ... cats I in the garden ... DP for Incremental Parsing 18

  37. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I 4 shift feed cats nearby in the ... I 5a r-reduce feed nearby in the ... cats I 5b shift in the garden ... feed cats nearby I DP for Incremental Parsing 19

Recommend

Dynamic Programming for Linear Time Incremental Parsing Liang Huang Information Sciences

Dynamic Programming for Linear Time Incremental Parsing Liang Huang Information Sciences

Dynamic Programming for Linear Time Incremental Parsing Liang Huang Information Sciences Institute University of Southern California Kenji Sagae Institute for Creative Technologies University of Southern California ACL 2010, Uppsala, Sweden,

1.1k views • 64 slides
Linear Time Constituency Parsing with RNNs and Dynamic Programming Juneki Hong 1 Liang Huang 1,2 1

Linear Time Constituency Parsing with RNNs and Dynamic Programming Juneki Hong 1 Liang Huang 1,2 1

Linear Time Constituency Parsing with RNNs and Dynamic Programming Juneki Hong 1 Liang Huang 1,2 1 Oregon State University 2 Baidu Research Silicon Valley AI Lab Span Parsing is SOTA in Constituency Parsing Cross+Huang 2016 introduced Span

691 views • 53 slides
TAG, Dynamic Programming, and the Perceptron for Efficent, Feature-Rich Parsing Xavier Carreras,

TAG, Dynamic Programming, and the Perceptron for Efficent, Feature-Rich Parsing Xavier Carreras,

TAG, Dynamic Programming, and the Perceptron for Efficent, Feature-Rich Parsing Xavier Carreras, Michael Collins and Terry Koo MIT CSAIL Discriminative Models for Parsing Structured Prediction methods like CRF or Perceptron train linear models

403 views • 27 slides
Computational Geometry Linear programming Exercise session #8: Incremental construction &

Computational Geometry Linear programming Exercise session #8: Incremental construction &

Computational Geometry Linear programming Exercise session #8: Incremental construction & Problem : maximize c T x s.t. Ax b Arrangements c = ( c 1 ,, c d ) T Linear Programming x = ( x 1 ,, x d ) T

521 views • 4 slides
Dynamic Programming 11.1 Overview Dynamic Programming is a powerful technique that allows one to

Dynamic Programming 11.1 Overview Dynamic Programming is a powerful technique that allows one to

Lecture 11 Dynamic Programming 11.1 Overview Dynamic Programming is a powerful technique that allows one to solve many different types of problems in time O ( n 2 ) or O ( n 3 ) for which a naive approach would take exponential time. In this

259 views • 7 slides
Linear Programming Linear Programming In a linear programming problem, there is a set of

Linear Programming Linear Programming In a linear programming problem, there is a set of

Linear Programming Linear Programming In a linear programming problem, there is a set of variables, and we want to assign real values to CISC5835, Algorithms for Big Data them so as to satisfy a set of linear equations and/or linear

414 views • 6 slides
Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs

Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs

Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs Semantic Parsing Dependency parsing models the syntactic structure between words in a sentence. Dependency Parsing vs Semantic Parsing

637 views • 31 slides
Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Two

Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Two

CS11-747 Neural Networks for NLP Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Two Types of Linguistic Structure Dependency: focus on relations between words ROOT I saw a girl with a

730 views • 38 slides
Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Two

Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Two

CS11-747 Neural Networks for NLP Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Two Types of Linguistic Structure Dependency: focus on relations between words ROOT I saw a girl with a

396 views • 37 slides
A Dynamic Near-Optimal Algorithm for Online Linear Programming Yinyu Ye Department of Management

A Dynamic Near-Optimal Algorithm for Online Linear Programming Yinyu Ye Department of Management

Online Linear Programming Main Results and Key Ideas Related and More Recent Work A Dynamic Near-Optimal Algorithm for Online Linear Programming Yinyu Ye Department of Management Science and Engineering and Institute of Computational and

662 views • 52 slides
Incremental Parsing in Bounded Memory William Schuler Department of Linguistics The Ohio State

Incremental Parsing in Bounded Memory William Schuler Department of Linguistics The Ohio State

Incremental Parsing in Bounded Memory William Schuler Department of Linguistics The Ohio State University September 16, 2010 William Schuler Incremental Parsing in Bounded Memory Motivation Goal: simple processing model, matches observations

1.27k views • 80 slides
Type-driven Incremental Semantic Parsing with Polymorphism Kai Zhao and Liang Huang City

Type-driven Incremental Semantic Parsing with Polymorphism Kai Zhao and Liang Huang City

Type-driven Incremental Semantic Parsing with Polymorphism Kai Zhao and Liang Huang City University of New York The Problem weakly supervised semantic fully supervised semantic parsing parsing aka. parsing from Q/A pairs

547 views • 44 slides
Space/time tradeoffs; dynamic programming; y g g transform and conquer 1. Space/time

Space/time tradeoffs; dynamic programming; y g g transform and conquer 1. Space/time

6. Space-time tradeoffs and dynamic programming D. Keil Analysis of Algorithms 1/11 Topic 6: Space/time tradeoffs; dynamic programming; y g g transform and conquer 1. Space/time tradeoffs 2. Dynamic programming 3 Example: sequence

498 views • 18 slides
Batch-Incremental vs. Instance-Incremental Learning in Dynamic and Evolving Data Jesse Read 1 ,

Batch-Incremental vs. Instance-Incremental Learning in Dynamic and Evolving Data Jesse Read 1 ,

Batch-Incremental vs. Instance-Incremental Learning in Dynamic and Evolving Data Jesse Read 1 , Albert Bifet 2 , Bernhard Pfahringer 2 , Geoff Holmes 2 1 Department of Signal Theory and Communications Universidad Carlos III Madrid, Spain 2

568 views • 19 slides
CKY Parsing Ling 571 Deep Processing Techniques for NLP January 12, 2011 Roadmap

CKY Parsing Ling 571 Deep Processing Techniques for NLP January 12, 2011 Roadmap

CKY Parsing Ling 571 Deep Processing Techniques for NLP January 12, 2011 Roadmap Motivation: Parsing (In) efficiency Dynamic Programming Cocke-Kasami-Younger Parsing Algorithm Chomsky Normal Form Conversion

689 views • 42 slides
Programming Languages: Parsing Onur Tolga S ehito glu Computer Engineering,METU 27 May

Programming Languages: Parsing Onur Tolga S ehito glu Computer Engineering,METU 27 May

Programming Languages:Parsing Programming Languages: Parsing Onur Tolga S ehito glu Computer Engineering,METU 27 May 2009 Programming Languages:Parsing Outline 1 Parsing Top-down Parsing Recursive Descent Parser LL Parsers

215 views • 8 slides
Dynamic Programming Has nothing to do with programming in the way we normally use that term

Dynamic Programming Has nothing to do with programming in the way we normally use that term

Dynamic Programming Has nothing to do with programming in the way we normally use that term Dynamic Programming More like TV programming (scheduling) Filling in a table Chapter 16 Algorithm design technique CPTR 318 1 2

462 views • 6 slides
Optimal Control and Dynamic Programming 4SC000 Q2 2017-2018 Duarte Antunes Outline

Optimal Control and Dynamic Programming 4SC000 Q2 2017-2018 Duarte Antunes Outline

Optimal Control and Dynamic Programming 4SC000 Q2 2017-2018 Duarte Antunes Outline Stochastic dynamic programming and linear quadratic control Output feedback linear quadratic control Separation principle Kalman filter LQG

740 views • 51 slides
Dynamic Programming Part 2 Algorithm Theory WS 2012/13 Fabian Kuhn Dynamic Programming

Dynamic Programming Part 2 Algorithm Theory WS 2012/13 Fabian Kuhn Dynamic Programming

Chapter 3 Dynamic Programming Part 2 Algorithm Theory WS 2012/13 Fabian Kuhn Dynamic Programming Memoization for increasing the efficiency of a recursive solution: Only the first time a sub problem is encountered, its solution is

656 views • 26 slides
RNA Secondary Structure Prediction allowed pairs: G-C A-U G-U example: transfer RNA (tRNA)

RNA Secondary Structure Prediction allowed pairs: G-C A-U G-U example: transfer RNA (tRNA)

LinearFold: Linear-Time Approximate RNA Folding by 5-to-3 dynamic programming and beam search x GCGGGAAUAGCUCAGUUGGUAGAGCACGACCUUGCCAAGGUCGGGGUCGCGAGUUCGAGUCUCGUUUCCCGCUCCA y

625 views • 44 slides
Efficient Computation of Extensions for Dynamic Abstract Argumentation Frameworks: An Incremental

Efficient Computation of Extensions for Dynamic Abstract Argumentation Frameworks: An Incremental

Introduction Incremental Computation Experiments Conclusions and future work Efficient Computation of Extensions for Dynamic Abstract Argumentation Frameworks: An Incremental Approach Gianvincenzo Alfano, Sergio Greco, Francesco Parisi {

541 views • 29 slides
Incremental Computation of Warranted Arguments in Dynamic Defeasible Argumentation: The Rule

Incremental Computation of Warranted Arguments in Dynamic Defeasible Argumentation: The Rule

Introduction Background Incremental Computation Implementation & Experiments Conclusions and future work Incremental Computation of Warranted Arguments in Dynamic Defeasible Argumentation: The Rule Addition Case Gianvincenzo Alfano 1 ,

440 views • 27 slides
Linear Programming Problems Linear programming problems come up in many applications. In a linear

Linear Programming Problems Linear programming problems come up in many applications. In a linear

Linear Programming Problems Linear programming problems come up in many applications. In a linear programming problem, we have a function, called the objective function , which depends linearly on a number of independent variables, and which we

404 views • 8 slides
CS137: Electronic Design Automation Day 2: March 31, 2004 Dual Objective Dynamic Programming

CS137: Electronic Design Automation Day 2: March 31, 2004 Dual Objective Dynamic Programming

CS137: Electronic Design Automation Day 2: March 31, 2004 Dual Objective Dynamic Programming CALTECH CS137 Spring2004 -- DeHon Today Cover and Place Linear GAMA Optimal Tree-based Area and Time covering for

597 views • 21 slides

More recommend