recursive subtree composition in lstm based dependency
play

Recursive Subtree Composition in LSTM-Based Dependency Parsing - PowerPoint PPT Presentation

Jan 10, 2024 •321 likes •1.04k views

Recursive Subtree Composition in LSTM-Based Dependency Parsing Miryam de Lhoneux , Miguel Ballesteros and Joakim Nivre @mdlhx 4 June 2019 Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 1/22

  1. Recursive Subtree Composition in LSTM-Based Dependency Parsing Miryam de Lhoneux , Miguel Ballesteros and Joakim Nivre @mdlhx 4 June 2019 Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 1/22

  2. Overview 1 Tree vs. sequential LSTMs for parsing 2 BiLSTM parsing 3 Results 4 Conclusion Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 2/22

  3. Outline for section 1 1 Tree vs. sequential LSTMs for parsing 2 BiLSTM parsing 3 Results 4 Conclusion Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 3/22

  4. Recursive vs recurrent NNs Recurrent The largest city in Minnesota Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 4/22

  5. Recursive vs recurrent NNs Recurrent The largest city in Minnesota Recursive The largest city in Minnesota Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 4/22

  6. Recursive NN for Transition-Based Parsing the largest city left-arc left-arc Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 5/22

  7. Recursive NN for Transition-Based Parsing nmod the largest city left-arc left-arc Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 5/22

  8. Recursive NN for Transition-Based Parsing det nmod the largest city left-arc left-arc Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 5/22

  9. Recursive NN for Transition-Based Parsing det nmod the largest city left-arc left-arc Recursive composition function in the stack-LSTM parser (Dyer et al., 2015): Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 5/22

  10. Recursive NN for Transition-Based Parsing det nmod the largest city left-arc left-arc Recursive composition function in the stack-LSTM parser (Dyer et al., 2015): c ( h , d , r ) = tanh ( W [ h ; d ; r ] + b ) W [ city , largest , left − nmod ] Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 5/22

  11. Recursive NN for Transition-Based Parsing det nmod the largest city left-arc left-arc Recursive composition function in the stack-LSTM parser (Dyer et al., 2015): c ( h , d , r ) = tanh ( W [ h ; d ; r ] + b ) W [ city , largest , left − nmod ] h i = c ( h i − 1 , d , r ) W [ city , largest , left − nmod ] Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 5/22

  12. Recursive NN for Transition-Based Parsing det nmod the largest city left-arc left-arc Recursive composition function in the stack-LSTM parser (Dyer et al., 2015): c ( h , d , r ) = tanh ( W [ h ; d ; r ] + b ) W [ city , largest , left − nmod ] city 1 = c ( city 0 , largest , left − nmod ) Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 5/22

  13. Recursive NN for Transition-Based Parsing det nmod the largest city left-arc left-arc Recursive composition function in the stack-LSTM parser (Dyer et al., 2015): c ( h , d , r ) = tanh ( W [ h ; d ; r ] + b ) W [ city , largest , left − nmod ] city 1 = c ( city 0 , largest , left − nmod ) city 2 = c ( city 1 , the , left − det ) Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 5/22

  14. Recursive vs recurrent NNs Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 6/22

  15. Recursive vs recurrent NNs English PTB Chinese CTB Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 6/22

  16. Recursive vs recurrent NNs English PTB Chinese CTB S-LSTM without composition 89.6 83.6 Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 6/22

  17. Recursive vs recurrent NNs English PTB Chinese CTB S-LSTM without composition 89.6 83.6 S-LSTM with composition 90.9 85.7 Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 6/22

  18. Recursive vs recurrent NNs English PTB Chinese CTB S-LSTM without composition 89.6 83.6 S-LSTM with composition 90.9 85.7 BiLSTM 91.2 85.0 Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 6/22

  19. Recursive vs recurrent NNs English PTB Chinese CTB S-LSTM without composition 89.6 83.6 S-LSTM with composition 90.9 85.7 BiLSTM 91.2 85.0 Goals Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 6/22

  20. Recursive vs recurrent NNs English PTB Chinese CTB S-LSTM without composition 89.6 83.6 S-LSTM with composition 90.9 85.7 BiLSTM 91.2 85.0 Goals BiLSTM + composition? Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 6/22

  21. Recursive vs recurrent NNs English PTB Chinese CTB S-LSTM without composition 89.6 83.6 S-LSTM with composition 90.9 85.7 BiLSTM 91.2 85.0 Goals BiLSTM + composition? Examine composition in simple architecture Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 6/22

  22. Recursive vs recurrent NNs English PTB Chinese CTB S-LSTM without composition 89.6 83.6 S-LSTM with composition 90.9 85.7 BiLSTM 91.2 85.0 Goals BiLSTM + composition? Examine composition in simple architecture Typologically varied languages Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 6/22

  23. Outline for section 2 1 Tree vs. sequential LSTMs for parsing 2 BiLSTM parsing 3 Results 4 Conclusion Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 7/22

  24. Transition-Based Parsing using BiLSTMs Scoring: (score(LEFT−ARC),score(RIGHT−ARC),score(SHIFT),score(SWAP)) Configuration: STACK BUFFER MLP the brown fox jumped root Vfox Vjumped Vroot Vthe Vbrown concat concat concat concat concat LSTM b LSTM b LSTM b LSTM b LSTM b LSTM f LSTM f LSTM f LSTM f LSTM f X the X brown X X X fox jumped root b r o w n f o x j u m p e d t h e e(the) e(brown) e(fox) e(jumped) pe(the) pe(brown) pe(fox) pe(jumped) Kiperwasser and Goldberg (2016); de Lhoneux et al. (2017) Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 8/22

  25. Transition-Based Parsing using BiLSTMs Xthe Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 9/22

  26. Transition-Based Parsing using BiLSTMs Xthe e(the) Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 9/22

  27. Transition-Based Parsing using BiLSTMs Xthe e(the) pe(the) Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 9/22

  28. Transition-Based Parsing using BiLSTMs Xthe concat e(the) pe(the) Cf Cf Cf Cb Cb Cb t h e Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 9/22

  29. Transition-Based Parsing using BiLSTMs X the X brown X X X fox jumped root Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 10/22

  30. Transition-Based Parsing using BiLSTMs Vfox Vjumped Vroot Vthe Vbrown concat concat concat concat concat LSTM b LSTM b LSTM b LSTM b LSTM b LSTM f LSTM f LSTM f LSTM f LSTM f X the X brown X X X fox jumped root Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 10/22

  31. Transition-Based Parsing using BiLSTMs Vfox Vjumped Vroot Vthe Vbrown concat concat concat concat concat LSTM b LSTM b LSTM b LSTM b LSTM b LSTM f LSTM f LSTM f LSTM f LSTM f X the X brown X X X fox jumped root Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 10/22

  32. Transition-Based Parsing using BiLSTMs Vfox Vjumped Vroot Vthe Vbrown concat concat concat concat concat LSTM b LSTM b LSTM b LSTM b LSTM b LSTM f LSTM f LSTM f LSTM f LSTM f X the X brown X X X fox jumped root Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 10/22

  33. Transition-Based Parsing using BiLSTMs (score(LEFT−ARC),score(RIGHT−ARC),score(SHIFT),score(SWAP)) MLP Vfox Vjumped Vroot Vthe Vbrown concat concat concat concat concat LSTM b LSTM b LSTM b LSTM b LSTM b LSTM f LSTM f LSTM f LSTM f LSTM f X the X brown X X X fox jumped root Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 10/22

  34. Recursive Composition in the BiLSTM parser Vfox Vjumped Vroot Vthe Vbrown concat concat concat concat concat LSTM b LSTM b LSTM b LSTM b LSTM b LSTM f LSTM f LSTM f LSTM f LSTM f X the X brown X X X fox jumped root Miryam de Lhoneux, Miguel Ballesteros and Joakim Nivre Recursive Subtree Composition in Parsing 11/22

Recommend

Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representation Eliyahu

Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representation Eliyahu

Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representation Eliyahu Kiperwasser & Yoav Goldberg 2016 Presented by: Yaoyang Zhang Outline Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature

344 views • 22 slides
Source side Dependency Tree Reordering Models with Subtree Movements and Constraints Nguyen

Source side Dependency Tree Reordering Models with Subtree Movements and Constraints Nguyen

Source side Dependency Tree Reordering Models with Subtree Movements and Constraints Nguyen Bach, Qin Gao and Stephan Vogel Carnegie Mellon University 1 Overview We introduce source side dependency tree reordering models Inspired

869 views • 53 slides
Dependency Grammars Topological Dependency Trees: A Constraint-based Account of Linear

Dependency Grammars Topological Dependency Trees: A Constraint-based Account of Linear

Dependency Grammars Topological Dependency Trees: A Constraint-based Account of Linear Precedence Extensible Dependency Grammar: A New Methodology Sibel Ciddi NPFL106 - Linguistics 2013 Summer Framework Immediate dependency

656 views • 18 slides
USING SUBTREE SPLITS TO SPREAD DRUPAL INTO EVERYTHING D AV I D B A R R AT T ( D AV I D W B A R

USING SUBTREE SPLITS TO SPREAD DRUPAL INTO EVERYTHING D AV I D B A R R AT T ( D AV I D W B A R

USING SUBTREE SPLITS TO SPREAD DRUPAL INTO EVERYTHING D AV I D B A R R AT T ( D AV I D W B A R R AT T ) CLARIFICATION Drupal : Everything except the /core folder. Drupal core : Everything in the /core folder. DEPENDENCY MANAGEMENT npmjs.com

717 views • 53 slides
Dependency Parsing & Feature-based Parsing Ling571 Deep Processing Techniques for NLP

Dependency Parsing & Feature-based Parsing Ling571 Deep Processing Techniques for NLP

Dependency Parsing & Feature-based Parsing Ling571 Deep Processing Techniques for NLP February 2, 2015 Roadmap Dependency parsing Graph-based dependency parsing Maximum spanning tree CLE Algorithm Learning

899 views • 87 slides
Last class Dependency parsing and logistic regression Dependency parsing: a fully lexicalized

Last class Dependency parsing and logistic regression Dependency parsing: a fully lexicalized

Last class Dependency parsing and logistic regression Dependency parsing: a fully lexicalized formalism; tree edges connect words in the Shay Cohen sentence based on head-dependent relationships. (based on slides by Sharon Goldwater) a

359 views • 11 slides
Algorithms for NLP CS 11-711 Fall 2020 Lecture 14: Graph-based dependency parsing Emma

Algorithms for NLP CS 11-711 Fall 2020 Lecture 14: Graph-based dependency parsing Emma

Algorithms for NLP CS 11-711 Fall 2020 Lecture 14: Graph-based dependency parsing Emma Strubell Announcements No recitation on Friday (Tartan Community Day). 2 Dependency parsing 3 Dependency parsing Input buffer Transition-based

1.15k views • 77 slides
Advanced Dependency Parsing Joakim Nivre Uppsala University Linguistics and Philology Based on

Advanced Dependency Parsing Joakim Nivre Uppsala University Linguistics and Philology Based on

Advanced Dependency Parsing Joakim Nivre Uppsala University Linguistics and Philology Based on tutorials with Ryan McDonald Advanced Dependency Parsing 1(36) Introduction Plan for the Lecture 1. Graph-based vs. transition-based dependency

1.61k views • 109 slides
Marina Valeeva Outline 2 1. Introduction What is Dependency Parsing? What is a

Marina Valeeva Outline 2 1. Introduction What is Dependency Parsing? What is a

GRAPH-BASED DEPENDENCY PARSING Marina Valeeva Outline 2 1. Introduction What is Dependency Parsing? What is a Dependency Tree? Projectivity vs. Non-Projectivity 2. Graph-Based Dependency Parsing 3. Models Edge Based Factorization

572 views • 40 slides
End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning

End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning

End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning Authors: Jason D. Williams and Geoffrey Zweig Speaker: Hamidreza Shahidi Outline Introduction Model description Optimizing with

388 views • 28 slides
Attention Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Encoder-decoder Models

Attention Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Encoder-decoder Models

CS11-747 Neural Networks for NLP Attention Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Encoder-decoder Models (Sutskever et al. 2014) Encoder kono eiga ga kirai </s> LSTM LSTM LSTM LSTM LSTM I hate this movie

823 views • 35 slides
Attention Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Encoder-decoder Models

Attention Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Encoder-decoder Models

CS11-747 Neural Networks for NLP Attention Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Encoder-decoder Models (Sutskever et al. 2014) Encoder kono eiga ga kirai </s> LSTM LSTM LSTM LSTM LSTM I hate this movie

614 views • 35 slides
E-LSTM: Efficient Inference of Sparse LSTM on Embedded Heterogeneous System Runbin Shi 1 Junjie

E-LSTM: Efficient Inference of Sparse LSTM on Embedded Heterogeneous System Runbin Shi 1 Junjie

E-LSTM: Efficient Inference of Sparse LSTM on Embedded Heterogeneous System Runbin Shi 1 Junjie Liu 1 Shuo Wang 2 Yun Liang 2 Hayden So 1 1 Department of Electrical and Electronic Engineering The University of Hong Kong 2 Center for

317 views • 30 slides
Lesson 9 Recursive Types 2/19, 21 Chapters 20, 21 Recursive type Recursive type terms are

Lesson 9 Recursive Types 2/19, 21 Chapters 20, 21 Recursive type Recursive type terms are

Lesson 9: Recursive Types Lesson 9 Recursive Types 2/19, 21 Chapters 20, 21 Recursive type Recursive type terms are infinite, but regular For finite terms, use induction based on least fixed points For infinite terms/trees, we

232 views • 7 slides
LSTM M Based sed Ada dapt ptive ive Fil ilterin ering g for r Redu duced ced Pre redi

LSTM M Based sed Ada dapt ptive ive Fil ilterin ering g for r Redu duced ced Pre redi

LSTM M Based sed Ada dapt ptive ive Fil ilterin ering g for r Redu duced ced Pre redi diction ction Err rrors ors of Hype pers rspectral pectral Im Images ages Zhuocheng Jiang and W. David Pan Dept. of Electrical and Computer

988 views • 21 slides
Dependency parsing and logistic regression Shay Cohen (based on slides by Sharon Goldwater) 21

Dependency parsing and logistic regression Shay Cohen (based on slides by Sharon Goldwater) 21

Dependency parsing and logistic regression Shay Cohen (based on slides by Sharon Goldwater) 21 October 2019 Last class Dependency parsing: a fully lexicalized formalism; tree edges connect words in the sentence based on head-dependent

764 views • 43 slides
Recursive Methods Noter ch.2 Recursive Methods Recursive problem solution Problems

Recursive Methods Noter ch.2 Recursive Methods Recursive problem solution Problems

Recursive Methods Noter ch.2 Recursive Methods Recursive problem solution Problems that are naturally solved by recursion Examples: Recursive function: Fibonacci numbers Recursive graphics: Fractals Mutual recursion:

796 views • 61 slides
BST property: for each node v with key k , nodes in left subtree have keys k nodes in

BST property: for each node v with key k , nodes in left subtree have keys k nodes in

BST property: for each node v with key k , nodes in left subtree have keys k nodes in right subtree have keys k Have seen simple BST operations : Search Minimum Maximum Predecessor Successor Insert Delete

325 views • 18 slides
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting LSTM

Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting LSTM

Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting LSTM : VALSE 2016/03/23 Content Quick Review of Recurrent Neural Network

738 views • 27 slides
Chapter 3: Chapter 3: Primitive Recursive Primitive Recursive Functions Functions M. Farshi

Chapter 3: Chapter 3: Primitive Recursive Primitive Recursive Functions Functions M. Farshi

Computer Science Theory (Master Course) Yazd Univ. Chapter 3: Chapter 3: Primitive Recursive Primitive Recursive Functions Functions M. Farshi Composition- Mohammad Farshi Recursion Recursion Department of Computer Science PRC Classes

499 views • 26 slides
Graph-based Dependency Parsing (Chu-Liu-Edmonds algorithm) Sam Thomson (with thanks to Swabha

Graph-based Dependency Parsing (Chu-Liu-Edmonds algorithm) Sam Thomson (with thanks to Swabha

Graph-based Dependency Parsing (Chu-Liu-Edmonds algorithm) Sam Thomson (with thanks to Swabha Swayamdipta) University of Washington, CSE 490u February 22, 2017 Outline Dependency trees Three main approaches to parsing

648 views • 54 slides
Dependency Parsing II CMSC 470 Marine Carpuat Graph-based Dependency Parsing Slides credit:

Dependency Parsing II CMSC 470 Marine Carpuat Graph-based Dependency Parsing Slides credit:

Dependency Parsing II CMSC 470 Marine Carpuat Graph-based Dependency Parsing Slides credit: Joakim Nivre Directed Spanning Trees Dependency Parsing as Finding the Maximum Spanning Tree Views parsing as finding the best directed spanning

981 views • 61 slides
Graph Based Dependency Parsing Wei Qiu December 15, 2011 . . . . . . Graph Based

Graph Based Dependency Parsing Wei Qiu December 15, 2011 . . . . . . Graph Based

. . . . . . Graph Based Dependency Parsing Graph Based Dependency Parsing Wei Qiu December 15, 2011 . . . . . . Graph Based Dependency Parsing Outline Outline 1 Introduction to dependency parsing 2 Graph based dependency

935 views • 72 slides
Efficient Realization of Geometric Constraint Introduction Systems via Optimal Recursive

Efficient Realization of Geometric Constraint Introduction Systems via Optimal Recursive

Optimal Decomposition Meera Sitharam Efficient Realization of Geometric Constraint Introduction Systems via Optimal Recursive Decomposition Recursive De- composition and Cayley Convexification Main Result: Optimal DR-Plan Algorithm

711 views • 68 slides

More recommend