top down tree long short term memory networks
play

Top-down Tree Long Short-Term Memory Networks Xingxing Zhang , Liang - PowerPoint PPT Presentation

Dec 03, 2022 •257 likes •746 views

Top-down Tree Long Short-Term Memory Networks Xingxing Zhang , Liang Lu, Mirella Lapata School of Informatics, University of Edinburgh 12th June, 2016 Zhang et al., 2016 Tree LSTM 12th June, 2016 1 / 18 Sequential Language Models n P (

  1. Top-down Tree Long Short-Term Memory Networks Xingxing Zhang , Liang Lu, Mirella Lapata School of Informatics, University of Edinburgh 12th June, 2016 Zhang et al., 2016 Tree LSTM 12th June, 2016 1 / 18

  2. Sequential Language Models n � P ( S = w 1 , w 2 , . . . , w n ) = P ( w i | w 1: i − 1 ) (1) i =1 State of the Art based on Long Short Term Memory Network Language Model (Hochreiter and Schmidhuber, 1997; Sundermeyer et al., 2012) Billion word benchmark results reported in Jozefowicz et al., (2016) Models PPL KN5 67.6 LSTM 30.6 LSTM+CNN INPUTS 30.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 2 / 18

  3. Will tree structures help LMs? Zhang et al., 2016 Tree LSTM 12th June, 2016 3 / 18

  4. Will tree structures help LMs? Probably yes LMs based on Constituency Parsing (Chelba and Jelinek, 2000; Roark, 2001; Charniak, 2001) LMs based on Dependency Parsing (Shen et al., 2008; Zhang, 2009; Sennrich, 2015) Zhang et al., 2016 Tree LSTM 12th June, 2016 3 / 18

  5. LSTMs + Dependency Trees = TreeLSTMs + Why? Sentence Length N v.s. Tree Height log ( N ) Zhang et al., 2016 Tree LSTM 12th June, 2016 4 / 18

  6. LSTMs + Dependency Trees = TreeLSTMs + Why? Sentence Length N v.s. Tree Height log ( N ) How? Top-down Generation Breadth-first search reminiscent of Eisner (1996) Zhang et al., 2016 Tree LSTM 12th June, 2016 4 / 18

  7. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  8. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  9. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  10. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  11. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  12. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  13. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  14. Tree LSTM n � P ( S ) = P ( w i | w 1: i − 1 ) (2) i =1 ⇓ � P ( S | T ) = P ( w |D ( w )) (3) w ∈ BFS( T ) \ root D ( w ) is the Dependency Path of w . D ( w ) is a generated sub-tree. Works on projective and unlabeled dependency trees. Zhang et al., 2016 Tree LSTM 12th June, 2016 6 / 18

  15. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  16. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  17. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  18. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  19. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  20. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  21. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  22. One Limitation of Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 8 / 18

  23. Left Dependent Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 9 / 18

  24. Left Dependent Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 9 / 18

  25. Left Dependent Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 9 / 18

  26. Left Dependent Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 9 / 18

  27. Experiments Zhang et al., 2016 Tree LSTM 12th June, 2016 10 / 18

  28. MSR Sentence Completion Challenge Training set: 49 million words (around 2 million sentences) development set: 4000 sentences test set: 1040 completion questions. Zhang et al., 2016 Tree LSTM 12th June, 2016 11 / 18

  29. Zhang et al., 2016 Tree LSTM 12th June, 2016 12 / 18

  30. Zhang et al., 2016 Tree LSTM 12th June, 2016 12 / 18

  31. Zhang et al., 2016 Tree LSTM 12th June, 2016 12 / 18

  32. Zhang et al., 2016 Tree LSTM 12th June, 2016 12 / 18

  33. Dependency Parsing Reranking Rerank 2nd Order MSTParser (McDonald and Pereira, 2006) We train TreeLSTM and LdTreeLSTM as language models. We only use words as input features; POS tags, dependency labels or composition features are not used. Zhang et al., 2016 Tree LSTM 12th June, 2016 13 / 18

  34. Dependency Parsing Reranking NN: Chen & Manning, 2014; S-LSTM: Dyer et al., 2015 Zhang et al., 2016 Tree LSTM 12th June, 2016 14 / 18

  35. Dependency Parsing Reranking NN: Chen & Manning, 2014; S-LSTM: Dyer et al., 2015 Zhang et al., 2016 Tree LSTM 12th June, 2016 14 / 18

  36. Dependency Parsing Reranking NN: Chen & Manning, 2014; S-LSTM: Dyer et al., 2015 Zhang et al., 2016 Tree LSTM 12th June, 2016 14 / 18

  37. Dependency Parsing Reranking NN: Chen & Manning, 2014; S-LSTM: Dyer et al., 2015 Zhang et al., 2016 Tree LSTM 12th June, 2016 14 / 18

  38. Tree Generation Four binary classifiers: Add Left? No! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  39. Tree Generation Four binary classifiers: Add Right? Yes! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  40. Tree Generation Four binary classifiers: Add Right? Yes! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  41. Tree Generation Four binary classifiers: Add Next Right? No! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  42. Tree Generation Four binary classifiers: Add Left? Yes! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  43. Tree Generation Four binary classifiers: Add Left? Yes! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  44. Tree Generation Four binary classifiers: Add Next Left? No! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  45. Tree Generation Four binary classifiers: Add Left? Add Right? Add Next Left? Add Next Right? Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  46. Tree Generation Zhang et al., 2016 Tree LSTM 12th June, 2016 16 / 18

  47. Conclusions Syntax can help language modeling. Predicting tree structures with Neural Networks is possible. Next Steps: Sequence to Tree Models Tree to Tree Models code available: https://github.com/XingxingZhang/td-treelstm Thanks & Questions? Zhang et al., 2016 Tree LSTM 12th June, 2016 17 / 18

Recommend

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks by Kai

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks by Kai

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks by Kai Sheng Tai, Richard Socher, Christopher D. Manning Daniel Perez tuvistavie CTO @ Claude Tech M2 @ The University of Tokyo October 2, 2017

546 views • 22 slides
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks Kai Sheng

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks Kai Sheng

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks Kai Sheng Tai , Richard Socher , and Christopher D. Manning Stanford University, MetaMind July 29, 2015 Distributed Word Representations

556 views • 36 slides
An Introduction to Neural Networks Long Short Term Memory (LSTM) and the Attention mechanism Ange

An Introduction to Neural Networks Long Short Term Memory (LSTM) and the Attention mechanism Ange

An Introduction to Neural Networks Long Short Term Memory (LSTM) and the Attention mechanism Ange Tato Universit du Qubec Montral Montreal, Canada Agenda v Recurrent Neural Network (RNN) v Long Short Term Memory (LSTM) v

506 views • 27 slides
Lecture 23: Recurrent Neural Networks, Long Short Term Memory Networks, Conntectionist Temporal

Lecture 23: Recurrent Neural Networks, Long Short Term Memory Networks, Conntectionist Temporal

Lecture 23: Recurrent Neural Networks, Long Short Term Memory Networks, Conntectionist Temporal Classifjcation, Decision Trees Instructor: Prof. Ganesh Ramakrishnan . . . . . . . . . . . . . . . . . . . . . . . . . . .

699 views • 31 slides
Memory Memory processing is the ability to: Acquire (Short term memory) Manipulate

Memory Memory processing is the ability to: Acquire (Short term memory) Manipulate

Memory Memory processing is the ability to: Acquire (Short term memory) Manipulate (Working memory) Retain (Long term memory) Memory Retrieve (Long term memory) processing A difficulty with any one or more of these skills

361 views • 6 slides
Lo Long-short term memory (L (LSTM) Jeong Min Lee CS3750, University of Pittsburgh Outline

Lo Long-short term memory (L (LSTM) Jeong Min Lee CS3750, University of Pittsburgh Outline

3/3/2020 Recurrent neural networks and Lo Long-short term memory (L (LSTM) Jeong Min Lee CS3750, University of Pittsburgh Outline RNN RNN Unfolding Computational Graph Backpropagation and weight update Explode / Vanishing

760 views • 35 slides
Sequential Data with Neural Networks Recurrent Neural Networks Sequential input / output Greg

Sequential Data with Neural Networks Recurrent Neural Networks Sequential input / output Greg

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Sequential Data with Neural Networks Recurrent Neural

303 views • 4 slides
Long/Short-Term Memory Mark Hasegawa-Johnson All content CC-SA 4.0 unless otherwise specified.

Long/Short-Term Memory Mark Hasegawa-Johnson All content CC-SA 4.0 unless otherwise specified.

Review Vanishing Gradient Example RNN Forget Gate LSTM Backprop Conclusion Long/Short-Term Memory Mark Hasegawa-Johnson All content CC-SA 4.0 unless otherwise specified. University of Illinois ECE 417: Multimedia Signal Processing, Fall

970 views • 55 slides
Chapter 3 - Cognition Types of human memory Short term memory and cognitive processes

Chapter 3 - Cognition Types of human memory Short term memory and cognitive processes

Chapter 3 - Cognition Types of human memory Short term memory and cognitive processes Long term memory Learning Unlearning Attention Cognitive load Working memory load Multitasking Measurement of

710 views • 32 slides
Boosting the deep multidimensional long short- term memory network for handwritten recognition

Boosting the deep multidimensional long short- term memory network for handwritten recognition

Boosting the deep multidimensional long short- term memory network for handwritten recognition systems Dayvid Castro 1 Byron L. D. Bezerra 1 Muser Valena 1 1 Polytechnic School of Pernambuco University of Pernambuco Recife, Brazil The 16th

693 views • 32 slides
Class 15 - Long Short-Term Memory (LSTM) Class 15 - Long Short-Term Memory (LSTM) Study materials

Class 15 - Long Short-Term Memory (LSTM) Class 15 - Long Short-Term Memory (LSTM) Study materials

Class 15 - Long Short-Term Memory (LSTM) Class 15 - Long Short-Term Memory (LSTM) Study materials Study materials http://colah.github.io/posts/2015-08-Understanding-LSTMs/ (http://colah.github.io/posts/2015-08-Understanding-LSTMs/)

641 views • 23 slides
the human sensory, short-term , long-term Information processed and applied

the human sensory, short-term , long-term Information processed and applied

the human Information i/ o chapter 1 visual, auditory, haptic, m ovem ent Information stored in memory the human sensory, short-term , long-term Information processed and applied reasoning, problem solving,

431 views • 7 slides
Large Vocabulary Continuous Speech Recognition with Long Short-Term Recurrent Networks Ha sim

Large Vocabulary Continuous Speech Recognition with Long Short-Term Recurrent Networks Ha sim

Large Vocabulary Continuous Speech Recognition with Long Short-Term Recurrent Networks Ha sim Sak, Andrew Senior, Oriol Vinyals, Georg Heigold, Erik McDermott, Rajat Monga, Mark Mao, Fran coise Beaufays andrewsenior/hasim@google.com See

578 views • 37 slides
Transition-Based Dependency Parsing with Stack Long Short-Term Memory Chris Dyer, Miguel

Transition-Based Dependency Parsing with Stack Long Short-Term Memory Chris Dyer, Miguel

Transition-Based Dependency Parsing with Stack Long Short-Term Memory Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, Noah A. Smith Association for Computational Linguistics (ACL), 2015 Presented By: Lavisha Aggarwal (lavisha2)

253 views • 23 slides
Long-Term Memory Introduction STM versus LTM Episodic Memory Semantic Memory

Long-Term Memory Introduction STM versus LTM Episodic Memory Semantic Memory

3/16/17 Long-Term Memory Introduction STM versus LTM Episodic Memory Semantic Memory Procedural Memory Encoding in Long-Term Memory Depth of Processing (or Levels of Processing) Self Reference Effect

863 views • 19 slides
Joint Learning of Speech-Driven Facial Motion with Bidirectional Long-Short Term Memory N AJMEH S

Joint Learning of Speech-Driven Facial Motion with Bidirectional Long-Short Term Memory N AJMEH S

Joint Learning of Speech-Driven Facial Motion with Bidirectional Long-Short Term Memory N AJMEH S ADOUGHI AND C ARLOS B USSO Multimodal Signal Processing (MSP) lab The University of Texas at Dallas Erik Jonsson School of Engineering and Computer

426 views • 26 slides
Discrimination between genuine versus fake emotion using long-short term memory with parametric

Discrimination between genuine versus fake emotion using long-short term memory with parametric

Winner nner at at ICCV CCV 2017 2017 Re Real ve versus Fa Fake expr pressed essed emotio ions ch challe llenge Discrimination between genuine versus fake emotion using long-short term memory with parametric bias and facial landmarks Xuan-Phung

479 views • 15 slides
The Short- and Long-Term Economic Outlook Short-Term Update Jobs: PDX v US 3 3 Job Growth by

The Short- and Long-Term Economic Outlook Short-Term Update Jobs: PDX v US 3 3 Job Growth by

The Short- and Long-Term Economic Outlook Short-Term Update Jobs: PDX v US 3 3 Job Growth by Sector 2013Q2 v 2014Q2 12% 10% 8% 6% 4% 2% 0% -2% 4 4 Source: Brookings Metropolitan Monitor Unemployment 5 5 Median Household Income,

527 views • 15 slides
Towards Greater International Transparency of Clinical Trials Short Term Efforts for Long Term

Towards Greater International Transparency of Clinical Trials Short Term Efforts for Long Term

KKJ Karmela Krleza-Jeric,MD, M.Sc., D.Sc. CIHR - IRSC Towards Greater International Transparency of Clinical Trials Short Term Efforts for Long Term Benefits: Short Term Efforts for Long Term Benefits: CIHR trial policy 2010 KKJ 1 Outline

457 views • 43 slides
Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep Learning textbook Ch. 10 Recurrent Neural Networks Long

667 views • 18 slides
Return To Office Strategy Short-Term Strategy Mid-Term Strategy - Remote Work Long-Term

Return To Office Strategy Short-Term Strategy Mid-Term Strategy - Remote Work Long-Term

People Place Technology Return To Office Strategy Short-Term Strategy Mid-Term Strategy - Remote Work Long-Term Strategy Enhancement and - Temporary Changes Enablement - Reduce Seating Capacity - Priority-Driven Long- - Leverage

251 views • 4 slides
term Memory? Kowialiewski Benjamin Gorin Simon Majerus Steve Introduction Verbal short-term

term Memory? Kowialiewski Benjamin Gorin Simon Majerus Steve Introduction Verbal short-term

Does Semantic Knowledge Influence Serial Order Processing In Short- term Memory? Kowialiewski Benjamin Gorin Simon Majerus Steve Introduction Verbal short-term memory: temporary storage of verbal information Recall Introduction Item

518 views • 22 slides
Operating Systems CPU Scheduling ENCE 360 Operating System Schedulers Short-Term Long-Term

Operating Systems CPU Scheduling ENCE 360 Operating System Schedulers Short-Term Long-Term

Operating Systems CPU Scheduling ENCE 360 Operating System Schedulers Short-Term Long-Term (batch) Medium-Term Which Ready process to Which requested process Which Ready process to Running ? into Ready Queue? memory?

501 views • 27 slides
Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long

Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long

Deep Learning: multi-layer neural networks Recurrent Neural Networks: sequence data Long Short-Term Memory: LSTM

366 views • 18 slides

More recommend