scene graph parsing as dependency parsing
play

Scene Graph Parsing as Dependency Parsing Author: Yu-Siang Wang , - PowerPoint PPT Presentation

Sep 14, 2023 •38 likes •728 views

Scene Graph Parsing as Dependency Parsing Author: Yu-Siang Wang , Chenxi Liu, Xiaohui Zeng, Alan Yuille Conference: North American Chapter of the Association for Computational Linguistics, 2018 1 Outline Introduction Method

  1. Scene Graph Parsing as Dependency Parsing Author: Yu-Siang Wang , Chenxi Liu, Xiaohui Zeng, Alan Yuille Conference: North American Chapter of the Association for Computational Linguistics, 2018 1

  2. Outline Introduction ● Method ● Experiments ● Conclusion ● 2

  3. Introduction Introduction ● Method ● Experiments ● Conclusion ● 3

  4. Introduction Many multimodal tasks fit into this picture ● A young boy wearing Intermediate black shirt is in front Representation of a goal 4

  5. Image Generation from Text A young boy wearing Intermediate black shirt is in front Representation of a goal 5

  6. Image Captioning A young boy wearing Intermediate black shirt is in front Representation of a goal 6

  7. Image Retrieval A young boy wearing Intermediate black shirt is in front Representation of a goal 7

  8. Neural Network Embedding Neural network embeddings often used as the intermediate representation ● Pro: easy training; similarity with cosine distance ● Con: no explicit structure; no easy interpretability ● A young boy wearing 1.2, -1.3, 4.6, …, -3.7 black shirt is in front 2.3, -2.2, -2.6,…, 5.3 of a goal 3.8, -7.4,-5.9 …, -3.2 8

  9. Scene Graph More recently, people start exploring a more explainable representation ● Has 3 types of nodes: object, attribute, relation ● A young boy wearing black shirt is in front of a goal 9 Ref: Johnson et al., Image Retrieval Using Scene Graph, CVPR 2015

  10. Our Goal Parsing from sentence to scene graph (i.e., scene graph parsing) ● A young boy wearing black shirt is in front of a goal 10

  11. Previous Work: Separated Two-stage Standard Heuristic rules; Dependency Simple classifier Parsing a young boy wearing Ref: Anderson et al., SPICE: Semantic Propositional Image Caption Evaluation, ECCV 2016 black shirt is in front of a man 11

  12. Our Work: End-to-end One-stage a young boy wearing Ref: Anderson et al., SPICE: Semantic Propositional Image Caption Evaluation, ECCV 2016 black shirt is Customized in front of a Dependency man Equivalent Parsing 12

  13. Method Introduction ● Method ● Experiments ● Conclusion ● 13

  14. Scene Graph Node-centric View 14

  15. Pushing Labels from Node to Arc Node-centric View Edge-centric View Object node to attribute node Object node to relation node Relation node to object node Equivalent Different colors are different arc labels ● Under the edge-centric view, scene graphs begin to look like dependency parses ● 15

  16. Review of Dependency Parsing 3. Pick a System (e.g. 2. Define a Label Space! 1. Get a Corpus! Arc-Hybrid) and its Actions! NSUBJ LEFT NMOD RIGHT CASE SHIFT DET ... ... 16

  17. How we do Scene Graph Parsing? 3. Pick a System (e.g. 2. Define a Label Space! 1. Get a Corpus! Arc-Hybrid) and its Actions! ? ? ? 17

  18. How we do Scene Graph Parsing? 3. Pick a System (e.g. 2. Define a Label Space! 1. Get a Corpus! Arc-Hybrid) and its Actions! ? ? ? 18

  19. Visual Genome In Visual Genome, every image is annotated with 30 regions on average ● Every region is annotated with a (region) description and a (region) scene graph ● kid sit on ground A young boy wearing black A kid is sitting on the ground shirt is in front of a goal 19 Ref: Krishna et al., Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations, IJCV 2017

  20. Alignment Strategy To mimic a dependency parsing training corpus, we need alignment between ● nodes in the scene graph and words in the sentence We propose a two-round alignment strategy: ● Within each round, object, attribute, relation nodes are aligned in this order ○ First round is more “conservative” (word-by-word match) ○ Second round is more “aggressive” (synonyms match) ○ 20

  21. Alignments made in Round 1 a young boy wearing black shirt is in front of a goal ROOT 21 21

  22. Alignments made in Round 2 a young boy wearing black shirt is in front of a goal ROOT 22 22

  23. Alignment Result a young boy wearing black shirt is in front of a goal ROOT 23

  24. How we do Scene Graph Parsing? 3. Pick a System (e.g. 2. Define a Label Space! 1. Get a Corpus! Arc-Hybrid) and its Actions! ? ? 24

  25. Regular Labels 1. ATTR 2. SUBJ 3. OBJT Object to Attribute Object to Relation Relation to Object SUBJ OBJT SUBJ ATTR OBJT ATTR a young boy wearing black shirt is in front of a goal ROOT 25

  26. Auxiliary Labels 1. ATTR 2. SUBJ 3. OBJT 4. CONT 5. BEGN Object to Attribute Object to Relation Relation to Object Phrase ROOT to Obj without Head BEGN SUBJ OBJT SUBJ ATTR OBJT CONT CONT ATTR a young boy wearing black shirt is in front of a goal ROOT 26

  27. How we do Scene Graph Parsing? 3. Pick a System (e.g. 2. Define a Label Space! 1. Get a Corpus! Arc-Hybrid) and its Actions! ? BEGN SUBJ OBJT CONT ATTR 27

  28. Transition-Based Arc-Hybrid System Ref: Kuhlmann et al., Dynamic programming algorithms for transition-based dependency parsers, ACL 2011 28

  29. Transition-Based Arc-Hybrid System Ref: Kuhlmann et al., Dynamic programming algorithms for transition-based dependency parsers, ACL 2011 29

  30. Augmented Arc-Hybrid We augment Arc-Hybrid with one more action that is REDUCE ● This is because we don’t require every word to have a head (e.g. “is”) ● 30

  31. How we do Scene Graph Parsing? 3. Define Actions in a System 2. Define a Label Space! 1. Get a Corpus! (e.g. Arc-Hybrid)! BEGN LEFT SUBJ RIGHT OBJT SHIFT CONT REDUCE ATTR 31

  32. Detailed Architecture 1.Initialization Step Stack Buffer Action 0 a young boy wearing black shirt is in front of a goal ROOT SHIFT 1 a young boy wearing black shirt is in front of a goal ROOT REDUCE 2 young boy wearing black shirt is in front of a goal ROOT SHIFT 3 young boy wearing black shirt is in front of a goal ROOT LEFT( ATTR ) 4 boy wearing black shirt is in front of a goal ROOT SHIFT 2. Predict the next action to take 32 Ref: Kiperwasser and Goldberg, Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations, TACL 2016

  33. Detailed Architecture Step Stack Buffer Action 0 a young boy wearing black shirt is in front of a goal ROOT SHIFT a 1 young boy wearing black shirt is in front of a goal ROOT REDUCE 2 young boy wearing black shirt is in front of a goal ROOT SHIFT 3 young boy wearing black shirt is in front of a goal ROOT LEFT( ATTR ) 4 boy wearing black shirt is in front of a goal ROOT SHIFT 2 fully connected layers BiLSTM a young boy wearing black shirt is in front of a goal ROOT 33 Ref: Kiperwasser and Goldberg, Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations, TACL 2016

  34. Step Stack Buffer Action 0 a young boy wearing black shirt is in front of a goal ROOT SHIFT a young boy wearing black shirt is in front of a goal ROOT 34

  35. Step Stack Buffer Action 0 a young boy wearing black shirt is in front of a goal ROOT SHIFT 1 a young boy wearing black shirt is in front of a goal ROOT REDUCE a young boy wearing black shirt is in front of a goal ROOT 35

  36. Step Stack Buffer Action 0 a young boy wearing black shirt is in front of a goal ROOT SHIFT 1 a young boy wearing black shirt is in front of a goal ROOT REDUCE 2 young boy wearing black shirt is in front of a goal ROOT SHIFT a young boy wearing black shirt is in front of a goal ROOT 36

  37. Step Stack Buffer Action 0 a young boy wearing black shirt is in front of a goal ROOT SHIFT 1 a young boy wearing black shirt is in front of a goal ROOT REDUCE 2 young boy wearing black shirt is in front of a goal ROOT SHIFT 3 young boy wearing black shirt is in front of a goal ROOT LEFT( ATTR ) ATTR a young boy wearing black shirt is in front of a goal ROOT 37

  38. Step Stack Buffer Action 0 a young boy wearing black shirt is in front of a goal ROOT SHIFT 1 a young boy wearing black shirt is in front of a goal ROOT REDUCE 2 young boy wearing black shirt is in front of a goal ROOT SHIFT 3 young boy wearing black shirt is in front of a goal ROOT LEFT( ATTR ) 4 boy wearing black shirt is in front of a goal ROOT SHIFT ATTR a young boy wearing black shirt is in front of a goal ROOT 38

  39. Step Stack Buffer Action 1 a young boy wearing black shirt is in front of a goal ROOT REDUCE 2 young boy wearing black shirt is in front of a goal ROOT SHIFT 3 young boy wearing black shirt is in front of a goal ROOT LEFT( ATTR ) 4 boy wearing black shirt is in front of a goal ROOT SHIFT 5 boy wearing black shirt is in front of a goal ROOT SHIFT ATTR a young boy wearing black shirt is in front of a goal ROOT 39

  40. Step Stack Buffer Action 2 young boy wearing black shirt is in front of a goal ROOT SHIFT 3 young boy wearing black shirt is in front of a goal ROOT LEFT( ATTR ) 4 boy wearing black shirt is in front of a goal ROOT SHIFT 5 boy wearing black shirt is in front of a goal ROOT SHIFT 6 boy wearing black shirt is in front of a goal ROOT SHIFT ATTR a young boy wearing black shirt is in front of a goal ROOT 40

Recommend

Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs

Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs

Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs Semantic Parsing Dependency parsing models the syntactic structure between words in a sentence. Dependency Parsing vs Semantic Parsing

637 views • 31 slides
Dependency Parsing & Feature-based Parsing Ling571 Deep Processing Techniques for NLP

Dependency Parsing & Feature-based Parsing Ling571 Deep Processing Techniques for NLP

Dependency Parsing & Feature-based Parsing Ling571 Deep Processing Techniques for NLP February 2, 2015 Roadmap Dependency parsing Graph-based dependency parsing Maximum spanning tree CLE Algorithm Learning

899 views • 87 slides
Dependency Parsing II CMSC 470 Marine Carpuat Graph-based Dependency Parsing Slides credit:

Dependency Parsing II CMSC 470 Marine Carpuat Graph-based Dependency Parsing Slides credit:

Dependency Parsing II CMSC 470 Marine Carpuat Graph-based Dependency Parsing Slides credit: Joakim Nivre Directed Spanning Trees Dependency Parsing as Finding the Maximum Spanning Tree Views parsing as finding the best directed spanning

981 views • 61 slides
Algorithms for NLP CS 11-711 Fall 2020 Lecture 14: Graph-based dependency parsing Emma

Algorithms for NLP CS 11-711 Fall 2020 Lecture 14: Graph-based dependency parsing Emma

Algorithms for NLP CS 11-711 Fall 2020 Lecture 14: Graph-based dependency parsing Emma Strubell Announcements No recitation on Friday (Tartan Community Day). 2 Dependency parsing 3 Dependency parsing Input buffer Transition-based

1.15k views • 77 slides
Graph-based Dependency Parsing (Chu-Liu-Edmonds algorithm) Sam Thomson (with thanks to Swabha

Graph-based Dependency Parsing (Chu-Liu-Edmonds algorithm) Sam Thomson (with thanks to Swabha

Graph-based Dependency Parsing (Chu-Liu-Edmonds algorithm) Sam Thomson (with thanks to Swabha Swayamdipta) University of Washington, CSE 490u February 22, 2017 Outline Dependency trees Three main approaches to parsing

648 views • 54 slides
Marina Valeeva Outline 2 1. Introduction What is Dependency Parsing? What is a

Marina Valeeva Outline 2 1. Introduction What is Dependency Parsing? What is a

GRAPH-BASED DEPENDENCY PARSING Marina Valeeva Outline 2 1. Introduction What is Dependency Parsing? What is a Dependency Tree? Projectivity vs. Non-Projectivity 2. Graph-Based Dependency Parsing 3. Models Edge Based Factorization

572 views • 40 slides
Statistical Parsing Dependency parsing ar ltekin University of Tbingen Seminar fr

Statistical Parsing Dependency parsing ar ltekin University of Tbingen Seminar fr

Statistical Parsing Dependency parsing ar ltekin University of Tbingen Seminar fr Sprachwissenschaft November 2016 Recap/background Dependency grammar Dependency parsing Evaluation Summary Ingredients of a parser

1.44k views • 89 slides
Dependency Parsing CMSC 723 / LING 723 / INST 725 Marine Carpuat Fig credits: Joakim Nivre, Dan

Dependency Parsing CMSC 723 / LING 723 / INST 725 Marine Carpuat Fig credits: Joakim Nivre, Dan

Dependency Parsing CMSC 723 / LING 723 / INST 725 Marine Carpuat Fig credits: Joakim Nivre, Dan Jurafsky & James Martin Dependency Parsing Formalizing dependency trees Transition-based dependency parsing Shift-reduce parsing

676 views • 24 slides
Dependency Parsing 2 CMSC 723 / LING 723 / INST 725 Marine Carpuat Fig credits: Joakim Nivre,

Dependency Parsing 2 CMSC 723 / LING 723 / INST 725 Marine Carpuat Fig credits: Joakim Nivre,

Dependency Parsing 2 CMSC 723 / LING 723 / INST 725 Marine Carpuat Fig credits: Joakim Nivre, Dan Jurafsky & James Martin Dependency Parsing Formalizing dependency trees Transition-based dependency parsing Shift-reduce parsing

1.02k views • 52 slides
Natural Language Processing Other Syntactic Models Parsing IV Dan Klein UC Berkeley Dependency

Natural Language Processing Other Syntactic Models Parsing IV Dan Klein UC Berkeley Dependency

Natural Language Processing Other Syntactic Models Parsing IV Dan Klein UC Berkeley Dependency Parsing Dependency Parsing Pure dependency parsing is only cubic [Eisner 99] Lexicalized parsers can be seen as producing dependency trees

314 views • 5 slides
Graph Based Dependency Parsing Wei Qiu December 15, 2011 . . . . . . Graph Based

Graph Based Dependency Parsing Wei Qiu December 15, 2011 . . . . . . Graph Based

. . . . . . Graph Based Dependency Parsing Graph Based Dependency Parsing Wei Qiu December 15, 2011 . . . . . . Graph Based Dependency Parsing Outline Outline 1 Introduction to dependency parsing 2 Graph based dependency

935 views • 72 slides
Last class Dependency parsing and logistic regression Dependency parsing: a fully lexicalized

Last class Dependency parsing and logistic regression Dependency parsing: a fully lexicalized

Last class Dependency parsing and logistic regression Dependency parsing: a fully lexicalized formalism; tree edges connect words in the Shay Cohen sentence based on head-dependent relationships. (based on slides by Sharon Goldwater) a

359 views • 11 slides
Statistical Dependency Parsing in Korean: From Corpus Generation To Automatic Parsing Workshop on

Statistical Dependency Parsing in Korean: From Corpus Generation To Automatic Parsing Workshop on

Statistical Dependency Parsing in Korean: From Corpus Generation To Automatic Parsing Workshop on Statistical Parsing of Morphologically-Rich Languages 12th International Conference on Parsing Technologies Jinho D. Choi & Martha Palmer

199 views • 19 slides
a better and faster way Shu Kong CS, ICS, UCI Image Understanding --> Scene Parsing Scene

a better and faster way Shu Kong CS, ICS, UCI Image Understanding --> Scene Parsing Scene

Scene Parsing through Per-Pixel Labeling: a better and faster way Shu Kong CS, ICS, UCI Image Understanding --> Scene Parsing Scene Parsing semantic segmentation classifying each pixel into one of defined categories Scene Parsing semantic

1.29k views • 103 slides
Advanced Dependency Parsing Joakim Nivre Uppsala University Linguistics and Philology Based on

Advanced Dependency Parsing Joakim Nivre Uppsala University Linguistics and Philology Based on

Advanced Dependency Parsing Joakim Nivre Uppsala University Linguistics and Philology Based on tutorials with Ryan McDonald Advanced Dependency Parsing 1(36) Introduction Plan for the Lecture 1. Graph-based vs. transition-based dependency

1.61k views • 109 slides
Dependency Parsing CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu Slides

Dependency Parsing CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu Slides

Dependency Parsing CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu Slides credit: Joakim Nivre & Ryan McDonald Agenda Formalizing dependency graphs Formalizing transition-based parsing Graph-based

856 views • 32 slides
Dependency Grammars and Parsing CMSC 473/673 UMBC Outline Review: PCFGs and CKY Dependency

Dependency Grammars and Parsing CMSC 473/673 UMBC Outline Review: PCFGs and CKY Dependency

Dependency Grammars and Parsing CMSC 473/673 UMBC Outline Review: PCFGs and CKY Dependency Grammars Dependency Parsing (Shift-reduce) From syntax to semantics Probabilistic Context Free Grammar 1.0 S NP VP 1.0 PP P NP .4 NP Det

862 views • 75 slides
Dependency Parsing and Feature-based Parsing Ling 571 Deep Processing Techniques for

Dependency Parsing and Feature-based Parsing Ling 571 Deep Processing Techniques for

Dependency Parsing and Feature-based Parsing Ling 571 Deep Processing Techniques for NLP October 21, 2019 Shane Steinert-Threlkeld 1 Announcements Thanks for the feedback! HW3: mean 92 Handling ungrammaticality:

1.3k views • 70 slides
Thoughts on Learner Data and Motivation Learner Language Dependency Parsing and Dependency

Thoughts on Learner Data and Motivation Learner Language Dependency Parsing and Dependency

Thoughts on Learner Data and Dependency Parsing Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers Introduction and Thoughts on Learner Data and Motivation Learner Language Dependency Parsing and Dependency Annotation Approximated

812 views • 53 slides
Semantic Dependency Graph Parsing Using Tree Approximations eljko Agi Alexander Koller

Semantic Dependency Graph Parsing Using Tree Approximations eljko Agi Alexander Koller

Semantic Dependency Graph Parsing Using Tree Approximations eljko Agi Alexander Koller Stephan Oepen Center for Language Technology, University of Copenhagen Department of Linguistics, University of Potsdam

1.29k views • 32 slides
Dependency Parsing Lecture 2 Overview Nivre's Arc-Eager / Arc-Standard Algorithm

Dependency Parsing Lecture 2 Overview Nivre's Arc-Eager / Arc-Standard Algorithm

Dependency Parsing Lecture 2 Overview Nivre's Arc-Eager / Arc-Standard Algorithm Covington's Parsing Strategy Pseudo-projective Parsing Java ML Libraries - OpenNLP MaxEnt - Mallet - Liblinear Arc-Standard Algorithm Two

1.21k views • 92 slides
Dependency Parsing Dr. Besnik Fetahu Parsing so far Use context free grammars to

Dependency Parsing Dr. Besnik Fetahu Parsing so far Use context free grammars to

Dependency Parsing Dr. Besnik Fetahu Parsing so far Use context free grammars to determine constituents in a clause or sentence Use CFGs to parse entire sentences into constituency based parse trees, e.g. syntactic parse trees In

461 views • 42 slides
Projective Dependency Parsing with Perceptron Xavier Carreras , Mihai Surdeanu, and Llus

Projective Dependency Parsing with Perceptron Xavier Carreras , Mihai Surdeanu, and Llus

Projective Dependency Parsing with Perceptron Xavier Carreras , Mihai Surdeanu, and Llus Mrquez Technical University of Catalonia {carreras,surdeanu,lluism}@lsi.upc.edu 8th June 2006 Outline Introduction Parsing and Learning Parsing

356 views • 22 slides
Dependency Parsing Joakim Nivre Uppsala University Department of Linguistics and Philology

Dependency Parsing Joakim Nivre Uppsala University Department of Linguistics and Philology

Dependency Parsing Joakim Nivre Uppsala University Department of Linguistics and Philology joakim.nivre@lingfil.uu.se Dependency Parsing 1(14) 1. Dependency Trees 2. Arc-Factored Models 3. Online Learning 4. Eisners Algorithm 5.

726 views • 14 slides

More recommend