vision and language learning with graph neural networks
play

Vision and Language Learning with Graph Neural Networks Linchao Zhu - PowerPoint PPT Presentation

Aug 24, 2022 •188 likes •385 views

Vision and Language Learning with Graph Neural Networks Linchao Zhu 22 Apr, 2020 Recognition, LEarning, Reasoning UTS CRICOS 00099F Overview RNNs for Image Captioning Transformer for Image Captioning Graph Network for Visual

  1. Vision and Language Learning with Graph Neural Networks Linchao Zhu 22 Apr, 2020 Recognition, LEarning, Reasoning UTS CRICOS 00099F

  2. Overview • RNNs for Image Captioning • Transformer for Image Captioning • Graph Network for Visual Commonsense Reasoning Recognition, LEarning, Reasoning

  3. Image Captioning Zero-shot novel object captioning: the model How to generate descriptions for • needs to caption novel objects without additional unseen words? training sentence data about the object. Wu et al., Decoupled Novel Object Captioner, ACM MM 2018. Recognition, LEarning, Reasoning

  4. Image Captioning Wu et al., Decoupled Novel Object Captioner, ACM MM 2018. Recognition, LEarning, Reasoning

  5. Novel Image Captioning Results on eight novel objects in the held-out MS COCO dataset • A larger dataset: nocaps: novel object captioning at scale, ICCV 2019 Wu et al., Decoupled Novel Object Captioner, ACM MM 2018. Recognition, LEarning, Reasoning

  6. Image Captioning • Semantic attributes are useful Visual regions are useful • Li et al., Entangled Transformer for Image Captioning, ICCV 2019. Recognition, LEarning, Reasoning

  7. Image Captioning Li et al., Entangled Transformer for Image Captioning, ICCV 2019. Recognition, LEarning, Reasoning

  8. Image Captioning EnTangled Attention • Li et al., Entangled Transformer for Image Captioning, ICCV 2019. Recognition, LEarning, Reasoning

  9. Image Captioning Gated Bilateral Controller • Li et al., Entangled Transformer for Image Captioning, ICCV 2019. Recognition, LEarning, Reasoning

  10. Image Captioning Results on MSCOCO (Karpathy’s split) • Fuse two models Li et al., Entangled Transformer for Image Captioning, ICCV 2019. Recognition, LEarning, Reasoning

  11. Image Captioning Results on MSCOCO (Karpathy’s split) with sequence-level optimization • Transformer v : visual input only (w/o GBC) Transformer s : semantic attributes only (w/o GBC) Parallel: no ETA but use GBC Stacked v : stacked two visual layers(w/o GBC) Stacked s : stacked two semantic layers (w/o GBC) ETA: ours Li et al., Entangled Transformer for Image Captioning, ICCV 2019. Recognition, LEarning, Reasoning

  12. Visual Commonsense Reasoning Question -> Answer -> Rationale Zellers et al., 2015 Recognition, LEarning, Reasoning

  13. Visual Commonsense Reasoning Wu et al., Connective Cognition Network for Directional Visual Commonsense Reasoning, NeurIPS 2019. Recognition, LEarning, Reasoning

  14. Visual Commonsense Reasoning Local features Global features Wu et al., Connective Cognition Network for Directional Visual Commonsense Reasoning, NeurIPS 2019. Recognition, LEarning, Reasoning

  15. Visual Commonsense Reasoning Directional Reasoning • Conv Local features Global features attention Wu et al., Connective Cognition Network for Directional Visual Commonsense Reasoning, NeurIPS 2019. Recognition, LEarning, Reasoning

  16. Visual Commonsense Reasoning Loss: multi-class cross-entropy loss • • Results on the VCR dataset Wu et al., Connective Cognition Network for Directional Visual Commonsense Reasoning, NeurIPS 2019. Recognition, LEarning, Reasoning

  17. Visual Commonsense Reasoning Conditional centers • Wu et al., Connective Cognition Network for Directional Visual Commonsense Reasoning, NeurIPS 2019. Recognition, LEarning, Reasoning

  18. Visual Commonsense Reasoning Ablation studies for GraphVLAD • No-C: No conditional center No-G: No graph convolution Ablation studies for directional reasoning • No-R: no reasoning module LSTM-R: use LSTM for reasoning GCN: use GCN for reasoning D-GCN: directional GCN for reasoning Wu et al., Connective Cognition Network for Directional Visual Commonsense Reasoning, NeurIPS 2019. Recognition, LEarning, Reasoning

  19. Conclusion • Visual reasoning is challenging • Graph Networks are powerful. More studies to be investigated. • One model solves them all?

Recommend

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning and AI y t-1 y t y t+1 Computer vision h t-1 h t h t+1 Language translation Speech recognition h t-1 h t h t+1 Question answering x t-1 x t x t+1

454 views • 43 slides
Learning Discrete Structures for Graph Neural Networks Luca Franceschi , Mathias Niepert,

Learning Discrete Structures for Graph Neural Networks Luca Franceschi , Mathias Niepert,

Learning Discrete Structures for Graph Neural Networks Luca Franceschi , Mathias Niepert, Massimilano Potil, Xiao He Poster later: Pacific Ballroom # 177 Introduction & Motivations Aim: apply Graph Neural Networks (GNN) to settings in which

238 views • 12 slides
Graph Neural Networks Xiachong Feng TG 2019-04-08 Relies heavily on A Gentle Introduction

Graph Neural Networks Xiachong Feng TG 2019-04-08 Relies heavily on A Gentle Introduction

Graph Neural Networks Xiachong Feng TG 2019-04-08 Relies heavily on A Gentle Introduction to Graph Neural Networks (Basics, DeepWalk, and GraphSage) Structured deep models: Deep learning on graphs and beyond Representation Learning

4.09k views • 64 slides
DAG-GNN: DAG Structure Learning with Graph Neural Networks Yue Yu 1 , Jie Chen 2 , 3 , Tian Gao 3 ,

DAG-GNN: DAG Structure Learning with Graph Neural Networks Yue Yu 1 , Jie Chen 2 , 3 , Tian Gao 3 ,

DAG-GNN: DAG Structure Learning with Graph Neural Networks Yue Yu 1 , Jie Chen 2 , 3 , Tian Gao 3 , Mo Yu 3 1 Department of Mathematics, Lehigh University, USA 2 MIT-IBM Watson AI Lab, USA 3 IBM Research, USA ICML 2019 June 13th, 2019 Background

254 views • 10 slides
Graph Neural Networks E. Daller, S. Bougleux, and L. Brun April 19, 2018 () Graph Neural

Graph Neural Networks E. Daller, S. Bougleux, and L. Brun April 19, 2018 () Graph Neural

Graph Neural Networks E. Daller, S. Bougleux, and L. Brun April 19, 2018 () Graph Neural Networks April 19, 2018 1 / 10 Different Levels/Steps of Recognition Level 0: Hand made classification (Expert Systems) If x > 0 . 3 and y < 1 . 5

889 views • 10 slides
Natural Language Processing with Deep Learning Language Modeling with Recurrent Neural Networks

Natural Language Processing with Deep Learning Language Modeling with Recurrent Neural Networks

Natural Language Processing with Deep Learning Language Modeling with Recurrent Neural Networks Navid Rekab-Saz navid.rekabsaz@jku.at Institute of Computational Perception Agenda Language Modeling with n- grams Recurrent Neural

667 views • 55 slides
Learning Semantic Entity Representations with Knowledge Graph and Deep Neural Networks and its

Learning Semantic Entity Representations with Knowledge Graph and Deep Neural Networks and its

Learning Semantic Entity Representations with Knowledge Graph and Deep Neural Networks and its Application to Named Entity Disambiguation Hongzhao Huang 1 and Larry Heck 2 Computer Science Department, Rensselaer Polytechnic Institute 1 Microsoft

243 views • 23 slides
Introduction to Deep Learning Principles and applications in vision and natural language

Introduction to Deep Learning Principles and applications in vision and natural language

Introduction to Deep Learning Principles and applications in vision and natural language processing Jakob Verbeek (INRIA) Slides in collaboration with Laurent Besacier (Univ. Grenoble Alpes) 2018 Introduction Convolutional Neural Networks

517 views • 28 slides
Graph Representation Learning: Where Probability Theory, Data Mining, and Neural Networks Meet

Graph Representation Learning: Where Probability Theory, Data Mining, and Neural Networks Meet

Graph Representation Learning: Where Probability Theory, Data Mining, and Neural Networks Meet Bruno Ribeir eiro Assis istant nt Profess ssor Departm tment ent of Comp mputer er Scienc nce Purdue Unive Pu versit rsity Joint work

612 views • 44 slides
Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How

Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How

Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How do we evaluate language models? Traditional language models Feedforward neural networks for language modeling Recurrent neural networks for

1.58k views • 58 slides
Learning Algebraic Multigrid using Graph Neural Networks Ilay Luz, Meirav Galun, Haggai Maron,

Learning Algebraic Multigrid using Graph Neural Networks Ilay Luz, Meirav Galun, Haggai Maron,

Learning Algebraic Multigrid using Graph Neural Networks Ilay Luz, Meirav Galun, Haggai Maron, Ronen Basri, Irad Yavneh Goal: Large scale linear systems Solve = is huge, need solution! Some applications:

480 views • 24 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
IN4080 2020 FALL NATURAL LANGUAGE PROCESSING Jan Tore Lnning 2 Neural networks, Language

IN4080 2020 FALL NATURAL LANGUAGE PROCESSING Jan Tore Lnning 2 Neural networks, Language

1 IN4080 2020 FALL NATURAL LANGUAGE PROCESSING Jan Tore Lnning 2 Neural networks, Language models, word2wec Lecture 6, 21 Sept Today 3 Neural networks Language models Word embeddings Word2vec Artificial neural networks

1.02k views • 76 slides
1 Real Neural Learning Artificial Neuron Model Model network as a graph with cells as nodes

1 Real Neural Learning Artificial Neuron Model Model network as a graph with cells as nodes

Neural Networks Analogy to biological neural systems, the most robust learning systems we know. Attempt to understand natural biological systems CS 391L: Machine Learning through computational modeling. Massive parallelism allows

92 views • 6 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
Logical Expressiveness of Graph Neural Networks Mikal Monet October 10th, 2019 Millennium

Logical Expressiveness of Graph Neural Networks Mikal Monet October 10th, 2019 Millennium

Logical Expressiveness of Graph Neural Networks Mikal Monet October 10th, 2019 Millennium Institute for Foundational Research on Data, Chile Graph Neural Networks (GNNs) With: Pablo Barcel, Egor Kostylev, Jorge Prez, Juan Reutter,

777 views • 56 slides
Logical Expressiveness of Graph Neural Networks DIG seminar Mikal Monet March 12th, 2020

Logical Expressiveness of Graph Neural Networks DIG seminar Mikal Monet March 12th, 2020

Logical Expressiveness of Graph Neural Networks DIG seminar Mikal Monet March 12th, 2020 Millennium Institute for Foundational Research on Data, Chile Graph Neural Networks (GNNs) With: Pablo Barcel, Egor Kostylev, Jorge Prez, Juan

1.26k views • 86 slides
IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen University of Oslo March 10, 2019 Our Roadmap Today Language structure: sequences, trees, graphs Recurrent Neural Networks Different

780 views • 76 slides
SuperGlue: Learning Feature Matching with Graph Neural Networks Paul-Edouard Sarlin 1 Daniel

SuperGlue: Learning Feature Matching with Graph Neural Networks Paul-Edouard Sarlin 1 Daniel

SuperGlue: Learning Feature Matching with Graph Neural Networks Paul-Edouard Sarlin 1 Daniel DeTone 2 Tomasz Malisiewicz 2 Andrew Rabinovich 2 Feature matching is ubiquitous 3D reconstruction Visual localization SLAM Place

256 views • 25 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
DEEP LEARNING FOR NATURAL LANGUAGE PROCESSING Lecture 2: Recurrent Neural Networks (RNNs) Caio

DEEP LEARNING FOR NATURAL LANGUAGE PROCESSING Lecture 2: Recurrent Neural Networks (RNNs) Caio

DEEP LEARNING FOR NATURAL LANGUAGE PROCESSING Lecture 2: Recurrent Neural Networks (RNNs) Caio Corro LECTURE 1 RECALL Language modeling with a multi-layer perceptron n 2nd order Markov chain: p ( y 1 , . . . , y n ) = p ( y 1 ) p ( y 2 |

738 views • 69 slides
CS4501: Introduction to Computer Vision Neural Networks (NNs) Artificial Neural Networks (ANNs)

CS4501: Introduction to Computer Vision Neural Networks (NNs) Artificial Neural Networks (ANNs)

CS4501: Introduction to Computer Vision Neural Networks (NNs) Artificial Neural Networks (ANNs) Multi-layer Perceptrons (MLPs Previous Neural Networks The Perceptron Model The Multi-layer Perceptron (MLP) Forward-pass in an MLP

1.54k views • 40 slides

More recommend