learning quantities from vision and language
play

Learning quantities from vision and language Raffaella Bernardi - PowerPoint PPT Presentation

Sep 14, 2022 •458 likes •1.09k views

Learning quantities from vision and language Raffaella Bernardi University of Trento March 23, 2017 Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 1 / 44 Cardinals and Quantifiers Three of the animals are dogs.

  1. Learning quantities from vision and language Raffaella Bernardi University of Trento March 23, 2017 Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 1 / 44

  2. Cardinals and Quantifiers Three of the animals are dogs. vs. Most of the animals are dogs Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 2 / 44

  3. Cardinals and Quantifiers Three of the animals are dogs. vs. Most of the animals are dogs Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 2 / 44

  4. Quantifiers: are they in a scale? Expected abstract scale: < no, few, some, most, all > Q. How do we learn they are in this order? Q. Do we take this order into account when using them? Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 3 / 44

  5. Quantifiers: are they in a scale? Expected abstract scale: < no, few, some, most, all > Q. How do we learn they are in this order? Q. Do we take this order into account when using them? Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 3 / 44

  6. Quantifiers: are they in a scale? Expected abstract scale: < no, few, some, most, all > Q. How do we learn they are in this order? Q. Do we take this order into account when using them? Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 3 / 44

  7. Litteral vs. Pragmatic meaning What do we learn from language, what from vision, what from both? Conjecture 1: we can learn their litteral meaning (respecting the abstract scale) from images . Conjecture 2: they can be represented by a cross-modal function . Conjecture 3: text corpora could help learning their use . Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 4 / 44

  8. Litteral vs. Pragmatic meaning What do we learn from language, what from vision, what from both? Conjecture 1: we can learn their litteral meaning (respecting the abstract scale) from images . Conjecture 2: they can be represented by a cross-modal function . Conjecture 3: text corpora could help learning their use . Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 4 / 44

  9. Litteral vs. Pragmatic meaning What do we learn from language, what from vision, what from both? Conjecture 1: we can learn their litteral meaning (respecting the abstract scale) from images . Conjecture 2: they can be represented by a cross-modal function . Conjecture 3: text corpora could help learning their use . Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 4 / 44

  10. New Challenge for CV From content words to Function words Most tasks considered so far involve processing of objects and lexicalised relations amongst objects ( content words ). Humans (even pre-school children) can abstract over raw data to perform certain types of higher-level reasoning, expressed in natural language by function words . Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 5 / 44

  11. Operations inolved in quatifying A logical strategy Quantifiers require: 1 an approximate number estimation mechanism, acting over the relevant sets in the image; 2 a quantification comparison step. A “logical” strategy: 1 from raw data to abstract set representation 2 from the latter to quantifiers. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 6 / 44

  12. Operations inolved in quatifying A logical strategy Quantifiers require: 1 an approximate number estimation mechanism, acting over the relevant sets in the image; 2 a quantification comparison step. A “logical” strategy: 1 from raw data to abstract set representation 2 from the latter to quantifiers. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 6 / 44

  13. Comparison step Look, some green circles!: Learning to quantify from images (Sorodoc et al., 2016): Very high results: NNs should be able to learn the second subtask quite easily. Is the “logical” strategy a good one? Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 7 / 44

  14. Comparison step Look, some green circles!: Learning to quantify from images (Sorodoc et al., 2016): Very high results: NNs should be able to learn the second subtask quite easily. Is the “logical” strategy a good one? Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 7 / 44

  15. Learning quantification from images Layout Learning quantification from images 1 Quantifiers vs. Cardinals 2 Behavioral Study 3 Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 8 / 44

  16. Learning quantification from images Learning quantification from images Pay attention to those sets! Learning quantification from images Sorodoc et. al. just submitted. (a) (b) (c) (d) (e) Query: fish are red. Answers: (a) All, (b) Most, (c) Some, (d) Few, (e) No. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 9 / 44

  17. Learning quantification from images Learning quantification from images Pay attention to those sets! Learning quantification from images Sorodoc et. al. just submitted. (a) (b) (c) (d) (e) Query: fish are red. Answers: (a) All, (b) Most, (c) Some, (d) Few, (e) No. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 9 / 44

  18. Learning quantification from images Not raw data: All sorts of variances in place The system cannot memorize correlations between type of objects and quantifiers property of objects and quantifiers number of objects and quantifiers Quite challenging! Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 10 / 44

  19. Learning quantification from images Quantifiers as proportions Q of the fish are red . ���� ���� restrictor scope We take quantifiers to be a fiexed relation: | scope ∩ restrictor | ( e . g . | red ∩ fish | ) | restrictor | | fish | Prevalence estimates (Khemlain et al 2009): No: 0% Few: 1% - 17% (inc) Some: 17 % - 70% Most: 70% (inc) - 99% (inc.) All: 100% Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 11 / 44

  20. Learning quantification from images Quantifiers as proportions Q of the fish are red . ���� ���� restrictor scope We take quantifiers to be a fiexed relation: | scope ∩ restrictor | ( e . g . | red ∩ fish | ) | restrictor | | fish | Prevalence estimates (Khemlain et al 2009): No: 0% Few: 1% - 17% (inc) Some: 17 % - 70% Most: 70% (inc) - 99% (inc.) All: 100% Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 11 / 44

  21. Learning quantification from images Quantifiers as proportions Q of the fish are red . ���� ���� restrictor scope We take quantifiers to be a fiexed relation: | scope ∩ restrictor | ( e . g . | red ∩ fish | ) | restrictor | | fish | Prevalence estimates (Khemlain et al 2009): No: 0% Few: 1% - 17% (inc) Some: 17 % - 70% Most: 70% (inc) - 99% (inc.) All: 100% Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 11 / 44

  22. Learning quantification from images Computer Vision Models Start simple: concatenation. CNN+BOW Zhou et al. Simple Baseline for Visual Question Answering 2015 (iBOWIMG) Memorize correlations, no higher level abstraction Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 12 / 44

  23. Learning quantification from images Computer Vision Models Lesson learned from SoA: Memory and Attention Memory process new information based on previous ones. (LSTM, GRU) Attention Mechanism Use language to help making the representation of the image more focused Stacked Attention use language to focus the visual representation and use the later to focus the linguistic representation. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 13 / 44

  24. Learning quantification from images Computer Vision Models Lesson learned from SoA: Memory and Attention Memory process new information based on previous ones. (LSTM, GRU) Attention Mechanism Use language to help making the representation of the image more focused Stacked Attention use language to focus the visual representation and use the later to focus the linguistic representation. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 13 / 44

  25. Learning quantification from images Computer Vision Models Lesson learned from SoA: Memory and Attention Memory process new information based on previous ones. (LSTM, GRU) Attention Mechanism Use language to help making the representation of the image more focused Stacked Attention use language to focus the visual representation and use the later to focus the linguistic representation. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 13 / 44

  26. Learning quantification from images Sequential Processing CNN+LSTM model Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 14 / 44

  27. Learning quantification from images Attention Mechanism: SAN’s attention layer Yang, Z., et al. (CVPR 2016). Stacked attention networks (SAN) for image question answering. Linguistic Nonlinear input transformation Linear Tanh Softmax transformation transformation transformation + + Visual + input + + + Gist Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 15 / 44

Recommend

Vision, Language, Interaction and Generation Qi Wu Australian Institute for Machine Learning

Vision, Language, Interaction and Generation Qi Wu Australian Institute for Machine Learning

Vision, Language, Interaction and Generation Qi Wu Australian Institute for Machine Learning Australia Centre for Robotic Vision University of Adelaide Vision-and-Language Computer Vision (CV) Natural Language Processing (NLP) Image

618 views • 57 slides
Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning

Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning

Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning Jiasen Lu April 21, 2020 1 1 Vision and Language Visual Question Answering Image Captioning Visual Commonsense Reasoning Refer Expression 2

1.13k views • 53 slides
Language and Vision at UniTN Raffaella Bernardi University of Trento LaVi @ UniTn Learning the

Language and Vision at UniTN Raffaella Bernardi University of Trento LaVi @ UniTn Learning the

Language and Vision at UniTN Raffaella Bernardi University of Trento LaVi @ UniTn Learning the meaning of Quantifiers from Language, Vision (and Audio): https://quantit-clic.github.io/ none almost none few the smaller part some many most

753 views • 57 slides
Vision, Language, Learning Communication, Participation: An Approach to AAC for Students with CVI

Vision, Language, Learning Communication, Participation: An Approach to AAC for Students with CVI

Vision, Language, Learning Communication, Participation: An Approach to AAC for Students with CVI Christine Roman-Lantzy, Ph.D. Sarah W. Blackstone, Ph.D., CCC-SLP 1 1 Material from: The Bridge School &amp; CATIC Children with CVI and CCN

1.02k views • 65 slides
Introduction to Deep Learning Principles and applications in vision and natural language

Introduction to Deep Learning Principles and applications in vision and natural language

Introduction to Deep Learning Principles and applications in vision and natural language processing Jakob Verbeek (INRIA) Slides in collaboration with Laurent Besacier (Univ. Grenoble Alpes) 2018 Introduction Convolutional Neural Networks

517 views • 28 slides
Multimodality Learning from Text, Speech, and Vision CMU 11-4/611 Natural Language Processing

Multimodality Learning from Text, Speech, and Vision CMU 11-4/611 Natural Language Processing

Multimodality Learning from Text, Speech, and Vision CMU 11-4/611 Natural Language Processing Lecture 28 April 14, 2020 Shruti Palaskar Outline I. What is multimodality? II. Types of modalities III. Commonly used Models IV. Multimodal

1.48k views • 78 slides
From Activity to Language: Learning to recognise the meaning of motion Centre for Vision, Speech

From Activity to Language: Learning to recognise the meaning of motion Centre for Vision, Speech

From Activity to Language: Learning to recognise the meaning of motion Centre for Vision, Speech and Signal Processing Prof Rich Bowden 20 June 2011 Centre for Vision Speech and Signal Processing Overview Talk is about recognising spatio

538 views • 50 slides
Deep learning in computer vision and natural language processing Yifeng Tao School of Computer

Deep learning in computer vision and natural language processing Yifeng Tao School of Computer

Introduction to Machine Learning Deep learning in computer vision and natural language processing Yifeng Tao School of Computer Science Carnegie Mellon University Slides adapted from Matt Gormley, Russ Salakhutdinov Yifeng Tao Carnegie

415 views • 40 slides
Vision and Language Learning with Graph Neural Networks Linchao Zhu 22 Apr, 2020 Recognition,

Vision and Language Learning with Graph Neural Networks Linchao Zhu 22 Apr, 2020 Recognition,

Vision and Language Learning with Graph Neural Networks Linchao Zhu 22 Apr, 2020 Recognition, LEarning, Reasoning UTS CRICOS 00099F Overview RNNs for Image Captioning Transformer for Image Captioning Graph Network for Visual

385 views • 19 slides
ENGLISH LANGUAGE DEPARTMENT ENGLISH LANGUAGE LEARNING STELLAR SUPPORT PROGRAMME PROGRAMMES

ENGLISH LANGUAGE DEPARTMENT ENGLISH LANGUAGE LEARNING STELLAR SUPPORT PROGRAMME PROGRAMMES

ENGLISH LANGUAGE DEPARTMENT ENGLISH LANGUAGE LEARNING STELLAR SUPPORT PROGRAMME PROGRAMMES HOLISTIC &amp; ASSESSMENT EVENTS ST STrate tegi gies for or Englis lish Langua guage Learni rning ng And nd Readi ding ng VISION :

401 views • 19 slides
Self-supervised Learning for Vision-and-Language Licheng Yu, Yen-Chun Chen, Linjie Li Nowadays

Self-supervised Learning for Vision-and-Language Licheng Yu, Yen-Chun Chen, Linjie Li Nowadays

Self-supervised Learning for Vision-and-Language Licheng Yu, Yen-Chun Chen, Linjie Li Nowadays Machine Learning Algorithm Data Compute Nowadays Machine Learning Algorithm Data Compute Datasets + Labels MS COCOs Image Captioning:

3.23k views • 115 slides
Impact of Deep Learning Speech Recogni4on Computer Vision

Impact of Deep Learning Speech Recogni4on Computer Vision

Impact of Deep Learning Speech Recogni4on Computer Vision Recommender Systems Language Understanding Drug Discovery and Medical Image

526 views • 21 slides
WELCOME TO VISION NEXT BASE CAMP! Vision Next Learning Journey A city of informed, engaged

WELCOME TO VISION NEXT BASE CAMP! Vision Next Learning Journey A city of informed, engaged

WELCOME TO VISION NEXT BASE CAMP! Vision Next Learning Journey A city of informed, engaged philanthropists accelerating meaningful change for all. Vision Next Learning Goals The New Philanthropy Explore the collective and personal values

147 views • 14 slides
vision &amp; language CS 685, Fall 2020 Introduction to Natural Language Processing

vision &amp; language CS 685, Fall 2020 Introduction to Natural Language Processing

vision &amp; language CS 685, Fall 2020 Introduction to Natural Language Processing http://people.cs.umass.edu/~miyyer/cs685/ Mohit Iyyer College of Information and Computer Sciences University of Massachusetts Amherst some slides adapted from

737 views • 57 slides
VISION, MISSION, AND VALUE STATEMENT VISION: Transforming lives through learning. M ISSION :

VISION, MISSION, AND VALUE STATEMENT VISION: Transforming lives through learning. M ISSION :

VISION, MISSION, AND VALUE STATEMENT VISION: Transforming lives through learning. M ISSION : Provide outstanding learning opportunities that prepare students to meet community needs and future challenges of a complex, global society. V ALUE S

506 views • 21 slides
Neural Program Synthesis Rishabh Singh, Google Brain Great Collaborators! Deep Learning and

Neural Program Synthesis Rishabh Singh, Google Brain Great Collaborators! Deep Learning and

Neural Program Synthesis Rishabh Singh, Google Brain Great Collaborators! Deep Learning and Evolutionary Progression Vision Speech Language Deep Learning and Evolutionary Progression Vision Speech Language Programming Deep Learning

1.44k views • 142 slides
Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks Fengda Zhu, Yi Zhu,

Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks Fengda Zhu, Yi Zhu,

MONASH INFORMATION TECHNOLOGY Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks Fengda Zhu, Yi Zhu, Xiaojun Chang, Xiaodan Liang Outline 1. Embodied Navigation 2. Vision-language Navigation Task 3. Related Works 4.

559 views • 21 slides
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling Tsu-Jui Fu Xin Wang

Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling Tsu-Jui Fu Xin Wang

Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling Tsu-Jui Fu Xin Wang Matthew Peterson Scott Grafton Miguel Eckstein William Wang UC Santa Barbara Vision-and-Language Navigation (VLN) Achieve the goal based on the

387 views • 9 slides
A Joint Model of Language and Perception for Grounded Attribute Learning Cynthia Matuszek,

A Joint Model of Language and Perception for Grounded Attribute Learning Cynthia Matuszek,

A Joint Model of Language and Perception for Grounded Attribute Learning Cynthia Matuszek, Nicholas FitzGerald, Liefeng Bo, Luke Zettlemoyer, Dieter Fox Debidatta Dwibedi SE 367 The Vision Robots should learn about their environment by

143 views • 12 slides
What can language learning do for me? Why might it be important for YOU to learn another language?

What can language learning do for me? Why might it be important for YOU to learn another language?

What can language learning do for me? Why might it be important for YOU to learn another language? You have enjoyed some or all of what you have learned in Y7-Y9 and would like to continue developing your ability to speak French or Spanish

472 views • 8 slides
20160322 DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER Agenda Mission, Vision, and

20160322 DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER Agenda Mission, Vision, and

DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER 20160322 DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER Agenda Mission, Vision, and Values DLIFLC Overview and Profile Current Focus and Way Ahead Worldwide Presence 2

746 views • 29 slides
Individual Factors Influencing Outcomes Louise Courtney Early Language Learning The

Individual Factors Influencing Outcomes Louise Courtney Early Language Learning The

Individual Factors Influencing Outcomes Louise Courtney Early Language Learning The provision of early language programs is often based on the assumption (from naturalistic language learning) that young learners find language learning

413 views • 20 slides
developmental vision and rehabilitation Vision and Learning resources Linked slides Resources

developmental vision and rehabilitation Vision and Learning resources Linked slides Resources

developmental vision and rehabilitation Vision and Learning resources Linked slides Resources Research on Vision and Learning Dr. Fortenbachers VisionHelp Blog articles with research More information on Convergence Insufficiency

89 views • 8 slides
Learning from Language Jacob Andreas Doing things with language 2 Doing things with language

Learning from Language Jacob Andreas Doing things with language 2 Doing things with language

Learning from Language Jacob Andreas Doing things with language 2 Doing things with language Who is left of Go up, then go left. the truck? The hooded oriole is a large bird with black wings. A man with a white shirt and

1.73k views • 144 slides

More recommend