Representations of language in a model of visually grounded speech - PowerPoint PPT Presentation

Aug 21, 2022 •136 likes •379 views

Representations of language in a model of visually grounded speech signal Grzegorz Chrupaa Lieke Gelderloos Afra Alishahi Automatic Speech Recognition A major commercial success story in Language Technology Very heavy-handed

Representations of language in a model of visually grounded speech signal Grzegorz Chrupała Lieke Gelderloos Afra Alishahi
Automatic Speech Recognition A major commercial success story in Language Technology
Very heavy-handed supervision I can see you
Grounded speech perception
Data  Flickr8K Audio (Harwath & Glass 2015)  8K images, fjve audio captions each  MS COCO Synthetic Spoken Captions  300K images, fjve synthetically spoken captions each
Project speech and image to joint space a bird walks on a beam bears play in water
Image model Pre-classifcation layer BOAT BIRD BOAR
Speech model  Input: MFCC  Subsampling CNN  Recurrent Highway Network (Zilly et al 2016)  Attention
Model settings
Image retrieval Flickr8K MSCOCO Newer CNN architecture: Harwath et al 2016 (NIPS), Harwath and Glass 2017 (ACL)
Levels of representation  What aspects of sentences are encoded?  Which layers encode form, which encode meaning?  Auxiliary tasks (Adi et al 2017)
Form-related aspects Use activation vectors to decode  Utterance length in words  Presence of specifjc words
Number of words  Input  Activations for utterance  Model  Linear regression
Word presence  Input  Activations for utterance  MFCC for word  Model  MLP
Semantic aspects
Representational Similarity  Correlations between sets of pairwise similarities according to  Activations AND  Edit ops on written sentences  Human judgments (SICK dataset)
Homonym disambiguation
Follow-up work Afra Alishahi, Marie Barking and Grzegorz Chrupała. Encoding of phonology in a recurrent neural model of grounded speech Friday, session #4 at CoNLL
Conclusion Encodings of form and meaning emerge and evolve in hidden layers of stacked RHN listening to grounded speech Code: github.com/gchrupala/visually-grounded-speech Data: doi.org/10.5281/zenodo.400926
Error analysis  Text usually better  Speech better: a yellow and white birtd is in flight  Long descriptions  Misspellings Speech Text
Length
Text model  Convolution → word embedding  No attention

Recommend

A Structured Language Approach to Teach Language and Literacy to Hearing and Visually Impaired

A Structured Language Approach to Teach Language and Literacy to Hearing and Visually Impaired Pupils with Autism Enid Wolf-Schein Rhonda Bachmann Christine Polys Ruth Rogge Purpose of Presentation This paper describes how Structured

515 views • 30 slides

BERT Bidirectional Encoder Representations from Transformers Introduction What is BERT?

BERT Bidirectional Encoder Representations from Transformers Introduction What is BERT? Latest language representational model BERT is conceptually simple and empirically powerful. One of the biggest challenges in natural language

839 views • 20 slides

Learning Structured Natural Language Representations for Semantic Parsing Jianpeng Cheng, Siva

Learning Structured Natural Language Representations for Semantic Parsing Jianpeng Cheng, Siva Reddy, Vijay Saraswat and Mirella Lapata Presented by : Rishika Agarwal Outline - Introduction - Problem Setting - Model - Training Objective -

576 views • 57 slides

How the Marrakesh Model for the Visually Impaired Could Facilitate Access to Cross-Border

How the Marrakesh Model for the Visually Impaired Could Facilitate Access to Cross-Border Supplies of Patented Pharmaceuticals Jerome H. Reichman Bunyan S. Womble Professor of Law Duke University School of Law ATRIP 2018 1 The The D Doha

163 views • 15 slides

Abstract rule representations in a Abstract rule representations in a bilinear model bilinear

Abstract rule representations in a Abstract rule representations in a bilinear model bilinear model Computational and Systems Neuroscience Conference, 2009 Kai Krueger and Peter Dayan Gatsby Computational Neuroscience Unit Introduction A key

441 views • 16 slides

Mikolovs Language Models: Distributed Representations of Sentences and Documents Recurrent

Motivation Introduction and Background Paragraph Embeddings Performance Linguistic Regularities in Continuous Space Word Representations Mikolovs Language Models: Distributed Representations of Sentences and Documents Recurrent Neural

356 views • 20 slides

1 min 1-1 P ( W ) , W = w ; w ; : : : ; w 1 2 n Basic Language Modeling Estimate

Exploiting Syntactic Structure for Language Modeling Basic Language Modeling: Ciprian Chelba, Frederick Jelinek Model Component Parameterization A Structured Language Model: Pruning Strategy Language Model Requirements Word

579 views • 31 slides

Investigating neural representations of spoken language Grzegorz Chrupaa In collaboration

Investigating neural representations of spoken language Grzegorz Chrupaa In collaboration with Afra Alishahi Lieke Gelderloos Marie Barking Mark van der Laan Automatic Speech Recognition A major success story in Language Technology

811 views • 41 slides

Visually Grounded, Task-oriented Dialogue Elia Bruni Outline Language grounding Visual dialogue

Visually Grounded, Task-oriented Dialogue Elia Bruni Outline Language grounding Visual dialogue Q A Appendix: Current and future work Q A 2 Distributional semantics We found a cute, hairy wampimuk sleeping behind the tree 3

382 views • 20 slides

IN5550: Neural Methods in Natural Language Processing Lecture 4 Dense Representations of

IN5550: Neural Methods in Natural Language Processing Lecture 4 Dense Representations of Linguistic Features. Language Modeling. Andrey Kutuzov , Vinit Ravishankar, Jeremy Barnes, Lilja vrelid, Stephan Oepen, & Erik Velldal University of

541 views • 43 slides

Building Fair and Robust Representations for Vision and Language Vicente Ordez-Romn

EMNLP 2019 Tutorial on Bias and Fairness in Natural Language Processing, Hong Kong Building Fair and Robust Representations for Vision and Language Vicente Ordez-Romn Assistant Professor Department of Computer Science Outline Issues

739 views • 60 slides

Model structures on the category of complexes of quiver representations Payam Bahiraei (IPM) (A

Model structures and Hovey pairs Homotopy category of model category Model structure on C (Rep( Q , G )) Some descriptions of D ( Q ) Morphism categories References Model structures on the category of complexes of quiver representations Payam

902 views • 79 slides

Parsing of natural language sentences to syntactic and semantic graph representations

Parsing of natural language sentences to syntactic and semantic graph representations Abschlussvortrag zum Forschungsprojekt Pius Meinert April 13, 2018 Overview Graph Representations Corpora Parsing Techniques Parser 1 Overview Graph

819 views • 57 slides

Contextual Word Representations with BERT and Other Pre-trained Language Models Jacob Devlin

Contextual Word Representations with BERT and Other Pre-trained Language Models Jacob Devlin Google AI Language History and Background Pre-training in NLP Word embeddings are the basis of deep learning for NLP king queen [-0.5, -0.9,

852 views • 43 slides

Proximity Language Model A Language Model beyond Bag of Words through Proximity Jinglei Zhao 1

Proximity Language Model A Language Model beyond Bag of Words through Proximity Jinglei Zhao 1 1,2 Yeogirl Yun 1 iZENEsoft, Inc. 2 Wisenut, Inc. 2 Wi t I Outline Introduction The proposed model Proximity Language Model

693 views • 27 slides

Backpropagation, Self-Attention, Text Representations Through Language Modeling Karl Stratos

CS 533: Natural Language Processing Backpropagation, Self-Attention, Text Representations Through Language Modeling Karl Stratos Rutgers University Karl Stratos CS 533: Natural Language Processing 1/52 Dropout (Slide credit Danqi Chen

1.01k views • 56 slides

Blind/Visually Impaired Silvia Ludena Veronica Sarabia Katie Stoddard Huong Vo Blind/Visually

Blind/Visually Impaired Silvia Ludena Veronica Sarabia Katie Stoddard Huong Vo Blind/Visually Impaired Any loss of ability to gather information by seeing might be considered a visual impairment Total Blindness - vision Legally blind - is the

1.19k views • 42 slides

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks - Jiasen Lu et al. (NeurIPS 2019) Presented by - Chinmoy Samant, cs59688 Overview Introduction Motivation Approach BERT

1.9k views • 60 slides

Language Model Adaptation Hsin-min Wang References: X. Huang et. al., Spoken Language

Language Model Adaptation Hsin-min Wang References: X. Huang et. al., Spoken Language Processing (2001), Chapter 11. M. Bacchiani and B. Roark, Unsupervised language model adaptation, ICASSP2003. Marcello Federico,

372 views • 19 slides

South Dakotas Specialized Schools SD School for the Deaf SD School for the Blind and Visually

South Dakotas Specialized Schools SD School for the Deaf SD School for the Blind and Visually Impaired 1 Whats Different about Sensory Impairments? Language development Concept development Basis for all other learning

817 views • 16 slides

NLU lecture 6: Compositional character representations Adam Lopez alopez@inf.ed.ac.uk Credits:

NLU lecture 6: Compositional character representations Adam Lopez alopez@inf.ed.ac.uk Credits: Clara Vania 2 Feb 2018 Lets revisit an assumption in language modeling (& word2vec) When does this assumption make sense for language

801 views • 66 slides

Presentation 33 Pugh : Using GeoGebra to Model Four Representations of Linear Equations Download

Presentation 33 Pugh : Using GeoGebra to Model Four Representations of Linear Equations Download from: www.tinyurl.com/PughOct17 Students who have understood the mathematics And Im calling on our nations governors and state they

326 views • 4 slides

1 Finite Representations of Languages Languages may be infinite sets of strings. We need a finite

1 Finite Representations of Languages Languages may be infinite sets of strings. We need a finite notation for them. There are at least four ways to do this: 1. Language generators. The language can be represented as a math- ematical sequence w

199 views • 5 slides

Language Modelling Makes Sense Propagating Representations through WordNet for Full-Coverage Word

Language Modelling Makes Sense Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation Da Daniel l Lo Loureiro, Alpio Jorge ACL Florence, 31 July 2019 Sense Embeddings Exploiting the latest Neural

1.16k views • 66 slides