a neural model of adaptation in reading
play

A Neural Model of Adaptation in Reading Marten van Schijndel and Tal - PowerPoint PPT Presentation

Jan 19, 2024 •515 likes •1.06k views

A Neural Model of Adaptation in Reading Marten van Schijndel and Tal Linzen November 4, 2018 Department of Cognitive Science, Johns Hopkins University van Schijndel and Linzen November 4, 2018 1 / 24 Thought experiment cassowary

  1. A Neural Model of Adaptation in Reading Marten van Schijndel and Tal Linzen November 4, 2018 Department of Cognitive Science, Johns Hopkins University van Schijndel and Linzen November 4, 2018 1 / 24

  2. Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24

  3. Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24

  4. Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24

  5. Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24

  6. Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24

  7. Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24

  8. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24

  9. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24

  10. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24

  11. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24

  12. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24

  13. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24

  14. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24

  15. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24

  16. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24

  17. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24

  18. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24

  19. A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning van Schijndel and Linzen November 4, 2018 3 / 24 • Humans adapt to syntactic structures

  20. Adaptation is studied in NLP Learn new words from context But can we model human adaptation? van Schijndel and Linzen November 4, 2018 4 / 24 • Domain adaptation (Kuhn & de Mori, 1990; McClosky, 2010) News Model → Biomedical Text • Handling unknown words (Grave et al., 2015) • Style adaptation (Jaech & Ostendorf, 2017) Lawyer A → Lawyer B

  21. Adaptation is studied in NLP Learn new words from context But can we model human adaptation? van Schijndel and Linzen November 4, 2018 4 / 24 • Domain adaptation (Kuhn & de Mori, 1990; McClosky, 2010) News Model → Biomedical Text • Handling unknown words (Grave et al., 2015) • Style adaptation (Jaech & Ostendorf, 2017) Lawyer A → Lawyer B

  22. Our proposed model LSTM language model (Gives prob of next word in sequence) Base Model: Trained on Wikipedia (90M words) (Gulordava et al., 2018) Adaptation algorithm: 1 Test on a sentence 2 Update weights based on that sentence 3 Repeat on remaining sentences van Schijndel and Linzen November 4, 2018 5 / 24

  23. Our proposed model LSTM language model (Gives prob of next word in sequence) Base Model: Trained on Wikipedia (90M words) (Gulordava et al., 2018) Adaptation algorithm: 1 Test on a sentence 2 Update weights based on that sentence 3 Repeat on remaining sentences van Schijndel and Linzen November 4, 2018 5 / 24

  24. Our proposed model LSTM language model (Gives prob of next word in sequence) Base Model: Trained on Wikipedia (90M words) (Gulordava et al., 2018) Adaptation algorithm: 1 Test on a sentence 2 Update weights based on that sentence 3 Repeat on remaining sentences van Schijndel and Linzen November 4, 2018 5 / 24

  25. Our proposed model LSTM language model (Gives prob of next word in sequence) Base Model: Trained on Wikipedia (90M words) (Gulordava et al., 2018) Adaptation algorithm: 1 Test on a sentence 2 Update weights based on that sentence 3 Repeat on remaining sentences van Schijndel and Linzen November 4, 2018 5 / 24

  26. Our proposed model LSTM language model (Gives prob of next word in sequence) Base Model: Trained on Wikipedia (90M words) (Gulordava et al., 2018) Adaptation algorithm: 1 Test on a sentence 2 Update weights based on that sentence 3 Repeat on remaining sentences van Schijndel and Linzen November 4, 2018 5 / 24

  27. Experiment 1 (standard): Does adaptation improve model accuracy? van Schijndel and Linzen November 4, 2018 6 / 24

  28. Accuracy Evaluation Data Test data: Natural Stories Corpus (Futrell et al., 2017) van Schijndel and Linzen November 4, 2018 7 / 24 • 10 texts (485 sentences) • 7 Fairy Tales • 3 Documentaries

  29. Accuracy Results van Schijndel and Linzen November 4, 2018 8 / 24 Full Corpus Separate Story Types 180 Wikipedia Wikipedia+Adaptation 160 140 120 Perplexity 100 80 60 40 20 0 Natural Stories Fairy Tales Documentaries

  30. Experiment 2: Evaluate model against human adaptation van Schijndel and Linzen November 4, 2018 9 / 24

  31. Evaluation Measure: Surprisal Reading times can be predicted with surprisal (Smith and Levy, 2013) van Schijndel and Linzen November 4, 2018 10 / 24 Surprisal ( w i ) = − log P ( w i | w 1 .. i − 1 )

  32. Evaluation Data: Reading Times non-adaptive surprisal van Schijndel and Linzen November 4, 2018 11 / 24 • Timeline of adaptation is similar to human adaptation • Adaptive surprisal predicts reading times better than

  33. Evaluation: Reading Times The soldiers (who were) warned about the dangers conducted the raid. Figure 1: Human reading times Figure 2: Adaptive model surprisal van Schijndel and Linzen November 4, 2018 12 / 24

Recommend

Paper Reading Jun Gao June 26, 2018 Tencent AI Lab Neural Generative Question Answering

Paper Reading Jun Gao June 26, 2018 Tencent AI Lab Neural Generative Question Answering

Paper Reading Jun Gao June 26, 2018 Tencent AI Lab Neural Generative Question Answering [IJCAI2016] Introduction This paper presents an end-to-end neural network model, named Neural Generative Question Answering (GENQA), that can generate

650 views • 28 slides
NEPR208 - Adaptation properties and mechanisms Functional advantages in properties of a neural

NEPR208 - Adaptation properties and mechanisms Functional advantages in properties of a neural

NEPR208 - Adaptation properties and mechanisms Functional advantages in properties of a neural code and changes in those properties What is adaptation? Why do neural systems have a particular nonlinear and filter? Why does the nonlinearity and

931 views • 38 slides
Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly used for what sort of task? a) Recognizing images b) Transcribing speech c) Translating documents d) Playing games Neural network review

431 views • 13 slides
8 Neural MT 2: Attentional Neural MT In the past chapter, we described a simple model for neural

8 Neural MT 2: Attentional Neural MT In the past chapter, we described a simple model for neural

8 Neural MT 2: Attentional Neural MT In the past chapter, we described a simple model for neural machine translation, which uses an encoder to encode sentences as a fixed-length vector. However, in some ways, this view is overly simplified, and

635 views • 9 slides
CK3LI CT K-3 Reading Model Commitment to K-3 reading as the top priority Comprehensive

CK3LI CT K-3 Reading Model Commitment to K-3 reading as the top priority Comprehensive

CK3LI CT K-3 Reading Model Commitment to K-3 reading as the top priority Comprehensive school-wide reading plan Strong literacy leadership High quality classroom reading instruction Intensive reading interventions

519 views • 5 slides
Measuring Immediate Adaptation Performance for Neural Machine Translation Patrick Simianer , Joern

Measuring Immediate Adaptation Performance for Neural Machine Translation Patrick Simianer , Joern

Measuring Immediate Adaptation Performance for Neural Machine Translation Patrick Simianer , Joern Wuebker, John DeNero Lilt NAACL 19 Outline Motivation & Approach 1 2 Evaluation Conclusion 3 2 / 20 Motivation Online adaptation is

765 views • 52 slides
Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Paul Nuytten CPSC 607 Why Model Neural Networks? The Neuron The nervous

697 views • 4 slides
Modeling Neural Networks Paul Nuytten CPSC 607 Outline Why model neural networks? A

Modeling Neural Networks Paul Nuytten CPSC 607 Outline Why model neural networks? A

Modeling Neural Networks Paul Nuytten CPSC 607 Outline Why model neural networks? A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Why Model Neural Networks? The nervous system is a

1.22k views • 23 slides
Language Model Adaptation Hsin-min Wang References: X. Huang et. al., Spoken Language

Language Model Adaptation Hsin-min Wang References: X. Huang et. al., Spoken Language

Language Model Adaptation Hsin-min Wang References: X. Huang et. al., Spoken Language Processing (2001), Chapter 11. M. Bacchiani and B. Roark, Unsupervised language model adaptation, ICASSP2003. Marcello Federico,

372 views • 19 slides
Neural Networks (Reading: Kuncheva Section 2.5) Introduction Inspired by Biology But as used in

Neural Networks (Reading: Kuncheva Section 2.5) Introduction Inspired by Biology But as used in

Neural Networks (Reading: Kuncheva Section 2.5) Introduction Inspired by Biology But as used in pattern recognition research, have little relation with real neural systems (studied in neurology and neuroscience ) Kuncheva: the literature on

528 views • 25 slides
Machine Reading and Reasoning with Neural Program Interpreters Sebastian Riedel @riedelcastro

Machine Reading and Reasoning with Neural Program Interpreters Sebastian Riedel @riedelcastro

Machine Reading and Reasoning with Neural Program Interpreters Sebastian Riedel @riedelcastro Bloomsbury AI Machine Reading Collaborators Pontus Stenetorp Matko Bosnjak Johannes Welbl (UCL) (UCL) (UCL) Tim Rocktschel Jason Naradowsky

770 views • 49 slides
Machine Reading and Reasoning with Neural Program Interpreters Sebastian Riedel @riedelcastro

Machine Reading and Reasoning with Neural Program Interpreters Sebastian Riedel @riedelcastro

Machine Reading and Reasoning with Neural Program Interpreters Sebastian Riedel @riedelcastro Bloomsbury AI Machine Reading Collaborators Pontus Stenetorp Matko Bosnjak Johannes Welbl (UCL) (UCL) (UCL) Tim Rocktschel Jason Naradowsky

814 views • 68 slides
Translation Model Adaptation Using Genre-Revealing Text Features Marlies van der Wees, Arianna

Translation Model Adaptation Using Genre-Revealing Text Features Marlies van der Wees, Arianna

Translation Model Adaptation Using Genre-Revealing Text Features Marlies van der Wees, Arianna Bisazza, Christof Monz Domain adaptation for SMT Prioritize translation candidates that are most relevant to a specific task Translation Model

758 views • 74 slides
Overfitting, Cross-Validation Recommended reading: Neural nets: Mitchell Chapter 4

Overfitting, Cross-Validation Recommended reading: Neural nets: Mitchell Chapter 4

Overfitting, Cross-Validation Recommended reading: Neural nets: Mitchell Chapter 4 Decision trees: Mitchell Chapter 3 Machine Learning 10-701 Tom M. Mitchell Carnegie Mellon University Overview Followup on neural networks

439 views • 24 slides
ADAPTATION: An interdisciplinary and systemic approach to investigate drivers response to

ADAPTATION: An interdisciplinary and systemic approach to investigate drivers response to

ADAPTATION: An interdisciplinary and systemic approach to investigate drivers response to ADAS use. Ralf Risser Juliane Haupt ADAPTATION: The Project ADAPTATION: The Project Main Objectives Model Research Issues Applied Methods

270 views • 14 slides
Session 26 Spring Model / View 1 Reading & Reference Reading

Session 26 Spring Model / View 1 Reading & Reference Reading

Session 26 Spring Model / View Session 26 Spring Model / View 1 Reading & Reference Reading https://www.baeldung.com/spring-mvc-model-model-map-model-view https://www.thymeleaf.org/doc/tutorials/3.0/usingthymeleaf.html

335 views • 8 slides
Machine Reading and Reasoning with Neural Program Interpreters Sebastian Riedel @riedelcastro

Machine Reading and Reasoning with Neural Program Interpreters Sebastian Riedel @riedelcastro

Machine Reading and Reasoning with Neural Program Interpreters Sebastian Riedel @riedelcastro Machine Reading Collaborators Pontus Stenetorp Marzieh Saeidi Max Bartolo Johannes Welbl (Bloomsbury AI) (UCL) (UCL) (Bloomsbury AI)

1.61k views • 157 slides
1 Real Neural Learning Artificial Neuron Model Model network as a graph with cells as nodes

1 Real Neural Learning Artificial Neuron Model Model network as a graph with cells as nodes

Neural Networks Analogy to biological neural systems, the most robust learning systems we know. Attempt to understand natural biological systems CS 391L: Machine Learning through computational modeling. Massive parallelism allows

92 views • 6 slides
Out line Neural net wor ks Percept r on Neural Net works Supervised learning

Out line Neural net wor ks Percept r on Neural Net works Supervised learning

Out line Neural net wor ks Percept r on Neural Net works Supervised learning algorit hms f or neur al net works J uly 7, 2005 Reading: R&N Ch 20.5 CS 486/ 686 Univer sit y of Wat erloo 2 CS486/686 Lecture Slides (c)

85 views • 5 slides
Consensus Attention-based Neural Networks for Reading Comprehension Y IMING C UI , T ING L IU , Z

Consensus Attention-based Neural Networks for Reading Comprehension Y IMING C UI , T ING L IU , Z

Consensus Attention-based Neural Networks for Reading Comprehension Y IMING C UI , T ING L IU , Z HIPENG C HEN , S HIJIN W ANG AND G UOPING H U J OINT L ABORATORY OF HIT AND I FLYTEK R ESEARCH (HFL), C HINA 2016-12-15 O SAKA , J APAN O UTLINE

589 views • 45 slides
Adaptation to Climate Change Adaptation to Climate Change Impact in China Impact in China - A

Adaptation to Climate Change Adaptation to Climate Change Impact in China Impact in China - A

Adaptation to Climate Change Adaptation to Climate Change Impact in China Impact in China - A Case Study on Flood Damage for Investment Decision-making March 2001 Introduction Objectives Model Structure Scenarios Analysis

909 views • 35 slides
An Inventory of Actions that Contribute to Climate Change Adaptation in Ibero-American Model

An Inventory of Actions that Contribute to Climate Change Adaptation in Ibero-American Model

An Inventory of Actions that Contribute to Climate Change Adaptation in Ibero-American Model Forests, Using the Community Capitals Framework Fernando Carrera Josique Lorenzo Mo Model Forests A Model for

786 views • 10 slides
Analyzing Backprop 3-4-16 Reading Quiz Q1: If a neural network has 3 layers with 10 input, 6

Analyzing Backprop 3-4-16 Reading Quiz Q1: If a neural network has 3 layers with 10 input, 6

Analyzing Backprop 3-4-16 Reading Quiz Q1: If a neural network has 3 layers with 10 input, 6 hidden, and 8 output units, what is the dimension of backpropagations local search space? a) 10 + 6 + 8 = 24 b) 10 + 6 * 8 = 58 c) 10 * 6 + 6 *

381 views • 12 slides
Neural Networks + Convolutional Neural Networks Last Class Global Features The perceptron

Neural Networks + Convolutional Neural Networks Last Class Global Features The perceptron

CS4501: Introduction to Computer Vision Neural Networks + Convolutional Neural Networks Last Class Global Features The perceptron model Neural Networks multilayer perceptron model (MLP) Backpropagation Todays Class

789 views • 34 slides

More recommend