computational models of language learning
play

Computational Models of Language Learning Jelle Zuidema Institute - PowerPoint PPT Presentation

Oct 18, 2022 •356 likes •945 views

Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and Computation, U. of Amsterdam MSc Brain & Cognitive Science, Artificial Intelligence, Logic MoL Guest Lecture Computational Models of Language

  1. Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and Computation, U. of Amsterdam MSc Brain & Cognitive Science, Artificial Intelligence, Logic MoL Guest Lecture Computational Models of Language Learning

  2. Plan for today • Introduction: Grammars in cognitive science and language technology • What kind of grammars do we need? A quick intro to probabilistic grammars • How do we learn them? A quick intro to statistical inference • Efficiency • Accuracy MoL Guest Lecture Computational Models of Language Learning

  3. 2;5 *CHI: seen one those . 3;0 *CHI: I never seen a watch . 3;0 *CHI: I never seen a watch . 3;0 *CHI: I never seen a bandana . 3;0 *CHI: I never seen a monkey train . 3;0 *CHI: I never seen a tree dance . 3;2 *CHI: I never seen a duck like that # riding@o on a pony . 3;2 *CHI: I never seen (a)bout dat [: that] . 3;5 *CHI: I never seen this jet . 3;5 *CHI: I never seen this jet . 3;5 *CHI: I never seen a Sky_Dart . 3;5 *CHI: I never seen this before . 3;8 *CHI: yeah # I seen carpenters too . 3;8 *CHI: where had you seen carpenters do that ? 3;8 *CHI: I never seen her . 3;8 *CHI: I never seen people wear de [: the] fish flies . 3;8 *CHI: where have you seen a whale ? 3;8 *CHI: I never seen a bird talk . 3;11 *CHI: I never seen a kangaroo knit . 3;11 *CHI: I never seen dat [: that] to play . 3;11 *CHI: I never seen a dog play a piano # have you ? 3;11 *CHI: I never seen a rhinoceros eat with a hands . 4;7 *CHI: I seen one in the store some days .

  4. Grammar in child language MacWhinney et al. (1983) Sagae et al. (2007) Borensztajn, Zuidema & Bod (CogSci, 2008) Adam, 3;11.01

  5. Grammar in NLP applications • E.g., speech recognition – please, right this down – write now – who's write, and who's wrong • E.g., anaphora resolution – Mary didn't know who John was married to. He told her, and it turned out, she already knew her. • E.g., machine translation MoL Guest Lecture Computational Models of Language Learning

  6. Steedman, 2008, CL MoL Guest Lecture Computational Models of Language Learning

  7. MoL Guest Lecture Computational Models of Language Learning

  8. MoL Guest Lecture Computational Models of Language Learning

  9. Learning grammars from data • Syntactically annotated corpora – Penn WSJ Treebank trainset: 38k sentences, ~1M words – Tuebingen spoken/written English/German – Corpus Gesproken Nederlands • Unannotated corpora – the web ... – Google's ngram corpora MoL Guest Lecture Computational Models of Language Learning

  10. Spam www.culturomics.org Penn WSJ: 0 counts. MoL Guest Lecture Computational Models of Language Learning

  11. Kick the bucket www.culturomics.org Penn WSJ: 0 counts. MoL Guest Lecture Computational Models of Language Learning

  12. ... know but were afraid to ... www.culturomics.org Penn WSJ: 0 counts. MoL Guest Lecture Computational Models of Language Learning

  13. Probabilistic Grammar Paradigm • Generative models define the process by which sentences are generated, and assign probabilities to sentences. • Statistical inference lets us search through the space of possible generative models. • Empirical evaluation against a manually written 'gold standard' allows us to more-or- less objectively compare different models. MoL Guest Lecture Computational Models of Language Learning

  14. A very brief tour of generative models MoL Guest Lecture Computational Models of Language Learning

  15. Sequences: e.g., Hidden Markov Model MoL Guest Lecture Computational Models of Language Learning

  16. Syntax: e.g., Probabilistic Contextfree Grammars MoL Guest Lecture Computational Models of Language Learning

  17. MoL Guest Lecture Computational Models of Language Learning

  18. Semantics: e.g. Discourse Representation Structure • “It is not clear” negation present tense agent anaphor resolution MoL Guest Lecture Computational Models of Language Learning

  19. MoL Guest Lecture Computational Models of Language Learning

  20. Semantics, e.g. Discourse Representation Structure (Le & Zuidema, 2012, Coling) MoL Guest Lecture Computational Models of Language Learning

  21. A very brief tour of statistical learning MoL Guest Lecture Computational Models of Language Learning

  22. Bayes' Rule P(D|G) P(G) P(G|D)= P(D) MoL Guest Lecture Computational Models of Language Learning

  23. Bayes' Rule prior likelihood P(D|G) P(G) P(G|D)= P(D) posterior probability of data MoL Guest Lecture Computational Models of Language Learning

  24. Bayes' Rule G G prior likelihood P(D|G) P(G) P(G|D)= P(D) posterior probability of data G MoL Guest Lecture Computational Models of Language Learning

  25. Statistical inference P(G|D) G MoL Guest Lecture Computational Models of Language Learning

  26. Statistical inference P(D|G) P(G) P(G|D)= P(D) P(D|G) P(G|D) G G MoL Guest Lecture Computational Models of Language Learning

  27. Statistical inference P(D|G) P(G) P(G|D)= P(D) P(D|G) P(G|D) MoL Guest Lecture Computational Models of Language Learning

  28. Statistical inference Bayesian inversion P(D|G) P(G|D) Generative model MoL Guest Lecture Computational Models of Language Learning

  29. Stochastic hillclimbing P(G|D) MoL Guest Lecture Computational Models of Language Learning

  30. Stochastic hillclimbing P(G|D) MoL Guest Lecture Computational Models of Language Learning

  31. Stochastic hillclimbing P(G|D) MoL Guest Lecture Computational Models of Language Learning

  32. Stochastic hillclimbing P(G|D) MoL Guest Lecture Computational Models of Language Learning

  33. Stochastic hillclimbing P(G|D) MoL Guest Lecture Computational Models of Language Learning

  34. Stochastic hillclimbing MoL Guest Lecture Computational Models of Language Learning

  35. Local optimum P(G|D) MoL Guest Lecture Computational Models of Language Learning

  36. Statistical inference Bayesian inversion P(D|G) P(G|D) Generative model MoL Guest Lecture Computational Models of Language Learning

  37. Statistical inference Bayesian inversion P(D|G) P(G|D) Generative model MoL Guest Lecture Computational Models of Language Learning

  38. Statistical inference Bayesian inversion P(D|G) P(G|D) Generative model MoL Guest Lecture Computational Models of Language Learning

  39. MAP Bayesian inversion P(D|G) P(G|D) Generative model MoL Guest Lecture Computational Models of Language Learning

  40. Maximum likelihood P(D|G) MoL Guest Lecture Computational Models of Language Learning

  41. Learning a grammar • Choose a generative model – HMM, PCFG, PTSG, PTAG, … • Choose an objective function – Maximum Likelihood, Bayesian … • Choose an optimization strategy – Stochastic hillclimbing • Choose a dataset – Penn WSJ treebank • Find the generative model that maximizes the objective function on the dataset! MoL Guest Lecture Computational Models of Language Learning

  42. Does it work in practice? Two major issues for research • Efficiency: How can we optimize our objective functions given exponentially many grammars that assign exponentially many analyses to sentences? • Accuracy : Which combination of generative models, objective functions and efficiency heuristics actually works best? MoL Guest Lecture Computational Models of Language Learning

  43. MoL Guest Lecture Computational Models of Language Learning

  44. MoL Guest Lecture Computational Models of Language Learning

  45. MoL Guest Lecture Computational Models of Language Learning

  46. MoL Guest Lecture Computational Models of Language Learning

  47. Evaluation Treebank Parse MoL Guest Lecture Computational Models of Language Learning

  48. Evaluation The screen was a sea of red Treebank Parse MoL Guest Lecture Computational Models of Language Learning

  49. Evaluation The screen was a sea of red induction/parsing Unsupervised Parse Treebank Parse MoL Guest Lecture Computational Models of Language Learning

  50. Evaluation The screen was a sea of red induction/parsing Unsupervised Parse Treebank Parse MoL Guest Lecture Computational Models of Language Learning

  51. Evaluation • Precision : number of constituents in the unsupervised parse that are also in the treebank parse; correctness • Recall : number of constituents in the treebank parse that are also in the unsupervised parse; completeness • F-score : geometric mean, i.e. F=2*P*R / (P+R) • Labels usually ignored. MoL Guest Lecture Computational Models of Language Learning

  52. Evaluation • Precision : number of constituents in the unsupervised parse that are also in the treebank parse; correctness • Recall : number of constituents in the treebank parse that are also in the unsupervised parse; completeness • F-score : geometric mean, i.e. F=2*P*R / (P+R) • Labels usually ignored. MoL Guest Lecture Computational Models of Language Learning

Recommend

Advanced Probabilistic Models for Generalized SSMs Speech and Language Nonlinear SSMs Bayesian

Advanced Probabilistic Models for Generalized SSMs Speech and Language Nonlinear SSMs Bayesian

Probabilistic Models State-Space Models Advanced Probabilistic Models for Generalized SSMs Speech and Language Nonlinear SSMs Bayesian Learning Mark Andrews Inferring Trajectories Gatsby Computational Neuroscience Unit Learning Example

424 views • 10 slides
Reinforcement Learning Models of the Basal Ganglia Computational Models of Neural Systems

Reinforcement Learning Models of the Basal Ganglia Computational Models of Neural Systems

Reinforcement Learning Models of the Basal Ganglia Computational Models of Neural Systems Lecture 6.2 David S. Touretzky November, 2017 Dopamine Cells Located in SNc (substantia nigra pars compacta) and VTA (ventral tegmental area)

470 views • 30 slides
Natural Language Processing with Deep Learning Sequence-to-sequence Models with Attention Navid

Natural Language Processing with Deep Learning Sequence-to-sequence Models with Attention Navid

Natural Language Processing with Deep Learning Sequence-to-sequence Models with Attention Navid Rekab-Saz navid.rekabsaz@jku.at Institute of Computational Perception Institute of Computational Perception Agenda Sequence-to-sequence

900 views • 64 slides
Invertebrate Learning Computational Models of Neural Systems Lecture 3.3 David S. Touretzky

Invertebrate Learning Computational Models of Neural Systems Lecture 3.3 David S. Touretzky

Invertebrate Learning Computational Models of Neural Systems Lecture 3.3 David S. Touretzky October, 2007 Eric Kandel Nobel Prize in Physiology or Medicine, 2000 10/10/07 Computational Models of Neural Systems 2 Aplysia Californica

776 views • 55 slides
One-Shot Learning: Language Acquisition for Machine SS16 Computational Linguistics for

One-Shot Learning: Language Acquisition for Machine SS16 Computational Linguistics for

One-Shot Learning: Language Acquisition for Machine SS16 Computational Linguistics for Low-Resource Languages Mayumi Ohta July 6, 2016 Institute for Computational Linguistics Heidelberg University Table of contents 1. Introduction 2.

985 views • 70 slides
Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution ? What is language? never start with a dictionary definition!! the language of Google search query completion A language is a dialect with

1.13k views • 30 slides
Transfer learning with neural language models CS 685, Spring 2020 Advanced Natural Language

Transfer learning with neural language models CS 685, Spring 2020 Advanced Natural Language

Transfer learning with neural language models CS 685, Spring 2020 Advanced Natural Language Processing Mohit Iyyer College of Information and Computer Sciences University of Massachusetts Amherst many slides from Jacob Devlin & Matt Peters

869 views • 49 slides
Computational Models of Events Lecture 1: The Role of Events in Language and Computation James

Computational Models of Events Lecture 1: The Role of Events in Language and Computation James

Computational Models of Events Lecture 1: The Role of Events in Language and Computation James Pustejovsky Brandeis University ESSLLI 2018 Summer School Sofia, Bulgaria August 6-10, 2018 1/83 Pustejovsky - Brandeis Computational Event

683 views • 50 slides
Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018

Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018

Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018 Character-level Language models Want language models with an open vocabulary Character-level models give this for free Treat the probability of

1.07k views • 5 slides
4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language models: log-linear language models [9, 4], which take a very di ff erent approach than the count-based n -grams described above. 11 4.1 Model

502 views • 8 slides
The Rescorla-Wagner Learning Model (and one of its descendants) Computational Models of Neural

The Rescorla-Wagner Learning Model (and one of its descendants) Computational Models of Neural

The Rescorla-Wagner Learning Model (and one of its descendants) Computational Models of Neural Systems Lecture 5.1 David S. Touretzky Based on notes by Lisa M. Saksida November, 2015 Outline Classical and instrumental conditioning The

832 views • 40 slides
The Rescorla-Wagner Learning Model (and one of its descendants) Computational Models of Neural

The Rescorla-Wagner Learning Model (and one of its descendants) Computational Models of Neural

The Rescorla-Wagner Learning Model (and one of its descendants) Computational Models of Neural Systems Lecture 5.1 David S. Touretzky Based on notes by Lisa M. Saksida November, 2017 Outline Classical and instrumental conditioning The

925 views • 41 slides
Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How

Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How

Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How do we evaluate language models? Traditional language models Feedforward neural networks for language modeling Recurrent neural networks for

1.58k views • 58 slides
Statistical Natural Language Processing Statistical models: learning, inference, estimation,

Statistical Natural Language Processing Statistical models: learning, inference, estimation,

Statistical Natural Language Processing Statistical models: learning, inference, estimation, prediction ar ltekin University of Tbingen Seminar fr Sprachwissenschaft Summer Semester 2017 Statistical models: learning, inference,

497 views • 25 slides
Bayesian Learning in Undirected Graphical Models Zoubin Ghahramani Gatsby Computational

Bayesian Learning in Undirected Graphical Models Zoubin Ghahramani Gatsby Computational

Bayesian Learning in Undirected Graphical Models Zoubin Ghahramani Gatsby Computational Neuroscience Unit University College London, UK http://www.gatsby.ucl.ac.uk/ and Center for Automated Learning and Discovery Carnegie Mellon University,

305 views • 26 slides
Predictive Hebbian Learning Computational Models of Neural Systems Lecture 5.2 David S.

Predictive Hebbian Learning Computational Models of Neural Systems Lecture 5.2 David S.

Predictive Hebbian Learning Computational Models of Neural Systems Lecture 5.2 David S. Touretzky Based on slides by Mirella Lapata November, 2019 Outline The bee brain Classical conditioning in honeybees identification of VUMmx1

598 views • 34 slides
Predictive Hebbian Learning Computational Models of Neural Systems Lecture 5.2 David S.

Predictive Hebbian Learning Computational Models of Neural Systems Lecture 5.2 David S.

Predictive Hebbian Learning Computational Models of Neural Systems Lecture 5.2 David S. Touretzky Based on slides by Mirella Lapata November, 2015 Outline Clasical conditioning in honeybees identification of VUMmx1 properties of

664 views • 32 slides
Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models answer the question: How likely is a string of English words good English? Help with reordering p lm (the house is small) > p lm (small the is

1.06k views • 51 slides
N-Gram Language Models Jimmy Lin Jimmy Lin The iSchool University of Maryland Wednesday,

N-Gram Language Models Jimmy Lin Jimmy Lin The iSchool University of Maryland Wednesday,

CMSC 723: Computational Linguistics I Session #9 N-Gram Language Models Jimmy Lin Jimmy Lin The iSchool University of Maryland Wednesday, October 28, 2009 N-Gram Language Models What? LMs assign probabilities to sequences of

843 views • 48 slides
Computational Systems Biology Deep Learning in the Life Sciences 6.802 6.874 20.390 20.490

Computational Systems Biology Deep Learning in the Life Sciences 6.802 6.874 20.390 20.490

Computational Systems Biology Deep Learning in the Life Sciences 6.802 6.874 20.390 20.490 HST.506 David Gifford Lecture 9 March 5, 2020 Generative Models http://mit6874.github.io 1 Why generative models? We can sample new examples from a

747 views • 52 slides
Lecture 1: Introduction Statistical and Computational Methods for Learning through Graphical

Lecture 1: Introduction Statistical and Computational Methods for Learning through Graphical

Lecture 1: Introduction Statistical and Computational Methods for Learning through Graphical Models (aka Probabilistic Graphical Models) BIOSTAT 830 September 6 th , 2016 Zhenke Wu Some materials adapted from Eric Xings CMU Graphical Model

478 views • 18 slides
Understanding Machine Learning with Language and Tensors Jon Rawski Linguistics Department

Understanding Machine Learning with Language and Tensors Jon Rawski Linguistics Department

Machine Learning Language Tensors Reduplication Understanding Machine Learning with Language and Tensors Jon Rawski Linguistics Department Institute for Advanced Computational Science Stony Brook University 1 Machine Learning Language

602 views • 36 slides
Computational models of of biological biological Computational models systems systems

Computational models of of biological biological Computational models systems systems

Computational models of of biological biological Computational models systems systems Giancarlo Mauri Universit di Milano-Bicocca Complexity in in biology biology Complexity Molecular level Regulatory gene networks Protein

592 views • 42 slides
Synaptic Learning Rules Computational Models of Neural Systems Lecture 4.1 David S. Touretzky

Synaptic Learning Rules Computational Models of Neural Systems Lecture 4.1 David S. Touretzky

Synaptic Learning Rules Computational Models of Neural Systems Lecture 4.1 David S. Touretzky October, 2019 Why Study Synaptic Plasticity? Synaptic learning rules determine the information processing capabilities of neurons. Synaptic

889 views • 40 slides

More recommend