key value memory networks for
play

Key-Value Memory Networks for Directly Reading Documents (EMNLP16) - PowerPoint PPT Presentation

Mar 01, 2023 •368 likes •835 views

Key-Value Memory Networks for Directly Reading Documents (EMNLP16) Alexander H. Miller 1 , Adam Fisch 1 , Jesse Dodge 1,2 , Amir-Hossein Karimi 1 , Antoine Bordes 1 , Jason Weston 1 1 Facebook AI Research 2 Language Technologies Institute, Carnegie

  2. Key-Value Memory Networks for Directly Reading Documents (EMNLP16) Alexander H. Miller 1 , Adam Fisch 1 , Jesse Dodge 1,2 , Amir-Hossein Karimi 1 , Antoine Bordes 1 , Jason Weston 1 1 Facebook AI Research 2 Language Technologies Institute, Carnegie Mellon University

  3. Question Answering From Text Goal: • Read documents and answer questions about them Challenges for Raw Text: • Information is unstructured • Answers are indirectly or ambiguously expressed • Scattered across multiple documents Poor performance. Work-around: Knowledge Bases? But … intrinsically • Large data sources: e.g. FreeBase, Dbpedia, etc limited? • Have been effective tools for QA

  4. Knowledge Bases vs Text Limitations of Knowledge Base Learning: • Availability • Knowledge bases are often incomplete • Expressiveness Advantages of Raw Text Learning: • Plenty of data More difficult to interpret • Lossless

  5. Our Work Hypothesis: Intuitively, systems answering directly from text should be more powerful than those using KBs. They should be able to address more complex tasks. Problem… In the current state of the art, they are not as accurate, even on basic tasks. This work tries to close that gap .

  6. Contributions WikiMovies • A new dataset for QA in the movies domain • Raw text + Knowledge Base formats side by side • A useful analysis tool for comparing methods Key-Value Memory Networks • A new, more effective method for memory-based QA • Makes directly reading documents more viable

  7. Question Answering + Knowledge Base OMDd (+ MovieLens for tags) Knowledge Base Jurassic Park directed_by Steven Spielberg Jurassic Park written_by David Koepp, Michael Crichton Jurassic Park release_year 1993 Jurassic Park has_tags action, adventure, science fiction, …

  8. Question Answering + Knowledge Base OMDd (+ MovieLens for tags) Knowledge Base Jurassic Park directed_by Steven Spielberg Jurassic Park written_by David Koepp, Michael Crichton Jurassic Park release_year 1993 Jurassic Park has_tags action, adventure, science fiction, … Who directed Jurassic Park? Steven Spielberg Can you describe Jurassic Park in a few words? A science fiction, action, and adventure movie In Jurassic Park, what is the name of the island ??? where the park is built? KBs can suffer from missing information and fixed schemas

  9. Question Answering + Open IE Wikipedia Page: Jurassic Park (film) Jurassic Park is a 1993 American science fiction action adventure film. The film was directed by Steven Spielberg and is the first installment of the "Jurassic Park" franchise. It is based on the 1990 novel of the same name by Michael Crichton, with a screenplay written by Crichton and David Koepp. The film centers on the fictional Isla Nublar, an islet located off Costa Rica's Pacific Coast, where a billionaire philanthropist and a small team of genetic scientists have created a wildlife park of cloned dinosaurs.

  10. Question Answering + Open IE Wikipedia Page: Jurassic Park (film) Jurassic Park is a 1993 American science fiction action adventure film. The film was directed by Coreference Resolution Steven Spielberg and is the first installment of the + "Jurassic Park" franchise. It is based on the 1990 novel of the same name by Michael Crichton, with a Semantic Role Labelling screenplay written by Crichton and David Koepp. The film centers on the fictional Isla Nublar, an islet located off Costa Rica's Pacific Coast, where (or your favorite) a billionaire philanthropist and a small team of genetic scientists have created a wildlife park of cloned dinosaurs. Who directed Jurassic Park? Can you describe Jurassic Park in a few words? In Jurassic Park, what is the name of the island where the park is built?

  11. Question Answering + Open IE Wikipedia Page: Jurassic Park (film) Open IE Knowledge Base Jurassic Park is a 1993 American science fiction Jurassic Park directed by Steven Spielberg action adventure film. The film was directed by Steven Spielberg and is the first installment of the Jurassic Park written by David Koepp and Michael Crichton "Jurassic Park" franchise. It is based on the 1990 Jurassic Park is a 1993 novel of the same name by Michael Crichton, with a screenplay written by Crichton and David Koepp. Jurassic Park is a American science fiction The film centers on the fictional Isla Nublar, an islet located off Costa Rica's Pacific Coast, where Jurassic Park centers on Isla Nublar a billionaire philanthropist and a small team of genetic scientists have created a wildlife park of cloned dinosaurs. Who directed Jurassic Park? Steven Spielberg Can you describe Jurassic Park in a few words? A science fiction, action, and adventure movie In Jurassic Park, what is the name of the island Isla Nublar where the park is built? Open IE is not an easy problem

  12. Question Answering Directly From Text Wikipedia Page: Jurassic Park (film) Jurassic Park is a 1993 American science fiction action adventure film. The film was directed by Steven Spielberg and is the first installment of the "Jurassic Park" franchise. It is based on the 1990 novel of the same name by Michael Crichton, with a screenplay written by Crichton and David Koepp. The film centers on the fictional Isla Nublar, an islet located off Costa Rica's Pacific Coast, where a billionaire philanthropist and a small team of genetic scientists have created a wildlife park of cloned dinosaurs.

  13. Question Answering Directly From Text Wikipedia Page: Jurassic Park (film) Jurassic Park is a 1993 American science fiction action adventure film . The film was directed by Steven Spielberg and is the first installment of the "Jurassic Park" franchise. It is based on the 1990 novel of the same name by Michael Crichton, with a screenplay written by Crichton and David Koepp. The film centers on the fictional Isla Nublar , an islet located off Costa Rica's Pacific Coast, where a billionaire philanthropist and a small team of genetic scientists have created a wildlife park of cloned dinosaurs. Who directed Jurassic Park? Steven Spielberg Can you describe Jurassic Park in a few words? A science fiction, action, and adventure movie In Jurassic Park, what is the name of the island Isla Nublar where the park is built? Much more information available than in KB QA is harder but no need for open IE

  14. WikiMovies Objective: We want to compare the performance of various learning methods between all three settings (raw, open IE, KB). A new analysis tool for QA • A set of 100k question-answer pairs (based on SimpleQuestions) • 3 Knowledge Sources - A KB based on OMBD/MovieLens - Raw text extracted from Wikipedia - An imperfect KB built with open IE methods • Answers to all questions appear in both the KB and Wikipedia text

  15. Key-Value Memory Networks

  16. Memory Networks (Weston et al., ICLR15; Sukhbaatar et al., NIPS15) • Incorporates reasoning with attention over memory (RAM) • Uses collection of dynamic, addressable slots for storing information • Learns to read and weight memory slots • Combines input and memories to choose a response • Has been successfully applied to QA tasks (among others) - e.g. bAbi tasks, Children’s Book Tasks, etc.

  17. Memory Networks for QA (Weston et al., ICLR15; Sukhbaatar et al., NIPS15) Question

  18. Memory Networks for QA (Weston et al., ICLR15; Sukhbaatar et al., NIPS15) Question Embed question

  19. Memory Networks for QA (Weston et al., ICLR15; Sukhbaatar et al., NIPS15) Embed memories Question Embed question Memories

  20. Memory Networks for QA (Weston et al., ICLR15; Sukhbaatar et al., NIPS15) Embed memories Question Embed question Inner product to compute attention scores Memories

  21. Memory Networks for QA (Weston et al., ICLR15; Sukhbaatar et al., NIPS15) Embed memories Embed memories Weight memories Question Embed question Inner product to compute attention scores Memories

  22. Memory Networks for QA (Weston et al., ICLR15; Sukhbaatar et al., NIPS15) Repeat Embed memories Embed memories Weight memories Question Combine input + output Embed question Inner product to compute attention scores Memories

  23. Memory Networks for QA (Weston et al., ICLR15; Sukhbaatar et al., NIPS15) Repeat Embed candidates Embed memories Embed memories Weight memories Question Combine input + output Embed question Inner product to compute attention scores Memories

  24. Memory Networks for QA (Weston et al., ICLR15; Sukhbaatar et al., NIPS15) Repeat Embed candidates Embed memories Embed memories Weight memories Score Question Combine Answer input + output Embed question Inner product to compute attention scores Memories

  25. Memory Networks for QA from KB (Bordes et al., arxiv15) Candidates Steven Spielberg 1993 Question Who directed Jurassic Park? Blade Runner Answer Jeff Goldblum Steven Spielberg Memories … Knowledge Base Jurassic Park directed_by Steven Spielberg Jurassic Park written_by David Koepp, Michael Crichton Jurassic Park release_year 1993 Jurassic Park has_tags action, adventure, science fiction, …

Recommend

Dynamic memory networks for Dynamic memory networks for visual and textual question visual and

Dynamic memory networks for Dynamic memory networks for visual and textual question visual and

Dynamic memory networks for Dynamic memory networks for visual and textual question visual and textual question answering answering Stephen Merity (@smerity) Joint work with the MetaMind team: Caiming Xiong, Richard Socher, and more

378 views • 33 slides
Outline Introduction MemNN: Memory Networks Memory Networks: General Framework MemNNs for Text

Outline Introduction MemNN: Memory Networks Memory Networks: General Framework MemNNs for Text

Outline Introduction MemNN MemNN-WSH Appendix Outline Introduction MemNN: Memory Networks Memory Networks: General Framework MemNNs for Text Experiments MemNN-WSH: Weakly Supervised Memory Networks Introduction MemNN-WSH: Memory via

407 views • 40 slides
CHAPTER II III I CHAPTER Neural Networks as Neural Networks as Associative Memory

CHAPTER II III I CHAPTER Neural Networks as Neural Networks as Associative Memory

Ugur HALICI - METU EEE - ANKARA 11/18/2004 CHAPTER II III I CHAPTER Neural Networks as Neural Networks as Associative Memory Associative Memory CHAPTER III : III : Neural Networks as Associative Memory CHAPTER Neural Networks as

513 views • 22 slides
Short Summery Classification according to memory organization distributed memory

Short Summery Classification according to memory organization distributed memory

Short Summery Classification according to memory organization distributed memory shared-address space Interconnection networks dynamic networks crossbar, bus-based, multistage (Omega network) Omega Network 2i,

251 views • 14 slides
Networks Used As Episodic Memory for An Autonomous Robot Outline

Networks Used As Episodic Memory for An Autonomous Robot Outline

Fusion Adaptive Resonance Theory Networks Used As Episodic Memory for An Autonomous Robot Outline 2 Goals for autonomous robots IRL-1 3 Role of episodic memory 4 Proposed model: EM-ART 5

392 views • 22 slides
Memory Networks and Neural Turing Machines Diego Marcheggiani University of Amsterdam ILLC

Memory Networks and Neural Turing Machines Diego Marcheggiani University of Amsterdam ILLC

Memory Networks and Neural Turing Machines Diego Marcheggiani University of Amsterdam ILLC Unsupervised Language Learning 2016 Outline Motivation Memory Networks End-to-end Memory Networks Neural Turing Machines Outline Motivation

1.25k views • 82 slides
Lecture 23: Recurrent Neural Networks, Long Short Term Memory Networks, Conntectionist Temporal

Lecture 23: Recurrent Neural Networks, Long Short Term Memory Networks, Conntectionist Temporal

Lecture 23: Recurrent Neural Networks, Long Short Term Memory Networks, Conntectionist Temporal Classifjcation, Decision Trees Instructor: Prof. Ganesh Ramakrishnan . . . . . . . . . . . . . . . . . . . . . . . . . . .

699 views • 31 slides
The Potential of Memory Augmented Neural Networks Dalton Caron Montana Technological University

The Potential of Memory Augmented Neural Networks Dalton Caron Montana Technological University

The Potential of Memory Augmented Neural Networks Dalton Caron Montana Technological University November 15, 2019 Overview Review of Perceptron and Feed Forward Networks Recurrent Neural Networks Neural Turing Machines

1.27k views • 57 slides
Memory II. Memory improvement III. Problems with memory 3 systems/stages of Memory: memory

Memory II. Memory improvement III. Problems with memory 3 systems/stages of Memory: memory

Main Focus I. Memory as a process Memory II. Memory improvement III. Problems with memory 3 systems/stages of Memory: memory the process by which I. Sensory Memory information is - acquired, II. Short -Term Memory - stored,

169 views • 5 slides
Memory networks Zhirong Wu Feb 9th, 2015 Outline motivation Most machine learning algorithms

Memory networks Zhirong Wu Feb 9th, 2015 Outline motivation Most machine learning algorithms

Memory networks Zhirong Wu Feb 9th, 2015 Outline motivation Most machine learning algorithms try to learn a static mapping, and it has been elusive to incorporate memory in the learning. Despite its wide-ranging success in modelling

845 views • 37 slides
Investigating Relational Recurrent Neural Networks with Variable Length Memory Pointer Mahtab

Investigating Relational Recurrent Neural Networks with Variable Length Memory Pointer Mahtab

Investigating Relational Recurrent Neural Networks with Variable Length Memory Pointer Mahtab Ahmed and Robert E. Mercer Department of Computer Science University of Western Ontario London, ON, Canada Introduction Memory based Neural

595 views • 13 slides
1 Memory SoC Persistent Memory-Driven Memory Memory Processor-Centric Memory SoC SoC

1 Memory SoC Persistent Memory-Driven Memory Memory Processor-Centric Memory SoC SoC

1 Memory SoC Persistent Memory-Driven Memory Memory Processor-Centric Memory SoC SoC Computing Computing + Fabric SoC Memory HYPERCONVERGED Exascale EDGE DEVICE SYSTEM Eliminate data movement via shared

400 views • 11 slides
Learning Beyond Finite Memory in Recurrent Networks Of Spiking Neurons Peter Ti no Ashley

Learning Beyond Finite Memory in Recurrent Networks Of Spiking Neurons Peter Ti no Ashley

Learning Beyond Finite Memory in Recurrent Networks Of Spiking Neurons Peter Ti no Ashley J.S. Mills School Of Computer Science University of Birmingham, UK Learning Beyond Finite Memory in Recurrent Networks Of Spiking Neurons Some

488 views • 24 slides
Document Context Neural Machine Translation with Memory Networks Sameen Maruf, Gholamreza Haffari

Document Context Neural Machine Translation with Memory Networks Sameen Maruf, Gholamreza Haffari

Document Context Neural Machine Translation with Memory Networks Document Context Neural Machine Translation with Memory Networks Sameen Maruf, Gholamreza Haffari Faculty of Information Technology Monash University July 17, 2017 1 / 30

901 views • 78 slides
Distributed Transactional Memory for General Networks Gokarna Sharma Costas Busch Srivathsan

Distributed Transactional Memory for General Networks Gokarna Sharma Costas Busch Srivathsan

Distributed Transactional Memory for General Networks Gokarna Sharma Costas Busch Srivathsan Srinivasagopalan Louisiana State University May 24, 2012 Distributed Transactional Memory Transactions run on network nodes They ask for

614 views • 45 slides
Sequential Data with Neural Networks Recurrent Neural Networks Sequential input / output Greg

Sequential Data with Neural Networks Recurrent Neural Networks Sequential input / output Greg

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Sequential Data with Neural Networks Recurrent Neural

303 views • 4 slides
Natural Nervous Systems and the Brain Neural Networks Neural Networks - Brain and Nervous System

Natural Nervous Systems and the Brain Neural Networks Neural Networks - Brain and Nervous System

Natural Nervous Systems and the Brain Neural Networks Neural Networks - Brain and Nervous System von Neumann Bottleneck P M Bottleneck (Bandwidth/Memory Hierarchy) Separation of Processing and Memory State-to-State (Sequential) Prescriptive

498 views • 33 slides
Machines and Virtualization Machines and Virtualization Systems and Networks Jeff Chase Spring

Machines and Virtualization Machines and Virtualization Systems and Networks Jeff Chase Spring

Machines and Virtualization Machines and Virtualization Systems and Networks Jeff Chase Spring 2006 Memory Protection Memory Protection Paging Virtual memory provides protection by: Each process (user or OS) has different virtual memory

716 views • 27 slides
Using Tensor Swapping and NVLink to Overcome GPU Memory Limits with TensorFlow Sam Matzek Deep

Using Tensor Swapping and NVLink to Overcome GPU Memory Limits with TensorFlow Sam Matzek Deep

Using Tensor Swapping and NVLink to Overcome GPU Memory Limits with TensorFlow Sam Matzek Deep learning is memory constrained GPUs have limited memory Neural networks are growing deeper and wider Amount and size of data to process is

397 views • 23 slides
UMBC M A L F T U M B C I O M Y O T R 1 (April 10, 2000 12:36 pm) I E S R C

UMBC M A L F T U M B C I O M Y O T R 1 (April 10, 2000 12:36 pm) I E S R C

Systems Programming 8086/88 Memory Interface II CMPE 310 Memory Address Decoding The processor can usually address a memory space that is much larger than the memory space covered by an individual memory chip. In order to splice a memory device

713 views • 21 slides
Debugging Distributed-Shared-Memory Communication at Multiple Granularities in Networks on Chip

Debugging Distributed-Shared-Memory Communication at Multiple Granularities in Networks on Chip

Debugging Distributed-Shared-Memory Communication at Multiple Granularities in Networks on Chip Bart Vermeulen 1 Kees Goossens 1,2 Siddharth Umrani 2 1 Research, NXP Semiconductors 2 Computer Engineering, Delft University of Technology 2

801 views • 26 slides
1 Brenda Milner 1918 There are Two Major Forms of Long Term Memory Explicit (Declarative)

1 Brenda Milner 1918 There are Two Major Forms of Long Term Memory Explicit (Declarative)

The Study of Memory Has Two Parts: Cellular Mechanisms of Learning (1) The Systems Problem of Memory: Where in the brain is memory stored? and the Biological Basis of Individuality (2) The Molecular Problem of Memory: How is memory stored at

536 views • 8 slides
An Introduction to Neural Networks Long Short Term Memory (LSTM) and the Attention mechanism Ange

An Introduction to Neural Networks Long Short Term Memory (LSTM) and the Attention mechanism Ange

An Introduction to Neural Networks Long Short Term Memory (LSTM) and the Attention mechanism Ange Tato Universit du Qubec Montral Montreal, Canada Agenda v Recurrent Neural Network (RNN) v Long Short Term Memory (LSTM) v

506 views • 27 slides
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks by Kai

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks by Kai

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks by Kai Sheng Tai, Richard Socher, Christopher D. Manning Daniel Perez tuvistavie CTO @ Claude Tech M2 @ The University of Tokyo October 2, 2017

546 views • 22 slides

More recommend