Machine Reading and Reasoning with Neural Program Interpreters - PowerPoint PPT Presentation

Machine Reading and Reasoning with Neural Program Interpreters Sebastian Riedel @riedelcastro Bloomsbury AI Machine Reading

Collaborators Pontus Stenetorp Matko Bosnjak Johannes Welbl (UCL) (UCL) (UCL) Tim Rocktäschel Jason Naradowsky (now at Oxford) (now Johns Hopkins University) 2

Convolutional 2D Knowledge Graph Embeddings, Tim Dettmers et al. “Should we separate meaning from AAAI18, Mon 11:30-12:30, Room 8 The “Classic” NLP Paradigm language?” [Maybe not?] Chris Manning @AKBC 2013 ? ! [Language] [Meaning] [Information Need] 3

End-to-End Reading and Comprehension (Hermann et. al 2015, Seo et al., 2016, Rajpurkar et al., 2016, Weissenborn 2016 … ) ? ! [Language] [Information Need] 4

Machine Reading What city did Tesla move to in 1880? Nicola Tesla Nicola Tesla Nicola Tesla Prague … … … In January 1880, two of Tesla's uncles put together In January 1880, two of Tesla's uncles put together In January 1880, two of Tesla's uncles put together enough money to help him leave Gospi ć for Prague enough money to help him leave Gospi ć for Prague enough money to help him leave Gospi ć for Prague where he was to study. Unfortunately, he arrived too where he was to study. Unfortunately, he arrived too where he was to study. Unfortunately, he arrived too Why was he unable to enroll at the university? late to enroll at Charles-Ferdinand University; he late to enroll at Charles-Ferdinand University; he late to enroll at Charles-Ferdinand University; he never studied Greek, a required subject; and he was never studied Greek, a required subject; and he was never studied Greek, a required subject; and he was illiterate in Czech, another required subject. Tesla did, illiterate in Czech, another required subject. Tesla did, illiterate in Czech, another required subject. Tesla did, however, attend lectures at the university, although, as however, attend lectures at the university, although, as however, attend lectures at the university, although, as arrived too late to enroll an auditor, he did not receive grades for the courses. an auditor, he did not receive grades for the courses. an auditor, he did not receive grades for the courses. 5

6

DEEP LEARNING KILLED THE LINGUISTS BOSTON SCIENTIST CLAIMS 7

BOSTON SCIENTIST IS COMPLETELY WRONG CLAIMS PHILADELPHIA PROFESSOR 8

How to read and reason end-to-end ? 9

Machine Reading and Reasoning Which medical specialty deals with pituitary ACTH Pituitary ACTH hypersecretion ... is a form of hypersecretion? hyperpituitarism characterized by an abnormally high level of ACTH produced by the anterior pituitary … Endocrinology A major organ of the endocrine system , the anterior pituitary is the glandular, anterior lobe that ... The endocrine system is ... ... The field of study dealing with the endocrine system and its disorders is endocrinology , a branch of internal medicine. 10

Machine Reading and Reasoning Isabel uploaded 2 pictures from her phone and 4 How many pictures were in each of the albums? from her camera to facebook. She sorted the pics into 3 different albums with the same amount of pics in each album. 2 11

Can we learn this end-to-end ? 12

Matko Bosnjak Tim Rocktäschel Jason Naradowsky Part 1: Learning to Read and Calculate 13

Machine Reading and Reasoning: Math Isabel uploaded 2 pictures from her phone and 4 How many pictures were in each of the albums? from her camera to facebook. She sorted the pics into 3 different albums with the same amount of pics in each album. 2 14

Differentiable Program Interpreters Isabel uploaded 2 pictures from her phone and 4 How many pictures were in each of the albums? from her camera to facebook. She sorted the pics into 3 different albums with the same amount of pics in each album. Model Program Interpreter Reader 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d Stack Heap Code 3 4 2 Bosnjak et al. ICML 2017 15

Differentiable Program Interpreters Isabel uploaded 2 pictures from her phone and 4 How many pictures were in each of the albums? from her camera to facebook. She sorted the pics into 3 different albums with the same amount of pics in each album. Model Program Interpreter Reader 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d Stack Code Stack Heap Code 3 4 4 2 2 3 Bosnjak et al. ICML 2017 16

Differentiable Program Interpreters Isabel uploaded 2 pictures from her phone and 4 How many pictures were in each of the albums? from her camera to facebook. She sorted the pics into 3 different albums with the same amount of pics in each album. Model Program Interpreter Reader 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d Stack Code Stack Code Stack Heap Code 3 4 4 2 6 2 3 3 Bosnjak et al. ICML 2017 17

Differentiable Program Interpreters Isabel uploaded 2 pictures from her phone and 4 How many pictures were in each of the albums? from her camera to facebook. She sorted the pics into 3 different albums with the same amount of pics in each album. Model Program Interpreter Reader 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d Stack Code Stack Code Stack Code Stack Heap Code 3 4 4 2 6 3 2 3 3 6 Bosnjak et al. ICML 2017 18

Differentiable Program Interpreters Isabel uploaded 2 pictures from her phone and 4 How many pictures were in each of the albums? from her camera to facebook. She sorted the pics into 3 different albums with the same amount of pics in each album. Model Program Interpreter Reader 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d Stack Code Stack Code Stack Code Stack Code Stack Heap Code 3 4 4 2 6 3 2 3 3 6 2 Bosnjak et al. ICML 2017 19

Training Isabel uploaded 2 pictures from her phone and 4 How many pictures were in each of the albums? from her camera to facebook. She sorted the pics into 3 different albums with the same amount of pics in each album. Model Program Interpreter Reader 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d Stack Code Stack Code Stack Code Stack Code Stack Heap Code 3 4 4 2 6 3 2 3 3 6 2 Bosnjak et al. ICML 2017 20

Training Isabel uploaded 2 pictures from her phone and 4 How many pictures were in each of the albums? from her camera to facebook. She sorted the pics into 3 different albums with the same amount of pics in each album. Model Program Interpreter Reader 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d Stack Code Stack Code Stack Code Stack Code Stack Heap Code 3 4 4 2 6 3 2 3 3 6 2 Bosnjak et al. ICML 2017 21

Training Isabel uploaded 2 pictures from her phone and 4 How many pictures were in each of the albums? from her camera to facebook. She sorted the pics into 3 different albums with the same amount of pics in each album. Model Program Interpreter Reader 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d Stack Code Stack Code Stack Code Stack Code Stack Heap Code 3 4 4 2 6 3 2 3 3 6 2 Bosnjak et al. ICML 2017 22

Training Isabel uploaded 2 pictures from her phone and 4 How many pictures were in each of the albums? from her camera to facebook. She sorted the pics into 3 different albums with the same amount of pics in each album. Model Program Interpreter Reader 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 p d Stack Code Stack Code Stack Code Stack Code Stack Heap Code def solve(x): 3 {+|-|%|*} 4 solve(y) 4 2 6 3 2 3 3 6 2 Bosnjak et al. ICML 2017 23