Question asking as program induction Anselm Rothe WITH Todd - PowerPoint PPT Presentation

Question asking 
 as 
 program induction Anselm Rothe

WITH Todd Gureckis Brenden Lake

Computers are useless. They can only give you answers. (attributed to) Pablo Picasso

What does it take Computers are to build a machine 
 useless. They can only that asks good give you answers. questions? (attributed to) Pablo Picasso

What does it take to build a machine 
 that asks good questions? Representing questions • as programs that, when executed on the state of the world, output an answer Anselm Rothe - Question asking as program induction 5

What does it take to build a machine 
 Key ingredients 
 that asks good questions? Generativity • Compositionality • Representing questions • Informativeness • as programs that, when executed on the Simplicity • state of the world, output an answer Anselm Rothe - Question asking as program induction 6

HUMAN QUESTIONS random samples A B C D E F A B C D E F 1x We need a task that allows people to intuitively ask interesting 1 1 questions and is still amenable to formal modeling 2 2 Identify the hidden 1x 3 3 gameboard! 4 4 5 5 1x 6 6 Possible ships Hidden gameboard Revealed gameboard G enerative model Current data/context Goal World model Ambiguous context Rothe, Lake, & Gureckis 2016, CogSci 
 Rothe, Lake, & Gureckis 2018, Computational Brain & Behavior Anselm Rothe - Question asking as program induction 7

HUMAN QUESTIONS random samples random A B C D E F A B C D E F 1x samples 1 1 A B C D E F A B C D E F 2 2 Identify the hidden 1x 1 1 1x 3 3 gameboard! 2 2 Identify the hidden 4 4 1x 3 3 5 5 gameboard! 1x 4 4 6 6 5 5 1x Possible ships Hidden gameboard Revealed gameboard 6 6 Possible ships Hidden gameboard Revealed gameboard G enerative model Current data/context Goal World model Ambiguous context G Rothe, Lake, & Gureckis 2016, CogSci 
 Rothe, Lake, & Gureckis 2018, Computational Brain & Behavior Anselm Rothe - Question asking as program induction 8

HUMAN QUESTIONS random samples A B C D E F A B C D E F 1x 1 1 A B C D E F 2 2 Identify the hidden 1x 1 1x 3 3 gameboard! 2 Identify the hidden People were dropped 4 4 1x 3 5 5 gameboard! 1x into the middle of a 4 6 6 5 1x game and were given Possible ships Hidden gameboard Revealed gameboard 6 the ‘magic’ opportunity Revealed gameboard to ask whatever they want * G enerative model Current data/context Goal Ambiguous context * only one-word-answer questions, 
 G type in your question no combination of questions Is the red ship horizontal? | Anselm Rothe - Question asking as program induction 9

HUMAN QUESTIONS Context Example questions from people A B C D E F At what location is the top left part of the purple ship? 1 What is the location of one purple tile? 2 Is the blue ship horizontal? 3 Is the red ship 2 tiles long? 4 Is the purple ship horizontal? 5 Is the red ship horizontal? 6 ... Rothe, Lake, & Gureckis 2016, CogSci 
 Rothe, Lake, & Gureckis 2018, Computational Brain & Behavior Anselm Rothe - Question asking as program induction 10

Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Trial 6 A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F 1 1 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 5 5 5 5 5 5 6 6 6 6 6 6 Trial 7 Trial 8 Trial 9 Trial 10 Trial 11 Trial 12 A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F 1 1 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 5 5 5 5 5 5 6 6 6 6 6 6 Trial 13 Trial 14 Trial 15 Trial 16 Trial 17 Trial 18 A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F 1 1 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 5 5 5 5 5 5 6 6 6 6 6 6 • 40 MTurk participants • 605 human questions Rothe, Lake, & Gureckis 2016, CogSci 
 Rothe, Lake, & Gureckis 2018, Computational Brain & Behavior Anselm Rothe - Question asking as program induction 11

Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Trial 6 A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F 1 1 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 Key ingredients 
 5 5 5 5 5 5 6 6 6 6 6 6 Trial 7 Trial 8 Trial 9 Trial 10 Trial 11 Trial 12 A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F 1 1 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 5 5 5 5 5 5 6 6 6 6 6 6 Generativity • Trial 13 Trial 14 Trial 15 Trial 16 Trial 17 Trial 18 A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F A B C D E F 1 1 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 5 5 5 5 5 5 Compositionality • 6 6 6 6 6 6 • 15% of participants’ questions Informativeness • were only asked in a single Simplicity • context • Our model needs the ability to generate novel questions Anselm Rothe - Question asking as program induction 12

“How long is the blue ship?” Key ingredients 
 “Is the blue ship less then 4 tiles?” Generativity • “Are there any ships with 4 tiles?” Compositionality • “Does the blue ship have 3 tiles?” Informativeness • “Does the red ship have more 
 tiles than the blue ship?” Simplicity • “Are all 3 ships the same size?” more red 3 size ship less blue 4 Anselm Rothe - Question asking as program induction 13

COMPOSITIONALITY IN QUESTION STRUCTURE size blue (size Blue) • Questions are represented as programs that, when executed on the state of the world, output an answer Anselm Rothe - Question asking as program induction 14

COMPOSITIONALITY IN QUESTION STRUCTURE size blue (size Blue) equal (> (size Blue) (size Red)) more red (= (size Blue) (size Red)) • Questions are represented as programs that, when executed on the state of the world, output an answer Anselm Rothe - Question asking as program induction 15

COMPOSITIONALITY IN QUESTION STRUCTURE size blue (size Blue) equal (> (size Blue) (size Red)) more red (= (size Blue) (size Red)) orientation (= (orientation Blue) (orientation Red)) “Are the blue ship and the red ship parallel?” • Questions are represented as programs that, when executed on the state of the world, output an answer Anselm Rothe - Question asking as program induction 16

COMPOSITIONALITY IN QUESTION STRUCTURE How many ships are three tiles long? Are any ships 3 tiles long? Are all ships three tiles long? ( + ( > ( = ( map ( + ( + ( lambda ( map ( map x ( lambda ( lambda ( = x x ( size x ) ( = ( = 3 ( size x ) ( size x ) ) 3 3 ) ) ) ( set Blue Red Purple ) ) ) ) ( set Blue Red Purple ) ( set Blue Red Purple ) ) ) ) ) ) 0 3 ) ) • Questions are represented as programs that, when executed on the state of the world, output an answer Anselm Rothe - Question asking as program induction 17

A GRAMMAR OF QUESTIONS O → H N → 0 A → B (boolean) O → V ... A → N (number) O → (orient S) b N → 10 A → C (color) N → (+ N N) A → O (orientation) N → (+ B B) A → L (location) L → A1 N → (+ setN) ... N → (+ setB) L → F6 N → (– N N) L → (topleft S) b N → (size S) b B → TRUE L → (bottomright S) b N → (row L) B → FALSE L → (draw setL) * N → (col L) B → (not B) B → (and B B) setB → (map fxB setS) → B → (or B B) fxB → ( λ x B) B → (= B B) C → S (ship color) B → (= N N) C → Water setN → (map fxN setS) B → (= O O) C → (color L) b fxN → ( λ x N) B → (= setN) B → ( > N N) S → Blue B → (touch S S) b setS → (set Blue Red Purple) S → Red S → Purple S → x λ setL → (set A1 ... F6) setL → (shipTiles S) b * setL → (map fxL setS) fxL → ( λ x L) Rothe, Lake, & Gureckis 2017, NIPS 18

Key ingredients 
 ✔ Question space 
 Generativity • as defined by grammar ✔ Compositionality • ? ? ? Informativeness • ? ? Simplicity • Generating questions • Drawing samples from grammar • Evolutionary search cost / fitness function 19

Key ingredients 
 Generativity • Compositionality • Informativeness • Simplicity • Anselm Rothe - Question asking as program induction 20

human questions Key ingredients 
 A B C D E F 1 2 Generativity • 3 4 5 Compositionality • 6 Informativeness • Simplicity • Anselm Rothe - Question asking as program induction 21

human questions Key ingredients 
 A B C D E F Using a genetic algorithm 1 with EIG as fitness function 2 Generativity • 3 to search for the “best 4 question” for a given context 5 ? Compositionality • 6 Informativeness • Simplicity • Anselm Rothe - Question asking as program induction 22