Continually Improving Grounded Natural Language Understanding - PowerPoint PPT Presentation

[Thomason et al., IJCAI’16] Technical Contributions ● Ensemble SVMs over multi-modal squishy object features to perform press haptic language grounding . ● Get language labels from natural language game with human users 49

[Thomason et al., IJCAI’16] 50

[Thomason et al., IJCAI’16] Experiments Playing I Spy vs multi-modal vision only 51

[Thomason et al., IJCAI’16] Experiments Playing I Spy Four folds of objects for four rounds of training. 52

[Thomason et al., IJCAI’16] Problematic I Spy Object Future : Be mindful of object novelty both for the learning algorithm and for human users. 53

[Thomason et al., IJCAI’16] Polysemy Induction and Synonymy Detection (IJCAI’17) NLP Robotics Papers Human- before Robot proposal Dialog Improving Learning Semantic Parsing Groundings with through Dialog Human Interaction (IJCAI’15) (IJCAI’16) Dialog 54

[Thomason et al., IJCAI’17] Polysemy Induction and Synonymy Detection NLP Robotics Papers Human- before Robot proposal Dialog Improving Learning Semantic Parsing Groundings with through Dialog Human Interaction (IJCAI’15) (IJCAI’16) Dialog 55

[Thomason et al., IJCAI’17] Unsupervised Word Synset Induction “chinese grapefruit” “kiwi” “kiwi vine” 56

[Thomason et al., IJCAI’17] Unsupervised Word Synset Induction “kiwi”, “chinese grapefruit”, “kiwi vine” “kiwi” “kiwi” 57

[Thomason et al., IJCAI’17] Polysemy Induction and Synonymy Detection NLP Robotics Papers Human- before Robot proposal Dialog Improving Learning Semantic Parsing Groundings with through Dialog Human Interaction (IJCAI’15) (IJCAI’16) Dialog 58

Faster Object Exploration for Grounding (AAAI’18) NLP Robotics Papers Human- since Jointly Improving Robot Parsing & Perception proposal Dialog ( in submission ) Learning Groundings with Opportunistic Active Learning (CoRL’17) Dialog 59

[Thomason et al., AAAI’18] Faster Object Exploration for Grounding (AAAI’18) NLP Robotics Papers Human- since Jointly Improving Robot Parsing & Perception proposal Dialog ( in submission ) Learning Groundings with Opportunistic Active Learning (CoRL’17) Dialog 60

[Thomason et al., AAAI’18] Exploratory Behaviors 104s to explore an object once. +hold (5.7s) 520s to explore an object five times. +look 4.5 hours to (0.8s) fully explore 32 objects. 61

[Thomason et al., AAAI’18] Guiding Exploratory Behaviors rigid: squishy? press press? haptic look look? VGG 62

[Thomason et al., AAAI’18] Guiding Exploratory Behaviors rigid: squishy press press haptic haptic look look VGG VGG 63

[Thomason et al., AAAI’18; Mikolov et al., NIPS’13] Guiding Exploratory Behaviors d2 tall similarity(rigid, squishy) = cos ( � ) rigid squishy � mug d1 64

[Thomason et al., AAAI’18] Shared Structure: Embeddings and Features 2D-projection of 2D-projection of word embeddings behavior context features 65

[Thomason et al., AAAI’18] Guiding Exploratory Behaviors using Embeddings Surrogate reliability Nearest Reliability weights for weights for new word-embedding trained neighbor classifiers for q predicates to q classifiers p 66

[Thomason et al., AAAI’18] Technical Contributions ● Reduce exploration time when learning a target new word . ● Use word embeddings and human annotations to guide behaviors. 67

[Thomason et al., AAAI’18] Results Color predicates Weight predicates Contents predicates Agreement with Gold Number of Behaviors Number of Behaviors Number of Behaviors (dotted lines show standard error) 68

[Thomason et al., AAAI’18] Other Findings ● Human annotations help; grasp lift on held lifted “how would you tell if an table drop lower object is tall ?” look hold press push ● Human annotations + word embeddings work better than either alone. 69

[Thomason et al., AAAI’18] Faster Object Exploration for Grounding (AAAI’18) NLP Robotics Papers Human- since Jointly Improving Robot Parsing & Perception proposal Dialog ( in submission ) Learning Groundings with Opportunistic Active Learning (CoRL’17) Dialog 70

[Thomason et al., CoRL’17] Faster Object Exploration for Grounding (AAAI’18) NLP Robotics Papers Human- since Jointly Improving Robot Parsing & Perception proposal Dialog ( in submisison ) Learning Groundings with Opportunistic Active Learning Dialog 71

[Thomason et al., CoRL’17] Active Learning for Perceptual Questions The object for which the predicate classifier is least sure of the predicted label. d( bottle , ) = 0.8 d( bottle , ) = -0.6 d( bottle , ) = 0.4 d( bottle , ) = -0.2 72

[Thomason et al., CoRL’17] Active Learning for Perceptual Questions empty bottle sensorimotor sensorimotor w p,c w p,c context context lift-haptics ? look-shape 0.6 lift-audio ? look-vgg 0.5 ... ... ... ... look-vgg ? lower-haptics 0.02 73

[Thomason et al., CoRL’17] Active Learning for Perceptual Questions Ask for a label with probability proportional to un confidence in least confident training object. Ask for a positive label for any predicate we have insufficient data for. 74

[Thomason et al., CoRL’17] Active Learning for Perceptual Questions Ask for a label with “Could you use the probability proportional to word bottle when un confidence in least describing this object?” confident training object. Ask for a positive label for “Can you show me any predicate we have something empty?” insufficient data for. 75

[Thomason et al., CoRL’17] 76

[Thomason et al., CoRL’17] Technical Contributions “A full, yellow bottle.” ● Introduce an opportunistic active learning strategy for “Would you getting high-value labels. describe this object as full?” ● Show that off-topic questions “Show me something red.” improve performance. 77

[Thomason et al., CoRL’17] Experiments with Object Identification “Would you “Show me vs describe this something red.” object as full?” Baseline Agent Inquisitive Agent 78

[Thomason et al., CoRL’17] Results “Would you Baseline Agent describe this Rated less annoying. object as full?” Inquisitive Agent “Show me something red.” Correct object more often. Rated better for real-world use. 79

[Thomason et al., CoRL’17] Faster Object Exploration for Grounding (AAAI’18) NLP Robotics Papers Human- since Jointly Improving Robot Parsing & Perception proposal Dialog ( in submission ) Learning Groundings with Opportunistic Active Learning Dialog 80

[ in submission ] Faster Object Exploration for Grounding (AAAI’18) NLP Robotics Papers Human- Jointly Improving since Robot Parsing & proposal Perception Dialog ( in submission ) Learning Groundings with Opportunistic Active Learning (CoRL’17) Dialog 81

Human-Robot Dialog Natural Language Perception Understanding Models Utterance Annotated Semantic User World Parser Knowledge Robot Meaning Behavior Question Agent Belief Dialog Dialog Policy Agent 82

[ in submission ] Jointly Improving Parsing and Perception “Move a rattling container from lounge by the conference room to Bob’s office.” 83

[ in submission ] Experiments via Amazon Mechanical Turk Training x 113 Object / Induced Predicate Training Pairs Labels Semantic Perception Parser Models 84

[ in submission ] Experiments via Amazon Mechanical Turk Testing - Baseline x ~45 Semantic Perception Parser Models 85

[ in submission ] Experiments via Amazon Mechanical Turk Testing - Perception x ~45 Object / Predicate Labels Semantic Perception Perception Parser Models Models 86

[ in submission ] Getting Object/Predicate Labels in Dialog Object / Predicate Labels Perception Models 87

[ in submission ] Getting Object/Predicate Labels in Dialog Object / Predicate Labels Perception Models 88

[ in submission ] Experiments via Amazon Mechanical Turk Testing - Parsing + Perception x ~45 Object / Induced Predicate Training Pairs Labels Semantic Perception Perception Parser Models Models 89

[ in submission ] Inducing New Training Examples from Dialog Induced Training Pairs Semantic Parser 90

[ in submission ] Inducing New Training Examples from Dialog Expect whole command Expect goal task : navigate goal : room_3 91

[ in submission ] Inducing New Training Examples from Dialog Induced Utterance/Denotation Pairs “go to the middle lab” navigate(room_3) “the lab in the middle” room_3 92

[ in submission ] Natural Language Understanding Natural Language Understanding Perception “the lab in the Models room_3 middle” Semantic Parser Annotated World Knowledge something that is a lab room_3, room_7, ... something that is both a lab and is central room_3 something that is central room_3, room_1, ... ... ... 93

[ in submission ] Inducing New Training Examples from Dialog Induced Induced Parser Semantic Utterance/Denotation Training Data Parser Pairs “go to the middle lab” Perception “go to the middle lab” navigate(lab+central ) Models navigate(room_3) “the lab in the middle” “the lab in the middle” Annotated lab+central room_3 World Knowledge 94

[Mikolov et al., NIPS’13; in submission ] Using Embeddings for Out-of-Vocabulary Words Induced Training Pairs task : deliver item : coffee “deliver “deliver person : bob java to bob” java to bob” Word Semantic Embeddings Parser “deliver” -> “bring” “java” -> “coffee” 95

[Mikolov et al., NIPS’13; in submission ] Using Embeddings to Find Perception Words d2 white tall tower long � d1 96

[ in submission ] Technical Contributions Induced Object / Training Predicate Pairs Labels ● Improve both parsing and perception from conversations. Semantic Perception Parser Models d2 white ● Use word embeddings to guide tall search for synonyms and tower long novel perceptual predicates . 97 d1

[ in submission ] Experiments via Amazon Mechanical Turk Parsing + Untrained Baseline Perception Training Perception Training Object / Induced Object / Predicate Training Predicate Labels Pairs Labels Semantic Perception Semantic Perception Semantic Perception Parser Models Parser Models Parser Models 98

[ in submission ] Metric - Semantic F1 99

[ in submission ] Results - Navigation Task Quantitative - Semantic F1 Qualitative - Usability Rating 100