Learning to Interpret Natural Language Navigation Instructions from - PowerPoint PPT Presentation

Learning to Interpret Natural Language Navigation Instructions from Observations Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Problem ● Follow free-form natural-language navigation instructions. ● No prior linguistic knowledge. ● Training data: {(e 1 , a 1 , w 1 ), (e 2 , a 2 , w 2 ), …, (e n , a n , w n )} ○ e i - natural language instruction ○ a i - action sequence ○ w i - current state of the world ● Goal: (e j , w j ) -> a j ● Three virtual worlds with same elements in different configurations. Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Example “Go towards the coat rack and take a left at the coat rack. go all the way to the end of the hall and this is 4.” “turn so that the wall is on your right side. walk forward once. turn left. walk forward twice.” “foward to the fish. first left. go tot [sic] the end.” “Place your back to the wall of the ’T’ intersection. Turn right. Go forward one segment to the intersection with the yellow-tiled hall. This intersection contains a hatrack. Turn left. Go forward two segments to the end of the hall. This is Position 4.” Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Motivation ● Interpretation of human instructions ● Learning language directly from observation ● Requires both linguistic and spatial context ● Grounding of natural language ● Error recovery Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Approach w i a i p i e i p ’ i (refined) e j p j w j a j Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Navigation Plans Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Navigation Plans Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Navigation Plans Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Lexicon Learning Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Lexicon Learning ● Score(w, g) = p(g|w) - p(g|¬w) ○ w - n-gram ○ g - graph ● A good (w, g) pair means that w should be indicative of g appearing. ● Ubiquitous g should be penalized. Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Refining Plans for (e i , p i ) repeat select highest-scoring (w, g) such that w and g appear in e i and p i , respectively remove w from e i and mark all occurrences of g in p i until no possible selection of (w, g) p ’ i <- p i - unmarked nodes end for Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Next Steps w i a i p i e i p ’ i (refined) e j p j w j a j Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Dataset Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Evaluation - Generating Navigation Plans ● Gold-standard: hand-annotated navigation plans ● Partial correctness metric ● Leave-one-map-out for ‘refined landmarks plans’ Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Evaluation - Semantic Parser ● Leave-one-map-out approach ● Precision of training data is more important than recall Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Evaluation - Executing Navigation Plans Percentage of trials reaching the correct destination Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Example Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Critique ● Only the instruction used for generating navigation plan ○ Can we use the current view of the follower to assist in generating next navigation plan? ○ “Turn so that the wall is on your right side.” ● How are ambiguous words handled in lexicon learning? ● How do the results change with change in hyperparameters e.g. k , t , and n ? ○ How were the mentioned choices made? ● Why does refining of plans help in error recovery? ● How well does it generalize? ○ What if new words are seen during testing? What if a new world with different items is encountered?

● What can be some other evaluation metrics? Discussion ● How about an end-to-end model? Neural Networks? ● Will prior linguistic knowledge be helpful?