Deep Reinforcement Learning with a Natural Language Action Space - PowerPoint PPT Presentation

Aug 17, 2022 •449 likes •575 views

Deep Reinforcement Learning with a Natural Language Action Space Authors: Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng and Mari Ostendorf Presented by: Victor Ge Background Motivation How to do credit assignment when

Deep Reinforcement Learning with a Natural Language Action Space Authors: Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng and Mari Ostendorf Presented by: Victor Ge
Background
Motivation ● How to do credit assignment when the action space is discrete and potentially unbounded. ● I.e. human-computer dialog systems, tutoring systems, and text-based games.
Q-learning architectures
Deep Reinforcement Relevance Network (DRRN) ● Factorize DQN into state representation and action representation. ● Interaction function – can be inner product, bilinear operation, nonlinear function, etc. ● In experiments, inner product and bilinear operation give similar results. ● Using nonlinear function (i.e. DNN) degrades performance.
Details ● Bag of words text embedding ● 1-2 hidden layers ● Experience replay buffer ● Softmax action selection:
Experiments – text-based games ● Parser-based games can be reduced to choice- based games if there is a finite number of phrases that the parser accepts.
Experiments – text-based games
Experiments – text-based games ● Human baselines: ● "Saving John": -5.5 ● "Machine of Death": 16.0
Experiments – paraphrased actions ● Question: Is DRRN memorizing the right action? ● State space is small (<1000) ● Replace 81.4% of action descriptions with human paraphrased descriptions. ● Standard 4-gram BLEU score between paraphrased and original actions is 0.325 ● DRRN gets 10.5 average reward on paraphrased game vs 11.2 for original "Machine of Death" game
Experiments – paraphrased actions
Experiments – paraphrased actions

Recommend

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides

A Survey of Reinforcement Learning Informed by Natural Language Luketina et al., IJCAI 2019

A Survey of Reinforcement Learning Informed by Natural Language Luketina et al., IJCAI 2019 Maria Fabiano Outline 1. Motivation 2. Background 3. Current Use of Natural Language in RL 4. Trends for Natural Language in RL 5. Future Work

266 views • 22 slides

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin Clark Deep Learning for NLP 5 years ago No Seq2Seq No Attention No large-scale QA/reading comprehension datasets No TensorFlow or

1.41k views • 92 slides

Natural Language Processing with Deep Learning CS224N/Ling284 Lecture 15: Natural Language

Natural Language Processing with Deep Learning CS224N/Ling284 Lecture 15: Natural Language Generation Christopher Manning Announcements Thank you for all your hard work! We know Assignment 5 was tough and a real challenge to do

1.24k views • 87 slides

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides

Towards Interpretable Deep Learning for Natural Language Processing Roy Schwartz University of

Towards Interpretable Deep Learning for Natural Language Processing Roy Schwartz University of Washington & Allen Institute for Artificial Intelligence December 2018 1 / 37 ( Deep-Learning -Based) AI Today 2 / 37 3 / 37 Deep Learning

1.63k views • 140 slides

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides

Deep Learning for Text analysis Jan Platos 2018-09-09 Table of Contents Natural Language

Deep Learning for Text analysis Jan Platos 2018-09-09 Table of Contents Natural Language Processing Human Language Properties Deep Learning in NLP Representation of the meaning of a word Word2vec Language Modeling n-Gram Language model

841 views • 73 slides

Language as an Abstraction for Hierarchical Deep Reinforcement Learning Paper Authors: Yiding

Language as an Abstraction for Hierarchical Deep Reinforcement Learning Paper Authors: Yiding Jiang, Shixiang Gu, Kevin Murphy, Chelsea Finn Problem Overview Learning a variety of compositional , long horizon skills while being able to

364 views • 34 slides

Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference

Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference Authors Boyuan Pan, Yazheng Yang, Zhou Zhao, Yueting Zhuang, Deng Cai, Xiaofei He Organization Zhejiang University, China What is Natural

468 views • 23 slides

Compositional Methods for Learning and Inference in Deep Probabilistic Programs Jan-Willem

Compositional Methods for Learning and Inference in Deep Probabilistic Programs Jan-Willem van de Meent Eli Sennesh Sam Stites Hao Wu Heiko Zimmermann Deep Learning Success Stories Computer Vision Natural Language Reinforcement

710 views • 40 slides

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides

Deep Learning for Natural Language Processing (in 2 hours) Eneko Agirre

Deep Learning for Natural Language Processing (in 2 hours) Eneko Agirre http://ixa2.si.ehu.eus/eneko IXA NLP group http://ixa.eus @eagirre Contents Introduction to Deep learning Deep Learning ~ Learning Representations Text as

1.58k views • 156 slides

deep learning for natural language processing . Sergey I. Nikolenko 1,2,3 Deep Machine

deep learning for natural language processing . Sergey I. Nikolenko 1,2,3 Deep Machine Intelligence Workshop Moscow, June 4, 2016 1 Laboratory for Internet Studies, NRU Higher School of Economics, St. Petersburg 2 Laboratory of Mathematical

815 views • 56 slides

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides

Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Introduction to Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 1 is due next Wednesday! Remember that Monday is a holiday, so no office hours 2. Remember to start forming final project

880 views • 48 slides

Language Understanding for Text-based Games Using Deep Reinforcement Learning Karthik

Language Understanding for Text-based Games Using Deep Reinforcement Learning Karthik Narasimhan, Tejas Kulkarni, Regina Barzilay MIT Text-based games (State 1: The old bridge) You are standing very close to the bridges eastern

522 views • 34 slides

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides

Deep learning for natural language processing A short primer on deep learning Benoit Favre <

Deep learning for natural language processing A short primer on deep learning Benoit Favre < benoit.favre@univ-mrs.fr > Aix-Marseille Universit, LIF/CNRS 20 Feb 2017 Benoit Favre (AMU) DL4NLP: deep learning 20 Feb 2017 1 / 25 Deep

530 views • 25 slides

Deep Learning Applications in Natural Language Processing Jindich Libovick December 5, 2018

Deep Learning Applications in Natural Language Processing Jindich Libovick December 5, 2018 B4M36NLP Introduction to Natural Language Processing Charles University Faculty of Mathematics and Physics Institute of Formal and Applied

930 views • 42 slides

Deep Learning for Natural Language processing Jindich Libovick March 1, 2017 Introduction

Deep Learning for Natural Language processing Jindich Libovick March 1, 2017 Introduction to Natural Language Processing Charles University Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics unless otherwise

1.33k views • 103 slides

Deep learning for natural language processing Introduction to natural language processing

Deep learning for natural language processing Introduction to natural language processing Aix-Marseille Universit, LIF/CNRS 20 Feb 2017 Benoit Favre (AMU) DL4NLP: NLP intro 20 Feb 2017 1 / 24 Benoit Favre < benoit.favre@univ-mrs.fr >

920 views • 24 slides

Deep Learning for NLP Kiran Vodrahalli Feb 11, 2015 Overview What is NLP? Natural

Deep Learning for NLP Kiran Vodrahalli Feb 11, 2015 Overview What is NLP? Natural Language Processing We try to extract meaning from text: sentiment, word sense, semantic similarity, etc. How does Deep Learning relate? NLP

2.02k views • 124 slides