Grid-Wise Control for Multi-Agent Reinforcement Learning in Video - PowerPoint PPT Presentation

Jul 28, 2023 •276 likes •358 views

Grid-Wise Control for Multi-Agent Reinforcement Learning in Video Game AI Lei Han* 1 , Peng Sun* 1 , Yali Du* 2 , Jiechao Xiong 1 , Qing Wang 1 , Xinghai Sun 1 , Han Liu 3 , Tong Zhang 4 1 Tencent AI Lab, Shenzhen, China 2 University of

Grid-Wise Control for Multi-Agent Reinforcement Learning in Video Game AI Lei Han* 1 , Peng Sun* 1 , Yali Du* 2 , Jiechao Xiong 1 , Qing Wang 1 , Xinghai Sun 1 , Han Liu 3 , Tong Zhang 4 1 Tencent AI Lab, Shenzhen, China 2 University of Technology Sydney, Australia 3 Northwestern University, IL, USA 4 Hong Kong University of Science and Technology, Hong Kong, China * Equal contribution Email: leihan.cs@gmail.com
Introduction q Considered Problem • Multi-agent reinforcement learning (MARL) • Grid-world environment (video game) • Challenge Ø flexibly control an arbitrary number of agents Ø while achieving effective collaboration q Existing MARL Approaches • Decentralized learning Ø IQL, IAC (Tan, 1993; Foerster et al., 2017) • Centralized learning Ø CommNet, BicNet (Sukhbaatar et al., 2016; Peng et al., 2017) • Mixture Ø COMA, QMIX, Mean-Field (Foerster et al., 2017; Rashid et al., 2018; Yang et al., 2018) v Unable/instable to deal with variant agent number
GridNet q Architecture • Encoder • Decoder Ø Inputs are represented as an image-like structure Ø Up-sampling to construct an action map Ø An agent will take the action in the grid it occupies Ø Using conv/pooling layers to generate an embedding
GridNet q Algorithms • Can be integrated with many general RL algorithms Ø Q-learning Ø Actor-critic q Properties • Collaboration is natural Ø Stacked convolutional and/or pooling layers provide a large receptive field Ø Each agent is aware of other agents in its neighborhood • Fast parallel exploration Ø Convolutional parameters are shared by all the agents Ø Once an agent takes a beneficial action during its own exploration, the other agents will acquire the knowledge as well • Transferrable policy Ø The trained policy is easy to be transferred to other settings with a various number of agents
Experiments on Battle Games in StarCraft II q Scenarios • 5Immortals vs. 5Immortals ( 5I ) • 3Immortals+2Zealots vs. 3Immortals+2Zealots ( 3I2Z ) • mixed army battle ( MAB ) with a random number of various Zerg units • including Baneling, Zergling, Roach, Hydralisk and Mutalisk. q Training Strategies • Against handcraft policies: random (Rand) , attack-nearest (AN) , hit-and-run (HR) • Against self historic versions: self-play (SP) q Compared Methods • IQL : independent Q-learning [Tan, 1993] • IAC : independent actor-critic [Foerster et al., 2017] • Central-V : centralized value with decentralized policy [Foerster et al., 2017] • CommNet : communication net [Sukhbaatar et al., 2016] q Video link: https://youtu.be/LTcr01iTgZA
Experiments on Battle Games in StarCraft II • On 5I and 3I2Z • Performance (against each other) • Performance (against handcraft policies)
Experiments on Battle Games in StarCraft II q Learned Tactics • Transferability On 5I and 3I2Z • Directly apply the trained policy to maps with more agents • 10I , 20I , 5I5Z , 10I10Z • Performance On MAB • CommNet and Central-V cannot be applied
Thanks! Poster at Pacific Ballroom #243 Jun 11 th , 6:30 pm

Recommend

Multi-agent learning Multi-agent reinforcement learning Gerard Vreeswijk , Intelligent Systems

Multi-agent learning Multi-agent reinforcement learning Multi-agent learning Multi-agent reinforcement learning Gerard Vreeswijk , Intelligent Systems Group, Computer Science Department, Faculty of Sciences, Utrecht University, The Netherlands.

752 views • 21 slides

REINFORCEMENT LEARNING IN MULTI-AGENT SYSTEMS MACHINE LEARNING MEETUP DR. ANA PELETEIRO

REINFORCEMENT LEARNING IN MULTI-AGENT SYSTEMS MACHINE LEARNING MEETUP DR. ANA PELETEIRO RAMALLO 29-08-2016 TABLE OF CONTENTS MULTI-AGENT SYSTEMS GAME THEORY REINFORCEMENT LEARNING MULTI-AGENT LEARNING 2 ZALANDO Our purpose: to Zalando

1.45k views • 20 slides

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning Kyunghwan Son , Daewoo Kim, Wan Ju Kang, David Hostallero, Yung Yi School of Electrical Engineering, KAIST Cooperative Multi-Agent Reinforcement

662 views • 8 slides

Multi-agent reinforcement learning for new generation control systems Manuel Graa 1 , 2 ; Borja

Multi-agent reinforcement learning for new generation control systems Manuel Graa 1 , 2 ; Borja Fernandez-Gauna 2 1 ENGINE centre, Wroclaw Technological University; 2 Computational Intelligence Group (www.ehu.eus/ccwintco) University of the

1.31k views • 92 slides

ROMA: Multi-Agent Reinforcement Learning with Emerging Roles Tonghan Wang, Heng Dong, Victor

ROMA: Multi-Agent Reinforcement Learning with Emerging Roles Tonghan Wang, Heng Dong, Victor Lesser, Chongjie Zhang Tsinghua University, UMass Amherst Multi-Agent Systems Robot Football Game Multi-Agent Assembly One Major Challenge of

426 views • 30 slides

YunQi 2050 - DRL Session Communication in Multi-agent Reinforcement Learning Ying Wen

YunQi 2050 - DRL Session Communication in Multi-agent Reinforcement Learning Ying Wen Department of Computer Science, University College London MediaGamma Ltd. ying.wen@cs.ucl.ac.uk 30 May, 2018 Multi-agent in Real-World Human

402 views • 27 slides

Multi-agent learning Gerard Vreeswijk , Intelligent Systems Group, Computer Science Department,

Multi-agent learning Multi-agent reinforcement learning Multi-agent reinfo rement lea rning Multi-agent learning Gerard Vreeswijk , Intelligent Systems Group, Computer Science Department, Faculty of Sciences, Utrecht University, The

528 views • 21 slides

Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Imitation Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Reinforcement learning Agent state reward action S t R t A t R t+ 1 Environment S t+ 1

1.07k views • 52 slides

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning

DeepMind Self-Learning Atari Agent Human - level control through deep reinforcement learning Nature Vol 518, Feb 26, 2015 The Deep Mind of Demis Hassabis Backchannel / Medium.com interview with David Levy Advanced Topics:

989 views • 68 slides

Model Based Reinforcement Learning Katerina Fragkiadaki Model Anything the agent can use to

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Model Based Reinforcement Learning Katerina Fragkiadaki Model Anything the agent can use to predict how the environment will respond to its actions,

826 views • 65 slides

Multi-armed bandit problem and its applications in reinforcement learning Pietro Lovato Ph.D.

University of Verona 28/01/2013 Multi-armed bandit problem and its applications in reinforcement learning Pietro Lovato Ph.D. Course on Special Topics in AI: Intelligent Agents and Multi-Agent Systems Overview Introduction: Reinforcement

628 views • 25 slides

Decentralized Stochastic Approximation, Optimization, and Multi-Agent Reinforcement Learning

Decentralized Stochastic Approximation, Optimization, and Multi-Agent Reinforcement Learning Justin Romberg, Georgia Tech ECE CAMDA/TAMIDS Seminar, Texas A& M College Station, Texas Streaming live from Atlanta, Georgia March 16, 2020

689 views • 64 slides

TextMed: A Multi-Agent System with Reinforcement Learning Agents for Biomedical Text Mining

TextMed: A Multi-Agent System with Reinforcement Learning Agents for Biomedical Text Mining Michael Camara Janyl Jumadinova Oliver Bonham-Carter September 9, 2015 Big Data Biomedical Research PubMed: U.S. National Library of Medicine

346 views • 31 slides

Decentralized Non- Communicating Multi-agent Collision Avoidance with Deep Reinforcement

Decentralized Non- Communicating Multi-agent Collision Avoidance with Deep Reinforcement Learning By Yu Fan Chen, Miao Liu, Michael Everett, and Jonathan P . How Presenter: Jared Choi Motivation Finding a path Computationally

808 views • 44 slides

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides

MARKOV GAMES A framework for multi-agent reinforcement learning Shen (Sean) Chen Review on

MARKOV GAMES A framework for multi-agent reinforcement learning Shen (Sean) Chen Review on MDPs An MDP is defined by a set of states, S, and actions, A. Transition function, T: S A PD(S), where PD(S) represents discrete prob

287 views • 17 slides

Actor-Attention-Critic for Multi-Agent Reinforcement Learning Shariq Iqbal and Fei Sha Outline

Actor-Attention-Critic for Multi-Agent Reinforcement Learning Shariq Iqbal and Fei Sha Outline Establish a baseline approach to MARL Demonstrate how recent approaches improve on said baseline through sharing information between

428 views • 11 slides

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring environment. Interactions with environment: action state Agent Environment reward Problem: find action policy that maximizes cumulative reward

828 views • 66 slides

Independent vs. Joint Estimation in Multi Agent Iterative Learning Control Angela Schoellig,

Independent vs. Joint Estimation in Multi Agent Iterative Learning Control Angela Schoellig, Javier Alonso Mora and Raffaello DAndrea Institute for Dynamic Systems and Control ETH Zrich, Switzerland 1 Control and Decision Conference

416 views • 16 slides

Multi-Agent Adversarial Inverse Reinforcement Learning Lantao Yu, Jiaming Song, Stefano Ermon

Multi-Agent Adversarial Inverse Reinforcement Learning Lantao Yu, Jiaming Song, Stefano Ermon Department of Computer Science, Stanford University Contact: lantaoyu@cs.stanford.edu <latexit

511 views • 24 slides

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides

The Reinforcement Learning Problem Robert Platt Northeastern University Agent Action Agent

The Reinforcement Learning Problem Robert Platt Northeastern University Agent Action Agent World Observation Reward On a single time step, agent does the following: 1. observe some information 2. select an action to execute 3. take note

374 views • 11 slides

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides