learning temporal point processes via reinforcement
play

Learning Temporal Point Processes via Reinforcement Learning Shuang - PowerPoint PPT Presentation

Sep 16, 2023 •278 likes •375 views

Learning Temporal Point Processes via Reinforcement Learning Shuang Li 1 , Shuai Xiao 2 , Shixiang Zhu 1 , Nan Du 3 , Yao Xie 1 , Le Song 1,2 1 Georgia Institute of Technology 2 Ant Financial 3 Google Brain Motivation 1:30 pm 1:18 pm 1:00 pm

  1. Learning Temporal Point Processes via Reinforcement Learning Shuang Li 1 , Shuai Xiao 2 , Shixiang Zhu 1 , Nan Du 3 , Yao Xie 1 , Le Song 1,2 1 Georgia Institute of Technology 2 Ant Financial 3 Google Brain

  2. Motivation 1:30 pm 1:18 pm 1:00 pm Dinner together? Funny joke Cool picture … ( " ∈ 0 ∪ , - ! time David " # " $ " % " ? History & ' u Event data : tweets/retweets, crime events, earthquakes, patient visits to hospital, finance transactions, …. u Learn temporal pattern of event data. u Event time is random u Complex dependency structure

  3. � Point Process Model Intensity function u 𝜇 𝑢|ℋ + 𝑒𝑢 = 𝔽 𝑂[𝑢, 𝑢 + 𝑒𝑢)]|ℋ + where 𝑂[𝑢, 𝑢 + 𝑒𝑢) is the number of events falling in the set [𝑢, 𝑢 + 𝑒𝑢) . Point Process Temporal Pattern 𝝁 𝜾 𝒖|𝓘 𝒖 Poisson constant ! time David Da " & " # " $ " % Inhomogeneous 𝜇 4 (𝑢) Poisson ! time David Da " # " $ " % " & " ' " # " $ " % " & " ' Hawkes 𝜈 + 𝛽 8 exp −|𝑢 − 𝑢 = | ! time + ? ∈𝓘 A Da David " # " $ " %

  4. � � Traditional Maximum-Likelihood Framework u Model conditional intensity 𝜇 4 𝑢|ℋ + Model- as a parametric/non-parametric form. misspecification! u Learn model by maximizing likelihood 𝑄 𝑢 C , 𝑢 D , … , 𝑢 F = exp − G 𝜇 4 𝑢 ℋ + 𝑒𝑢 J 𝜇 4 (𝑢 = |ℋ + ) H,I = " $ " # " % " & " ' ! time Da David " # " % " $

  5. New Reinforcement Learning Framework Learn policy u 𝜌 4 𝑏 𝑡 + = 𝑞 𝑢 = 𝑢 =OC, … , 𝑢 C ) where 𝑏 ∈ R Q is the next event time, to maximize cumulative reward . Learn reward u 𝑠(𝑏) to guide policy to imitate observed event data (expert). ' ( (!|+ , ) : LSTM ℎ $ ℎ " ℎ % ℎ & ! " ! % ! & imitate .(! " ) .(! % ) .(! & ) density " ∗ $ ≔ "($|( ) ) ! time David id $ + $ , $ - $ History ( )

  6. � � Optimal Reward Inverse Reinforcement Learning u 𝑠 ∗ = max ( 𝔽 XYZX[\ 8 𝑠 𝑢 = − max ] ^ 𝔽 ] ^ 8 𝑠 𝑏 = ) V∈ℱ = = Choose 𝑠 ∈ ℱ be unit ball in Reproducing Kernel Hilbert Space u (RKHS). We obtain analytical optimal reward Given L expert trajectories, M generated trajectories by policy u 𝑠̂ ∗ 𝑏 ∝ 1 f 1 i (d) , 𝑏 − (h) , 𝑏 𝑀 8 8 𝑙 𝑢 = 𝑁 8 8 𝑙 𝑏 = deC =eC heC =eC mean embedding of mean embedding of expert intensity function policy intensity function where 𝑙 𝑢, 𝑢′ is a universal RKHS kernel.

  7. Modeling Framework update optimal reward / ∗ (& + )/ ∗ (& , ) / ∗ (& - ) / ∗ (& . ) " # (&|( ) ) & + & , & - & . 0 Policy Gradient 1 + 1 , 1 - expert

  8. Numerical Results Our method: RLPP u Baselines: u State-of-the-art methods: RMTPP(Du et al. 2016 KDD ) WGANTPP (Xiao et al. 2017 NIPS) u Parametric baselines: Inhomogeneous Poisson (IP), Hawkes (SE), Self-correcting (SC) u Comparison of learned empirical intensity u 4 Methods Real RLPP WGAN IP SE RMTPP SC Methods Real RLPP WGAN IP SE RMTPP SC Methods Real RLPP WGAN IP SE RMTPP SC 6 7.5 3 4 Intensity Intensity Intensity 5.0 2 2 2.5 1 0 0 0.0 4 8 12 4 8 12 4 8 12 Time Index Time Index Time Index Comparison of runtime u Method RLPP WGANTPP RMTPP SE SC IP Time 80 m 1560m 60m 2m 2m 2m Ratio 40x 780x 30x 1x 1x 1x

  9. Poster u Tue Dec 4th 05:00 -- 07:00 PM u @ Room 210 & 230 AB # 124

Recommend

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning

Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning

Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning Richard S. Sutton, Doina Precup, Satinder Singh Presenters: Yining Chen, Will Deaderick, Neel Ramachandran, Ye Ye Motivation - Learning, planning, and

702 views • 24 slides
CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement

CSC2621 Topics in Robotics Reinforcement Learning in Robotics Week 11: Hierarchical Reinforcement Learning Animesh Garg Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning Richard S. Sutton , Doina

2.18k views • 29 slides
Machine Learning and Data Mining Reinforcement Learning Markov Decision Processes Kalev Kask

Machine Learning and Data Mining Reinforcement Learning Markov Decision Processes Kalev Kask

+ Machine Learning and Data Mining Reinforcement Learning Markov Decision Processes Kalev Kask Overview Intro Markov Decision Processes Reinforcement Learning Sarsa Q-learning Exploration vs Exploitation tradeoff 2

1.81k views • 146 slides
CSC2541: Deep Reinforcement Learning Lecture 3: Monte-Carlo and Temporal Difference Slides

CSC2541: Deep Reinforcement Learning Lecture 3: Monte-Carlo and Temporal Difference Slides

CSC2541: Deep Reinforcement Learning Lecture 3: Monte-Carlo and Temporal Difference Slides borrowed from David Silver, Andrew Barto Jimmy Ba Algorithms Multi-armed bandits UCB-1, Thompson Sampling Finite MDPs with

966 views • 33 slides
CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1

CS885 Reinforcement Learning Lecture 1b: May 2, 2018 Markov Processes [RusNor] Sec. 15.1 University of Waterloo CS885 Spring 2018 Pascal Poupart 1 Outline Environment dynamics Stochastic processes Markovian assumption

156 views • 14 slides
Reinforcement Learning January 28, 2010 CS 886 University of Waterloo Outline Russell

Reinforcement Learning January 28, 2010 CS 886 University of Waterloo Outline Russell

Reinforcement Learning January 28, 2010 CS 886 University of Waterloo Outline Russell & Norvig Sect 21.1-21.3 What is reinforcement learning Temporal-Difference learning Q-learning 2 CS886 Lecture Slides (c) 2010 K.

306 views • 28 slides
Markov Decision Processes and Reinforcement Learning Marco Chiarandini Department of Mathematics

Markov Decision Processes and Reinforcement Learning Marco Chiarandini Department of Mathematics

Lecture 14 Markov Decision Processes and Reinforcement Learning Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Slides by Stuart Russell and Peter Norvig Markov Decision Processes Course

321 views • 29 slides
Temporal Difference Methods CS60077: Reinforcement Learning Abir Das IIT Kharagpur Oct 12, 13,

Temporal Difference Methods CS60077: Reinforcement Learning Abir Das IIT Kharagpur Oct 12, 13,

Temporal Difference Methods CS60077: Reinforcement Learning Abir Das IIT Kharagpur Oct 12, 13, 19, 2020 Agenda Introduction TD Evaluation TD Control Agenda Understand incremental computation of Monte Carlo methods From incremental

1.03k views • 88 slides
Temporal Difference Learning Robert Platt Northeastern University If one had to identify one

Temporal Difference Learning Robert Platt Northeastern University If one had to identify one

Temporal Difference Learning Robert Platt Northeastern University If one had to identify one idea as central and novel to reinforcement learning, it would undoubtedly be temporal-difference (TD) learning. SB, Ch 6 Temporal Difference

1.17k views • 61 slides
Gaussian Processes for Sample Efficient Reinforcement Learning with RMAX-like Exploration Tobias

Gaussian Processes for Sample Efficient Reinforcement Learning with RMAX-like Exploration Tobias

Gaussian Processes for Sample Efficient Reinforcement Learning with RMAX-like Exploration Tobias Jung and Peter Stone Department of Computer Science University of Texas at Austin { tjung,pstone } @cs.utexas.edu Outline: 1. Motivation &

349 views • 18 slides
Time series vs. point processes for space-time surveillance in public health ohle 1 , 2 Thais

Time series vs. point processes for space-time surveillance in public health ohle 1 , 2 Thais

Motivation Case Study Temporal Monitoring Spatio-temporal Monitoring Discussion References Time series vs. point processes for space-time surveillance in public health ohle 1 , 2 Thais Correa 3 Michael H 1 Department of Statistics,

1.07k views • 28 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Examples and Videos of Markov Decision Processes (MDPs) and Reinforcement Learning Artificial

Examples and Videos of Markov Decision Processes (MDPs) and Reinforcement Learning Artificial

Examples and Videos of Markov Decision Processes (MDPs) and Reinforcement Learning Artificial Intelligence is interaction to achieve a goal Environment action state reward Agent complete agent temporally situated continual

516 views • 36 slides
Logistics Reading AIMA Ch 21 (Reinforcement Learning) Markov Decision Processes Project

Logistics Reading AIMA Ch 21 (Reinforcement Learning) Markov Decision Processes Project

Logistics Reading AIMA Ch 21 (Reinforcement Learning) Markov Decision Processes Project 1 due today 2 printouts of report Email Miao with CSE 573 Source code Document in .doc or .pdf Project 2 description on web New

400 views • 8 slides
Temporal Difference Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Used Materials

Temporal Difference Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Used Materials

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Temporal Difference Learning Spring 2019, CMU 10-403 Katerina Fragkiadaki Used Materials Disclaimer : Much of the material and slides for this lecture were

409 views • 27 slides
Markov Decision Processes (MDPs) and Reinforcement Learning (RL) Sven Koenig, USC Russell and

Markov Decision Processes (MDPs) and Reinforcement Learning (RL) Sven Koenig, USC Russell and

12/18/2019 Markov Decision Processes (MDPs) and Reinforcement Learning (RL) Sven Koenig, USC Russell and Norvig, 3 rd Edition, Sections 17.1-17.2 These slides are new and can contain mistakes and typos. Please report them to Sven

449 views • 23 slides
Reinforcement Learning Stephen D. Scott (Adapted from Tom Mitchells slides) 1 Outline

Reinforcement Learning Stephen D. Scott (Adapted from Tom Mitchells slides) 1 Outline

Reinforcement Learning Stephen D. Scott (Adapted from Tom Mitchells slides) 1 Outline Control learning Control policies that choose optimal actions Q learning Convergence Temporal difference learning 2 Control Learning

576 views • 20 slides
Deep Reinforcement Learning John Schulman 1 MLSS, May 2016, Cadiz 1 Berkeley Artificial

Deep Reinforcement Learning John Schulman 1 MLSS, May 2016, Cadiz 1 Berkeley Artificial

Deep Reinforcement Learning John Schulman 1 MLSS, May 2016, Cadiz 1 Berkeley Artificial Intelligence Research Lab Agenda Introduction and Overview Markov Decision Processes Reinforcement Learning via Black-Box Optimization Policy Gradient

865 views • 64 slides
Reinforcement Learning: From neural processes modelling to Robotics applications Mehdi Khamassi

Reinforcement Learning: From neural processes modelling to Robotics applications Mehdi Khamassi

TDRL Model Dopamine TD Applications Model-based RL slide # 1 / 141 Reinforcement Learning: From neural processes modelling to Robotics applications Mehdi Khamassi (CNRS, ISIR-UPMC, Paris) 30 January 2015 Michle Sebags course @ Univ.

1.64k views • 142 slides
Reinforcement learning Chapter 21, Sections 14 Chapter 21, Sections 14 1 Outline

Reinforcement learning Chapter 21, Sections 14 Chapter 21, Sections 14 1 Outline

Reinforcement learning Chapter 21, Sections 14 Chapter 21, Sections 14 1 Outline Examples Learning a value function for a fixed policy temporal difference learning Q-learning Function approximation Exploration

562 views • 14 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides

More recommend