[PPT] - Motions Using Reinforcement Learning Libin Liu http://libliu.info PowerPoint Presentation

SLIDE 1

Learning to Control Complex Human Motions Using Reinforcement Learning

Libin Liu DeepMotion Inc

1

http://libliu.info http://deepmotion.com

SLIDE 2

Physics-based Character Animation

Motion Controller Control Signal Physics Engine Character Animation

[Gang Beasts] [Totally Accurate Battle Simulator]

SLIDE 3

Designing Controllers for Locomotion

Hand-crafted control policy Simulating abstract model

SIMBICON, IPM, ZMP…

Optimization/policy search Reinforcement learning

Actor-critic

[Hodgins et al. 1995] [Tan et al. 2014] [Coros et al. 2010] [Peng et al. 2017] [Mordatch et al. 2010] SIMBICON [Yin et al. 2007]

SLIDE 4

Designing Controllers for Complex Motions

4

SLIDE 5

Designing controllers for complex motions

5

Motion Clip Tracking Controller

SLIDE 6

Tracking Control for Complex Human Motion

Motion Clip Open-loop Tracking Control Feedback Policy Feedback Policy

Control Scheduler

SLIDE 7

Reinforcement Learning

Feedback Policy Feedback Policy

Control Scheduler Guided Policy Learning Deep Q-Learning

SLIDE 8

Outline

Construct open-loop control

SAMCON (Sample-based Motion Control)

Guided learning of linear feedback policies Learning to schedule control fragment using deep Q-learning

8

SLIDE 9

Tracking Control

PD servo

𝜐 = 𝑙𝑞 ෨ 𝜄 − 𝜄 − 𝑙𝑒 ሶ 𝜄

9

SLIDE 10

Mocap Clips as Tracking Target

10

SLIDE 11

Correction with Sampling

[ ]

11

𝜀𝑢

SLIDE 12

SAMCON

SAmpling-based Motion CONtrol [Liu et al. 2010, 2015]
Motion Clip  Open-loop control trajectory

Start Sample Start1 Sample Start2 Sample Startn

…

End

12

Particle filtering / Sequential Monte Carlo

SLIDE 13

SAMCON

13

𝜀𝑢 𝜀𝑢 𝜀𝑢 𝜀𝑢 time Reference Trajectory State

SLIDE 14

𝜀𝑢 𝜀𝑢 𝜀𝑢 𝜀𝑢

Sampling & Simulation

14

time Reference Trajectory 𝑢 𝑏 State Actions (PD-control Targets)

SLIDE 15

𝜀𝑢 𝜀𝑢 𝜀𝑢 𝜀𝑢

Resampling

15

𝑢 time Reference Trajectory 𝑏 Actions (PD-control Targets) State

SLIDE 16

𝜀𝑢 𝜀𝑢 𝜀𝑢 𝜀𝑢

SAMCON Iterations

16

𝑏 𝜀𝑢 time Reference Trajectory State Actions (PD-control Targets)

SLIDE 17

𝜀𝑢 𝜀𝑢 𝜀𝑢 𝜀𝑢

SAMCON Iterations

17

𝑏 𝜀𝑢 time Reference Trajectory State Actions (PD-control Targets)

SLIDE 18

𝜀𝑢 𝜀𝑢 𝜀𝑢 𝜀𝑢

Constructed Open-loop Control Trajectory

18

𝑏 𝜀𝑢 time Reference Trajectory State Actions (PD-control Targets)

SLIDE 19

Control Reconstruction

19

SLIDE 20

Linear Policy

𝑡 − ǁ 𝑡 = 𝜀𝑡

𝜌: 𝜀𝑏 = 𝑁 𝜀𝑡 + ො 𝑏

𝜀𝑏 = 𝑏 − ෤ 𝑏

Control Trajectory Simulation

20

SLIDE 21

For complex motions

21

Uniform Segmentation Linear Feedback Policy

Control Fragments

SLIDE 22

Control Fragment

A short control unit:
𝜀𝑢 ≈ 0.1 seconds long
Open-loop control segment ෝ

𝑛

Linear Feedback policy 𝜌

22

𝒟 ∶ 𝜀𝑢, ෝ 𝑛, 𝜌

𝜌

ෝ 𝑛

𝜀𝑢

SLIDE 23

Controller

A chain of control fragments

𝒟1 𝒟2

⋯

𝒟𝐿

23

SLIDE 24

Guided Learning of Control Policies

24

Multiple Open-loop Solutions Feedback Policy

Regression

SLIDE 25

Guided Learning of Control Policies

25

Feedback Policy

Guided Learning

Multiple Open-loop Solutions

SLIDE 26

Guided Learning of Control Policies

26

Feedback Policy

Guided Learning

Multiple Open-loop Solutions

SLIDE 27

Guided Learning of Control Policies

27

Feedback Policy

Guided Learning

Multiple Open-loop Solutions

SLIDE 28

Example: Cyclical Motion

28

𝒟1 𝒟2 𝒟4 𝒟3

𝒟𝑙: ෝ 𝑛𝑙, 𝜀𝑢, 𝜌𝑙

𝑏 𝑢 SAMCON

𝓓𝟐, 𝓓𝟑, 𝓓𝟒, 𝓓𝟓, 𝓓𝟐, 𝓓𝟑, 𝓓𝟒, 𝓓𝟓, 𝓓𝟐, 𝓓𝟑, 𝓓𝟒, 𝓓𝟓, …

SLIDE 29

Example: Cyclical Motion

29

𝜌1

𝑡1 𝑏1

𝜌2

𝑡2 𝑏2 𝑏 𝑢 SAMCON

𝓓𝟐, 𝓓𝟑, 𝓓𝟒, 𝓓𝟓, 𝓓𝟐, 𝓓𝟑, 𝓓𝟒, 𝓓𝟓, 𝓓𝟐, 𝓓𝟑, 𝓓𝟒, 𝓓𝟓, … 𝜌𝟒

𝑡3 𝑏3

𝜌𝟓

𝑡4 𝑏4

SLIDE 30

Policy Update

30 30

𝑡 𝑏

SLIDE 31

Policy Update

31 31

𝑡 𝑏

Regression

SLIDE 32

Guided Learning Iterations

32 32

𝑡 𝑏

Regression

𝑡 𝑏

Guided SAMCON

SLIDE 33

Guided Learning Iterations

33 33

𝑡 𝑏

Regression

𝑡 𝑏

Regression Guided SAMCON

SLIDE 34

34

SLIDE 35

Control Graph

A graph whose nodes are control fragments

Control Graph

35

SLIDE 36

Control Graph

A graph whose nodes are control fragments
Converted from a motion graph

Control Graph Motion Graph

36

SLIDE 37

37

SLIDE 38

38

SLIDE 39

Problem of Fixed Time-Indexed Tracking

Reference Basin of attraction Simulation

SLIDE 40

Scheduling

Reference Basin of attraction Simulation

SLIDE 41

Scheduling

?

SLIDE 42

Deep Q-Learning

Learn to perform good actions Raw image input Deep convolutional network

[Mnih et al. 2015, DQN]

SLIDE 43

A Q-Network For Scheduling

… … … … … … … … … … state

max 0, 𝑨 max 0, 𝑨

Q-values

Fully Connected

𝑔 𝑔

300 ReLus 300 ReLus

SLIDE 44

A Q-Network For Scheduling

… … … … … … … … … … state

max 0, 𝑨 max 0, 𝑨

Q-values

Fully Connected

𝑔 𝑔

300 ReLus 300 ReLus

Input: motion state environmental state user command DoFs: 18 ~ 25

SLIDE 45

A Q-Network For Scheduling

… … … … … … … … … … state

max 0, 𝑨 max 0, 𝑨

Fully Connected

𝑔 𝑔

300 ReLus 300 ReLus

Action Set: Control Fragments # of actions: 39 ~ 146

SLIDE 46

A Q-Network For Scheduling

… … … … … … … … … …

max 0, 𝑨 max 0, 𝑨

Fully Connected

𝑔 𝑔

300 ReLus 300 ReLus

actions:

Q-Values

SLIDE 47

Training

Pipeline: Exploration / Exploitation Simulation Reward Replay Buffer Batch SGD

SLIDE 48

Reward Function

𝑆 = 𝐹tracking + 𝐹preference + 𝐹feedback + 𝐹task + 𝑆0

SLIDE 49

Importance of the Reference Sequence

riginal sequence is enforced
riginal sequence is not enforced

SLIDE 50

Tracking penalty term

Penalty

In-sequence action Out-of-sequence action

SLIDE 51

Tracking exploration strategy

with probability 𝜁𝑠 select a random action with probability 𝜁𝑝 select an in-sequence action

SLIDE 52

Bongo Board Balancing

Action Sequence

SLIDE 53

Effect of Feedback Policy

Open-loop Control Fragments Feedback-augmented Fragments

SLIDE 54

Discover New Transitions

SLIDE 55

Running

SLIDE 56

Tripping

SLIDE 57

Skateboarding

SLIDE 58

Skateboarding

SLIDE 59

Walking On A Ball

SLIDE 60

Push-Recovery

SLIDE 61

Conclusion

Motion Clip Open-loop Tracking Control Feedback Policy Feedback Policy Control Scheduler

Libin Liu, Michiel Van De Panne, and Kangkang Yin.

2016. Guided Learning of Control Graphs for Physics-

Based Characters. ACM Trans. Graph. 35, 3, Article 29 (May 2016), 14 pages. Libin Liu and Jessica Hodgins. 2017. Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning. ACM Trans. Graph. 36, 3, Article 29 (June 2017), 14 pages.

SLIDE 62

Future Work

Statistical/generative model Control with raw simulation state and terrain information Active human-object interaction

basketball, soccer dancing, boxing, martial arts

62

[Peng et al. 2017, DeepLoco] [Heess et al. 2017] [Holden et al. 2017]

SLIDE 63

Questions?

Libin Liu http://libliu.info DeepMotion Inc http://deepmotion.com