Learning to Control an Octopus Arm with Gaussian Process Temporal - PowerPoint PPT Presentation

Learning to Control an Octopus Arm with Gaussian Process Temporal Difference Methods Yaakov Engel Collaborators: Peter Szabo and Dmitry Volkinshtein

Bayesian RL Tutorial 2/16

The Octopus Arm Can bend and twist at any point Can do this in any direction Can be elongated and shortened Can change cross section Can grab using any part of the arm Virtually infinitely many DOF Bayesian RL Tutorial 3/16

Octopus Arm Anatomy 101 Bayesian RL Tutorial 4/16

✱ ✲ ✳ ✳ ✳ ✳ ✳ ✳ ✳ ✲ ✲ ✲ ✲ ✴ ✲ ✱ ✱ ✱ ✸ ✱ ✱ ✯✰ ✭✮ ✫✬ ✳ ✴ ✧★ ✶ ✷ ✷ ✷ ✷ ✷ ✷ ✶ ✶ ✶ ✶ ✶ ✴ ✵ ✵ ✵ ✵ ✵ ✵ ✴ ✴ ✴ ✴ ✴ ✩✪ ✥✦ ✸ ✂ ☎ ☎ ✄ ✄ ✄ ✄ ✄ ✄ ✂ ✂ ✂ ☎ ✂ ✂ ✁ ✁ ✁ ✁ � � � � ✸ ☎ ✆ ✣✤ ✟✠ ✜✢ ✚✛ ✘✙ ✖✗ ✔✕ ✒✓ ✏✑ ✍✎ ☞✌ ✡☛ ✸ ✆ ✸ ✞ ✞ ✞ ✞ ✝ ✝ ✝ ✝ ✆ ✆ ✸ Our Arm Model arm tip dorsal side C N pair #N+1 arm base C 1 ventral side pair #1 longitudinal muscle transverse muscle transverse muscle longitudinal muscle Bayesian RL Tutorial 5/16

✁ ✁ � ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ � ✁ � � � � � � � � � � � � � � � � � � ✁ The Muscle Model f ( a ) = ( k 0 + a ( k max − k 0 )) ( ℓ − ℓ 0 ) + β dℓ dt a ∈ [0 , 1] Bayesian RL Tutorial 6/16

Other Forces • Gravity • Buoyancy • Water drag • Internal pressures (maintain constant compartmental volume) Dimensionality 10 compartments ⇒ 22 point masses × ( x, y, ˙ x, ˙ y ) = 88 state variables Bayesian RL Tutorial 7/16

The Control Problem Starting from a random position, bring { any part, tip } of arm into contact with a goal region, optimally . Optimality criteria: Time, energy, obstacle avoidance Constraint: We only have access to sampled trajectories Our approach: Define problem as a MDP Apply Reinforcement Learning algorithms Bayesian RL Tutorial 8/16

The Task 0.15 t = 1.38 0.1 0.05 0 −0.05 −0.1 −0.1 −0.05 0 0.05 0.1 0.15 Bayesian RL Tutorial 9/16

Actions Each action specifies a set of fixed activations – one for each muscle in the arm. Action # 1 Action # 2 Action # 3 Action # 4 Action # 5 Action # 6 Base rotation adds duplicates of actions 1,2,4 and 5 with positive and negative torques applied to the base. Bayesian RL Tutorial 10/16

Rewards Deterministic rewards: +10 for a goal state, Large negative value for obstacle hitting, -1 otherwise. Energy economy: A constant multiple of the energy expended by the muscles in each action interval was deducted from the reward. Bayesian RL Tutorial 11/16

Now, to the Movies... Bayesian RL Tutorial 12/16

Fixed Base Task I Bayesian RL Tutorial 13/16

Fixed Base Task II Bayesian RL Tutorial 14/16

Rotating Base Task I Bayesian RL Tutorial 15/16

Rotating Base Task II Bayesian RL Tutorial 16/16