SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang 1 - PowerPoint PPT Presentation

SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang 1 Yuzhe Qin 1 Kaichun Mo 2 Yikuan Xia 1 Hao Zhu 1 Fangchen Liu 1 Minghua Liu 1 Hanxiao Jiang 3 Yifu Yuan 5 He Wang 2 Li Yi 4 Angel X. Chang 3 Leonidas J. Guibas 2 Hao Su 1 1 UC San Diego 2 Stanford University 3 Simon Fraser University 4 Google Research 5 UC Los Angeles

About Me Fanbo Xiang, UCSD UIUC BS CS, BS Math UCSD: MS CS Advisor: Hao Su, UCSD Research: Graphics, Vision, Robotics

Outline ● Background ○ Intelligence, vision and robotics ○ Simulated environment ● SAPIEN architecture ○ Physics and robotics ○ Renderer ○ Assets ● Future research problems

Intelligent Agent

Perceive information Intelligent Agent

Perceive information Intelligent Agent Adapt behavior

Perceive information Adapt behavior

Perceive information Computer vision Adapt behavior

Perceive information Computer vision Robotics control Adapt behavior

Perceive information Computer vision Robotics control Adapt behavior

Perceive information Computer vision Planning? Active perception? Robotics control Adapt behavior

Data driven approaches

Learn from data?

Real-world data collection

Reinforcement Learning? Manual data collection?

Reinforcement Learning

RL Agent

Observation RL Agent

Perception Observation RL Agent

Perception Observation RL Agent Action

Perception Observation RL Agent Action Control

Perception Observation State update RL Agent Reward Action Control

RL Problems ● Expensive hardware ● Slow data collection ● Low sample efficiency ● Overfit to specific agent

Imitation Learning (Manual Data Collection)

Perception Observation State update Human Action Control

Perception Observation Training State update Human IL Agent Action Control

IL Problems ● Unintuitive control ● May not be optimal ● Human expertise

Simulated Environment

Manual data collection RL ● More freedom in controller ● Will not break hardware Simulated Environment design ● Easy to scale ● Less expertise required

Simulated Environment

Simulated Environment ? Real World

Simulated Environment Physics

Simulated Environment Physics Robotics

Simulated Environment Physics Robotics Rendering

Simulated Environment Physics Robotics Rendering Simulation Content

Physics Robotics Rendering Simulation Content SAPIEN

SAPIEN SAPIEN SAPIEN Engine Renderer Asset Physics Robotics Rendering Simulation Content

SAPIEN Engine

SAPIEN Engine SAPIEN Engine PhysX Physical Simulator

SAPIEN Engine SAPIEN Engine PhysX Physical Simulator Articulation World Interface Interface

SAPIEN Engine SAPIEN Engine PhysX Physical Simulator Articulation World Interface Interface ROS Interface Controller Sensor Interface Interface

SAPIEN Engine SAPIEN Engine PhysX Physical Simulator Articulation World Interface Interface ROS Interface Controller Sensor Interface Interface 3D/IMU Force/Joint/Velocity Trajectory Inverse Kinematics Sensor Controller Controller Motion Planning Client API

SAPIEN Engine 3D/IMU Force/Joint/Velocity Trajectory Inverse Kinematics Sensor Controller Controller Motion Planning Client API

SAPIEN Engine Reinforcement Learning 3D/IMU Force/Joint/Velocity Trajectory Inverse Kinematics Sensor Controller Controller Motion Planning Client API

SAPIEN Engine Robot Reinforcement Learning Tasks 3D/IMU Force/Joint/Velocity Trajectory Inverse Kinematics Sensor Controller Controller Motion Planning Client API

SAPIEN Engine

SAPIEN Renderer SAPIEN Engine PhysX Physical Simulator Articulation World Interface Interface ROS Interface Controller Sensor Interface Interface 3D/IMU Force/Joint/Velocity Trajectory Inverse Kinematics Sensor Controller Controller Motion Planning Client API

SAPIEN Renderer SAPIEN Engine SAPIEN Renderer PhysX Physical Simulator Articulation World Interface Interface ROS Interface Controller Sensor Interface Interface 3D/IMU Force/Joint/Velocity Trajectory Inverse Kinematics Sensor Controller Controller Motion Planning Client API

SAPIEN Renderer SAPIEN Engine SAPIEN Renderer PhysX Physical Simulator Renderer Interface Articulation World Interface Interface ROS Interface Controller Sensor Interface Interface 3D/IMU Force/Joint/Velocity Trajectory Inverse Kinematics Sensor Controller Controller Motion Planning Client API

SAPIEN Renderer SAPIEN Engine SAPIEN Renderer PhysX Physical Simulator Renderer Interface Articulation World Interface Interface Rendered ROS Interface Images Controller Sensor Interface Interface 3D/IMU Force/Joint/Velocity Trajectory Inverse Kinematics Sensor Controller Controller Motion Planning Client API

SAPIEN Renderer SAPIEN Renderer Renderer Interface RGBD Normal GLSL Shaders Segmentation

SAPIEN Renderer SAPIEN Renderer Renderer Interface RGBD Normal GLSL Shaders Segmentation Ray Tracing OptiX Shaders

SAPIEN Renderer SAPIEN Renderer Renderer Interface RGBD Normal GLSL Shaders Segmentation Ray Tracing OptiX Shaders 4 spp, 3 bounce, 512x512 OptiX denoise, < 20 FPS

SAPIEN Renderer SAPIEN Renderer Renderer Interface RGBD Normal GLSL Shaders Segmentation Ray Tracing OptiX Shaders Customizable Renderer/Visualizer

SAPIEN Renderer SAPIEN Renderer Renderer Interface Considerations ● Needs to run at real time (~100 FPS) RGBD Normal ○ Rasterizer GLSL Shaders Segmentation ○ Ray tracer denoise? (1spp max) ○ GPU-CPU transfer? (>10 ms) Ray Tracing OptiX Shaders ○ Prebaking? (interaction) Customizable Renderer/Visualizer

SAPIEN Renderer SAPIEN Engine SAPIEN Renderer PhysX Physical Simulator Renderer Interface Articulation World RGBD Interface Interface Normal GLSL Shaders Segmentation ROS Interface Controller Sensor Ray Tracing OptiX Shaders Interface Interface 3D/IMU Force/Joint/Velocity Trajectory Inverse Kinematics Customizable Customizable Sensor Controller Controller Motion Planning Renderer/Visualizer Renderer/Visualizer Client API

SAPIEN Asset SAPIEN Asset ShapeNet PartNet-Mobility Dataset Motion Annotation PartNet 2K models, 14K parts

PartNet-Mobility Annotation Shape cleaning Part segmentation Motion annotation (ShapeNet, 3D Warehouse) (PartNet) (PartNet-Mobility)

Generalize Training Testing

SAPIEN Asset SAPIEN Asset PartNet-Mobility Dataset

SAPIEN Asset SAPIEN Asset PartNet-Mobility Dataset Robot Model

SAPIEN Asset SAPIEN Asset PartNet-Mobility Dataset Robot Model Object Layout

SAPIEN Asset SAPIEN Asset PartNet-Mobility Dataset Robot Model Standard URDF format Object Layout Python API Robot/Scene Builder

SAPIEN SAPIEN Asset SAPIEN Engine SAPIEN Renderer PhysX Physical Simulator Renderer Interface PartNet-Mobility Dataset Articulation World RGBD Interface Interface Robot Model Normal GLSL Shaders Segmentation ROS Interface Object Layout Controller Sensor Ray Tracing OptiX Shaders Interface Interface Robot/Scene 3D/IMU Force/Joint/Velocity Trajectory Inverse Kinematics Customizable Customizable Builder Sensor Controller Controller Motion Planning Renderer/Visualizer Renderer/Visualizer Client API

Task Demonstrations

Task Demonstrations Movable Part Segmentation

Task Demonstrations Movable Part Motion Parameter Segmentation Estimation

Task Demonstrations Movable Part Motion Parameter Segmentation Estimation Part Manipulation

Task Demonstrations Movable Part Motion Parameter Segmentation Estimation Long-horizon Part Manipulation Planning

Task Demonstrations Movable Part Segmentation ● Standard vision problem: detection and segmentation.

Task Demonstrations ● New vision task Motion Parameter Estimation ● Important for control tasks ○ Given the motion parameters, we can use control methods to manipulate the parts

Task Demonstrations ● Manipulation and control Part Manipulation ○ Reinforcement learning ○ Imitation learning ○ robotics control

Task Demonstrations ● Manipulation and control ○ Reinforcement learning ○ Imitation learning ○ robotics control

Task Demonstrations Long-horizon ● Planning Planning ○ Achieve meaningful tasks ○ “Home assistant”

Future Plans/Applications • Benchmark (SAPIEN Challenge) • Vision tasks • Manipulation tasks • Education Platform • Control Algorithms • Robot Learning

Demos

Demos