SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang 1 - PowerPoint PPT Presentation

SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang 1 Yuzhe Qin 1 Kaichun Mo 2 Yikuan Xia 1 Hao Zhu 1 Fangchen Liu 1 Minghua Liu 1 Hanxiao Jiang 3 Yifu Yuan 5 He Wang 2 Li Yi 4 Angel X. Chang 3 Leonidas J. Guibas 2 Hao Su 1 1 UC San Diego 2 Stanford University 3 Simon Fraser University 4 Google Research 5 UC Los Angeles

Perceive information Intelligent Agent Adapt behavior

Perceive information Computer vision Robotics control Adapt behavior

Perceive information Computer vision Planning? Active perception? Robotics control Adapt behavior

Data driven approaches

Learn from data?

Real-world data collection

Reinforcement Learning? Manual data collection?

Reinforcement learning? 1. Expensive hardware 2. Slow data collection 3. Low sample e ffi ciency 4. Overfit to specific agent Manual data collection? 1. Unintuitive control 2. May not be optimal 3. Expertise

Simulated Environment

Simulated Environment Physics Robotics Rendering Simulation Content

Simulated Environment ? Real World

Simulated Environment Physics Robotics Rendering Simulation Content

Physics Robotics Rendering Simulation Content SAPIEN

SAPIEN SAPIEN SAPIEN Engine Renderer Asset Physics Robotics Rendering Simulation Content

SAPIEN Engine SAPIEN Engine PhysX Physical Simulator World Articulation Interface Interface ROS Interface Controller Sensor Interface Interface 3D/IMU 
 Force/Joint/Velocity Trajectory Inverse Kinematics Sensor Controller Controller Motion Planning Client API

SAPIEN Engine Reinforcement Learning Robot Tasks 3D/IMU 
 Force/Joint/Velocity Trajectory Inverse Kinematics Sensor Controller Controller Motion Planning Client API

SAPIEN Renderer SAPIEN Engine SAPIEN Renderer PhysX Physical Simulator Renderer Interface World Articulation Interface Interface Rendered ROS Interface Images Controller Sensor Interface Interface 3D/IMU 
 Force/Joint/Velocity Trajectory Inverse Kinematics Sensor Controller Controller Motion Planning Client API

SAPIEN Renderer SAPIEN Renderer Renderer Interface RGBD GLSL Shaders Normal Segmentation

SAPIEN Renderer SAPIEN Renderer Renderer Interface RGBD GLSL Shaders Normal Segmentation OptiX Ray Tracing

SAPIEN Renderer SAPIEN Renderer Renderer Interface RGBD GLSL Shaders Normal Segmentation OptiX Ray Tracing Customizable Renderer/Visualizer

SAPIEN Asset SAPIEN Asset SAPIEN Engine SAPIEN Renderer PhysX Physical Simulator Renderer Interface PartNet-Mobility Dataset World Articulation RGBD Interface Interface GLSL Shaders Normal ROS Interface Segmentation Controller OptiX Sensor Ray Tracing Interface Interface 3D/IMU 
 Force/Joint/Velocity Trajectory Inverse Kinematics Customizable Renderer/Visualizer Sensor Controller Controller Motion Planning Client API

SAPIEN Asset SAPIEN Asset ShapeNet PartNet-Mobility Dataset Motion Annotation PartNet 2K models, 14K parts

Generalize Training Testing

SAPIEN Asset SAPIEN Asset PartNet-Mobility Dataset Robot Model Standard URDF format Object Layout Python API Robot/Scene Builder

SAPIEN SAPIEN Asset SAPIEN Engine SAPIEN Renderer PhysX Physical Simulator Renderer Interface PartNet-Mobility Dataset World Articulation RGBD Interface Interface Robot Model GLSL Shaders Normal ROS Interface Segmentation Object Layout Controller OptiX Sensor Ray Tracing Interface Interface Robot/Scene 3D/IMU 
 Force/Joint/Velocity Trajectory Inverse Kinematics Customizable Renderer/Visualizer Builder Sensor Controller Controller Motion Planning Client API

Task Demonstrations Movable Part Motion Parameter Segmentation Estimation Long-horizon Part Manipulation Planning

Future Plans/Applications • Benchmark Platform • Vision tasks • Manipulation tasks • Education Platform • Control Algorithms • Robot Learning

Summary • SAPIEN provides simulation and rendering for vision and manipulation. • SAPIEN hosts a large-scale dataset for articulated objects. • SAPIEN is open source. • pip install sapien https://sapien.ucsd.edu Requirements: Python 3, Linux / Latest MacOS