Robust Articulated-ICP for Real-Time Hand Tracking Andrea Tagliasacchi* Matthias Schröder* Anastasia Tkach Sofien Bouaziz Mario Botsch Mark Pauly * equal contribution Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 1 /36
Real-Time Tracking Setup Data from (single) RGBD Sensors PrimeSense (Carmine) SoftKinetic Intel RealSense low SNR along in depth (z-axis) completely discards small portions of geometry Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 2
Motivation? Augmented Reality Intel Perceptual SDK Oculus Research (VR) MagicLeap (AR) HoloLens (AR) Microsoft HoloLens - PR Video (hololens.com) Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 3
Previous Work Appearance-based Model-based (guess solely based on current frame) (registers model of previous frame) Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 4
Previous Work Appearance Model “vision” “geometry” [Keskin ICCV’12] [Qian CVPR’14] [Oikono. CVPR’14] [Tompson SIG’14] [Melax I3D’13] [Tang CVPR’14] [Sridhar 3DV’14] [Schroder ICRA’14] Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 5
Contributions • combined 2D/3D registration (within ICP) • occlusion-aware correspondences (ICP) • regularization with statistical pose-space prior • extensible and unified real-time solver (>60fps) • revamping ICP for articulated tracking Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 6
ICP: Iterative Closest Point • Step 1: optimizing correspondences • Step 2: optimizing transformations update correspondences update transformation update transformation update correspondences for more details please refer to Sparse-ICP [Bouaziz, Tagliasacchi, Pauly SGP’13] Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 7
Robust Articulated ICP [Wei et al. SIGA’12] our tracking process successfully tracks the entire motion sequence while ICP fails to track most of frames. This is because ICP is often sensitive to initial poses and prone to local minimum , particularly involving tracking high-dimensional human body poses from noisy depth data. [Zhang et al. SIGA’14] The accompanying video clearly shows that our tracking process is much more robust than the ICP algorithm […] our tracking process successfully tracks the entire motion sequence while ICP fails to track most of frames. [Qian et al. CVPR’14] It uses alternate and gradient based optimization, converges fast, and is suitable for realtime applications. However, it can be easily trapped in poor local optima and cannot handle non-rigid objects well. Yet, it is still insu ffi cient for high-dimensional articulated hands, especially under free viewpoints. Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 8
System Overview lost tracking? converged? silhouette posed model no no yes yes color image reinitialize linear 2D correspondences solve 3D correspondences data fitting temporal bounds pose space collision distance trans. point cloud depth image E points + E silh. + E wrist + E pose + E kin. + E temporal min θ | {z } | {z } Fitting terms Prior terms Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 9
Preprocessing S s - sensor silhouette • color to identify the Region-of-Interest • demo : assumption on picking “+y” for PCA • … but all this could be learned!! [Tompson et al. TOG’14] Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 10
Data Fitting Energies lost tracking? converged? silhouette posed model no no yes yes color image reinitialize linear 2D correspondences solve 3D correspondences data fitting temporal bounds pose space collision distance trans. point cloud depth image E points + E silh. + E wrist + E pose + E kin. + E temporal min θ | {z } | {z } Fitting terms Prior terms Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 11
3D Registration (w/ occlusions) x X s - sensor point cloud X k x � Π M ( x , θ ) k 1 E points = ω 1 = Π M ( 2 x ∈ X s ICP (case #1) ICP (case #2) our method (case #2) M - cylinder hand model 3D Registration Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 12
Lesson #1: insu ffi cient to render Correspondences of [Wei et al. SIGA’12] X s - sensor point cloud (renders the hand model into a point cloud) c 1 c 2 c 1 Correspondences of [Our Method] (computes correspondences in close form) c 2 c 1 Ground Truth Motion c 2 (finger 2 comes out of occlusion) M - cylinder hand model 3D Registration Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 13
2D/3D Registration X s - sensor point cloud M - cylinder hand model 3D Registration Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 14
2D/3D Registration S s - sensor silhouette X s - sensor point cloud M - cylinder hand model S r - rendered silhouette 3D Registration 2D Registration Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 15
2D Registration S s - sensor silhouette x X k p � Π S s ( p , θ ) k 2 E silhouette = ω 2 Π S s 2 p ∈ S r S r - rendered silhouette 2D Registration Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 16
Model Prior Energies lost tracking? converged? silhouette posed model no no yes yes color image reinitialize linear 2D correspondences solve 3D correspondences data fitting temporal bounds pose space collision distance trans. point cloud depth image E points + E silh. + E wrist + E pose + E kin. + E temporal min θ | {z } | {z } Fitting terms Prior terms Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 17
Joint Bounds Energy Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 18
Collision Energy Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 19
Temporal Coherence Energy Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 20
Statistical Pose Prior encodes correlation across joints Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 21
Statistical Pose Prior Recorded by VICON tracking system [Schroder ICRA’14] (they are accurate… for the person that have been recorded for) Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 22
Statistical Pose Prior (Soft) ω 4 k θ � ( µ + Π P ˜ + ω 5 k Σ ˜ θ ) k 2 θ k 2 E pose = 2 2 we optimize the but when DOF are current pose So that it is similar to a unconstrained we would reconstructed pose from the like to restore the neutral low dimensional subspace (i.e. mean) pose. θ ˜ θ µ Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 23
Statistical Pose Prior (Hard) ω 4 k θ � ( µ + Π P ˜ + ω 5 k Σ ˜ θ ) k 2 θ k 2 E pose = 2 2 only optimize in the subspace ω 4 = ∞ (i.e. variable replacement) … even with a high number of bases the animation remains sti ff !!! Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 24
CPU/GPU Optimization ⌘ − 1 ⇣ X ⇣ X ⌘ X k f i + J i δθ k 2 J T − J T min δθ = i J i i f i 2 δθ i i i ( n T ( J persp ( x ) J skel ( x ) δ θ + ( p − Π S s ( p , θ ))) 2 ¯ X E silh. = ω 2 p ∈ S r | J T | J silh | ≈ 20 k × 26 silh J silh | = 26 × 26 |S r | ≈ 20 k !!!! Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 25
Results and Limitations Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 26
Motion Transfer to Rig Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 27
Tracking with Fast Motion Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 28
Tracking of Interacting Hands Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 29
Limitation: Calibration Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 30
Limitations: Fist Rotation Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 31
State of the Art - Evaluations Dexter-1 Dataset (MPI) mm [Tang et al. 2014] [Sridhar et al. 2013] [Sridhar et al. 2014] [ours] [ours] + re-init. [Shroder ICRA’14] Subspace ICP [Sridhar ICCV’13] EM Tracker [Tang CVPR’14] Forest Classifiers 20 10 adbadd flexex1 pinch count tigergrasp wave random (re-initialization helps because Dexter-1 is a low frame-rate dataset… only 30Hz) [Melax’14] Intel Perceptual SDK [Tompson TOG’14] ConvNets [Qian CVPR’14] ICP/PSO Hybrid Qualitative Quantitative Robust Articulated ICP for Real-Time Hand Tracking Presented by: Andrea Tagliasacchi 32
Recommend
More recommend
