tCap : High-Speed Human Motion Capture using Eve EventC an Event - PowerPoint PPT Presentation

tCap : High-Speed Human Motion Capture using Eve EventC an Event Camera Lan XU 许岚 Hong Kong University of Science and Technology 2020/06/11

Background 2

1. Background q Previous MoCap systems 2 nd Generation 3 rd Generation 1 st Generation Marker-based MoCap : High-end marker-less system: Convenient Capture Only reconstruct makers Many, many cameras Handheld or single-view • • • Intrusive, restricted clothing Green background & fixed space Consumer-level • • • Not ready for daily usage Tedious synchronization, calibration Still fixed captured volume • • • Technological Trend: Realtime, convenient and high quality 4D human reconstruction is critical 3

1. Background q Bottleneck of high-speed human MoCap • high speed motion analysis is rare • RGB/RGBD: good lighting for high UnstructuredFusion frame rates • Throughput: a VGA RGB stream at 1000 fps for 60 s à 51.5 GB !!! RobustFusion 4 MonoPerfCap LiveCap

1. Background q Bottleneck of high-speed human MoCap • high speed motion analysis is rare • RGB/RGBD: good lighting for high frame rates Could we liberates these • Throughput: a VGA RGB stream at constraints ? EventCap: use new device !!! 1000 fps for 60 s à 51.5 GB !!! 5

Key idea 6

2. Human Modelling: EventCap q Basic idea Capturing high-speed human motions at 1000 fps • q Benefits: High temporal resolution, HDR (140 dB), low data bandwidth • q Challenges: Images & events: unstructured temporal information • Severe image blur • …… t_0 t_1 t_2 t0_n t0_0 t0_2 7

2. Human Modelling: EventCap q High-speed human motions 8

2. Human Modelling: EventCap V.S. Event camera: DAVIS240C High speed camera: Sony RX0 Only 3.4% data bandwidth 9

2. Human Modelling: EventCap q Reconstruction results for sports analysis Low FPS image Event polarity Reference view in Sony Camera 10

2. Human Modelling: EventCap q Results of capturing a Ninja in the dark Thanks to the high dynamic range (140 dB) of the event camera • (Original images) Low FPS image Event polarity (Gamma enhancement) Reference view in Sony Camera 11

Method 12

3. Algorithm details q Input of EventCap Intensity image stream DAVIS 240C Rigged Template Event stream 13

3. Algorithm details q Framework … … Event trajectory Events … ! "#$%& t 2D features at tracking fps Boundary information Detection 1. Event Trajectory Generation 2. Batch Optimization 3. Event-based Pose Refinement 14

3. Algorithm details q Stage I: Event Trajectory Generation … … Event trajectory Events … ! "#$%& t 2D features at tracking fps Detection Boundary information 15

3. Algorithm details q Stage I: Event Trajectory Generation … … Event trajectory Events … ! "#$%& t 2D features at tracking fps Detection Boundary information + = Intensity image stream Event stream Event trajectories 16

3. Algorithm details q Stage I: Event Trajectory Generation … … Event trajectory Events … ! "#$%& t 2D features at tracking fps Detection Boundary information 2D feature trajectory between adjacent images • Forward & backward alignment • Trajectory slicing à 2D correspondence pairs • 17

3. Algorithm details q Stage II: Batch Optimization … … Event trajectory Events … ! "#$%& t 2D features at tracking fps Detection Boundary information 18

3. Algorithm details q Stage II: Batch Optimization … … Event trajectory Events … ! "#$%& t 2D features at tracking fps Detection Boundary information + + = 2D detection 3D detection Event trajectories 19

3. Algorithm details q Stage II: Batch Optimization … … Event trajectory Events … ! "#$%& t 2D features at tracking fps Detection Boundary information 20

3. Algorithm details q Stage II: Event-based Pose Refinement … … Event trajectory Events … ! "#$%& t 2D features at tracking fps Detection Boundary information 21

3. Algorithm details q Stage II: Event-based Pose Refinement … … Event trajectory Events … ! "#$%& t 2D features at tracking fps Detection Boundary information 1 0 Normalized distance map Results of Stage II Our final results Event stream 22

3. Algorithm details q Stage II: Event-based Pose Refinement … … Event trajectory Events … ! "#$%& t 2D features at tracking fps Detection Boundary information 1 0 Normalized distance map Results of Stage II Our final results Event stream 23

3. Algorithm details q Stage II: Event-based Pose Refinement … … Event trajectory Events … ! "#$%& t 2D features at tracking fps Detection Boundary information Distance map Before refinement After refinement 24

Results 25

26

4. Results of EventCap q Reconstruction results https://www.xu-lan.com/research.html 27

Summary 28

5. Future Vision of Human Modeling q Aspect of MoCap Date: Throughput The Future (GBps) EventCap Year 29 Figure from Prof. Yaser Sheikh, CMU

Thanks for your attention! 30

The End 31