Depth Camera Based System for Auto-Stereoscopic Displays Fran cois - PowerPoint PPT Presentation

Depth Camera Based System for Auto-Stereoscopic Displays Fran¸ cois de Sorbier Yuko Uematsu Hideo Saito Keio University - Japan IEEE MMSP 2010 October 6 th 2010

Outline 1 Introduction 2 Depth Camera 3 GPU Rendering 4 Conclusion Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 2/21

Auto-Stereoscopic Display Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 3/21

Auto-Stereoscopic Display Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 3/21

Auto-Stereoscopic Display Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 3/21

Input Format of Auto-Stereoscopic Displays Input format depends on the: • number of required input views • interlacing • sub-pixel arrangement • on-board technology • Video plus depth Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 4/21

On-line Video Streams Input Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 5/21

On-line Video Streams Input Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 5/21

On-line Video Streams Input Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 5/21

Goal • Reduce the number of cameras • Generate several virtual viewpoints • Real-time process Depth Camera Based System for Auto-Stereoscopic Displays • Use a depth camera • Mesh based approach • GPU to speed-up the process Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 6/21

Depth Camera What is a Depth Camera? Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

Depth Camera What is a Depth Camera? Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

Depth Camera What is a Depth Camera? Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

Depth Camera What is a Depth Camera? Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

Depth Camera What is a Depth Camera? Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

Depth Camera What is a Depth Camera? Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

Depth Camera What is a Depth Camera? Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 7/21

Depth Camera Limitations • Low resolution (176 × 144) • Color information is missing ⇒ Add a color camera • Viewpoints are different ⇒ A calibration stage is required Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 8/21

Depth Camera Calibration and Rendering • Color camera’s pose estimation • 2D/3D correspondences between the depth map and the color image • Mesh-based approach • Rendering viewpoint corresponds to color camera viewpoint Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 9/21

Depth Camera Calibration and Rendering • Color camera’s pose estimation • 2D/3D correspondences between the depth map and the color image • Mesh-based approach • Rendering viewpoint corresponds to color camera viewpoint Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 9/21

Depth Camera Create content for Auto-Stereoscopic Displays • Video plus depth format: easy • Multiple views: similar to computer graphics stereoscopy. • Projective texture to resolve color occlusions � 0 . 5 0 0 0 � 0 0 . 5 0 0 M texture = (1) × M projection × M modelview 0 0 1 0 0 . 5 0 . 5 0 1 Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 10/21

Depth Camera Create content for Auto-Stereoscopic Displays • Video plus depth format: easy • Multiple views: similar to computer graphics stereoscopy. • Projective texture to resolve color occlusions � 0 . 5 0 0 0 � 0 0 . 5 0 0 M texture = (1) × M projection × M modelview 0 0 1 0 0 . 5 0 . 5 0 1 Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 10/21

Depth Camera Create content for Auto-Stereoscopic Displays Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 11/21

GPU Rendering Rendering limitations • Increasing the number of rendering passes decreases the frame-rate • Redundant computation among the different views • Compute properties of vertices only once using the GPU Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 12/21

GPU Rendering GPU for geometry duplication • 3 programmable units on GPU • Vertex Shader: Vertices properties ⇒ common part • Geometry shader: Geometry assembly ⇒ duplication • Pixel shader: No need Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 13/21

GPU Rendering Limitations of OpenGL • 8 output textures maximum ⇒ only 8 views • Results share a single depth buffer ⇒ Artifacts • Solution: A single shared texture Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 14/21

GPU Rendering Duplication stage Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 15/21

GPU Rendering Duplication stage Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 15/21

GPU Rendering Duplication stage Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 15/21

GPU Rendering Clipping stage • Some triangles can overlap other view’s areas • Apply openGL user’s clipping • Distance values to the borders of the area • If one value is negative, clip • Else keep Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 16/21

GPU Rendering Context of Results • bi-Xeon 2.5Ghz with nVIDIA GeForce GTX285 • Number of triangles: 50000 (176 × 144 × 2) • Resolution of each view : 1024 × 768 • Number of views : 5 (Tridility Screen) Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 17/21

GPU Rendering Results • The Frame-rate depends on the number of views and triangles • 1 view: 51 frames per second • 5 views: 28 frames per second (instead of 15) Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 18/21

Conclusion Summary • Capture system • Depth camera • Color camera • Real-Time mesh based approach • Multi-view rendering applied on GPU Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 19/21

Conclusion Future works • Reduce problem of occlusions • Several layers • Reduce noise of depth map • More cameras (depth/color) • Consistency over time Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 20/21

Conclusion Questions Thank you for your attention Fran¸ cois de Sorbier Depth Camera Based System for Auto-Stereoscopic Displays 21/21