Layered Depth Images for Multi-View Coding Vincent Jantet ENS-Cachan, Antenne de Bretagne, Campus de Ker Lann, 35170 Bruz – France INRIA Rennes, Bretagne Atlantique, Campus de Beaulieu, 35042 Rennes – France Ph.D. Thesis defense, Rennes, 2012 Collaboration with IRISA, INSA and Brittany Region in Futurim@ge project Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 1 / 43
Applicative context Functionalities 3DTV: Depth feeling by stereo-vision simulation FVV: Live viewpoint selection 3DTV FVV Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 2 / 43
3D video processing scheme Real World Z-Map; Mesh; . . . Acquisition Representation Server side Multi-cam; Z-cam; . . . Transmission TV Screen; Hologram; . . . Displaying Rendering Client side Projection; . . . Virtual World Each choice has an impact on following steps Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 3 / 43
Thesis objectives Intermediate representation Virtual View rendering Compact Fast Bit-Rate scalable Accurate Acquisition Representation Server side Transmission Displaying Rendering Client side Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 4 / 43
SoA: Rendering-optimized representations Multi-View Videos [DTM96] Plenoptic Function (Light Ray) [AB91] Microfacet Billboarding [YSK + 02] . . . Multi-View Video Plenoptic Function Microfacet Billboarding Advantages Limitations Photo-realistic rendering Huge amount of data Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 5 / 43
SoA: Transmission-optimized representations 2D plus depth video (2D+Z) [ISO07] Layered Depth Image (LDI) [SGHS98] Billboard Cloud [DDSD03] Polygon Mesh . . . 2D+Z LDI Billboard Cloud Polygon Mesh Advantages Limitations Compact representation Hard to construct from real scene Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 6 / 43
Contributions LDI representation JPF rendering method Compact representation Point-based projection (naturally remove correlations) method which handle artifacts JPF LDI Acquisition Representation Server side MVC MV+Depth Transmission Displaying Rendering Client side JPF Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 7 / 43
Table of contents View synthesis (JPF) 1 Layered Depth Image (LDI) 2 LDI-based multi-view compression 3 Conclusions 4 Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 8 / 43
Table of contents View synthesis (JPF) 1 Projection algorithm Joint Projection Filling (JPF) Rendering results Layered Depth Image (LDI) 2 LDI-based multi-view compression 3 Conclusions 4 Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 9 / 43
View synthesis: Classical Warping algorithm Reference view View synthesis methods use projection algorithm (warping) Warping algorithm Warping Geometrical projection Input: Texture + Depth map � �� � From reference View Point + Cameras parameters Output: Texture + Depth map � �� � Seen from new View Point Virtual View Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 10 / 43
View Synthesis: Warping common artifacts Disocclusions: Occluded areas which become visible Cracks: Small holes due to sampling Ghosting: Boundaries pixels with mixed foreground/background color Disocclusions Cracks Ghosting Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 11 / 43
View Synthesis: Classical scheme 1 2 Forward Filtering Warping 4 Inpainting 3 Projection 5 Reference Virtual Backward Depth-aided Warping Inpainting Backward Projection Depth-aided Inpainting 1 Forward Warping: Lose pixels connectivity Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 12 / 43
View Synthesis: Classical scheme 1 2 Forward Filtering Warping 4 Inpainting 3 Projection 5 Reference Virtual Backward Depth-aided Warping Inpainting Backward Projection Depth-aided Inpainting 2 Filtering: Fills Cracks and avoids Ghosting Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 12 / 43
View Synthesis: Classical scheme 1 2 Forward Filtering Warping 4 Inpainting 3 Projection 5 Reference Virtual Backward Depth-aided Warping Inpainting Backward Projection Depth-aided Inpainting 3 Backward Warping: Retrieves color from reference view Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 12 / 43
View Synthesis: Classical scheme 1 2 Forward Filtering Warping 4 Inpainting 3 Projection 5 Reference Virtual Backward Depth-aided Warping Inpainting Backward Projection Depth-aided Inpainting 4 Depth Inpainting: Fills disocclusions with mixed FG/BG depth Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 12 / 43
View Synthesis: Classical scheme 1 2 Forward Filtering Warping 4 Inpainting 3 Projection 5 Reference Virtual Backward Depth-aided Warping Inpainting Backward Projection Depth-aided Inpainting 5 Depth-aided Inpainting: Fills disocclusions with mixed FG/BG texture Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 12 / 43
View Synthesis: Classical scheme Limitations Forward Warping: Lose connectivity Depth Inpainting: Can not retrieve structure Texture Inpainting: May fill BG with FB texture Errors are amplified along the process Forward Proj. Need for an accurate virtual depth map synthesizing method Introducing a new Joint Projection Filling method Dir. inpaint. Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 13 / 43
JPF: Joint Projection Filling [Jantet et al., 3D Research] McMillan [McM95] Contribution Also provides pixels Pixel scanning order to avoid connectivity information the use of a zBuffer Projection Reference view Virtual view Projection without zBuffer Process direction BackGround pixels are projected before ForeGround pixels Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 14 / 43
JPF: For rectified views Process direction Ref. View p q New View p ′ q ′ q ′ x = p ′ x + 1 No artifact Consider p and q two pixels projected on p ′ and q ′ Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 15 / 43
JPF: For rectified views Process direction Ref. View p q Overlap New View p ′ q ′ q ′ x = p ′ x + 1 No artifact Consider p and q two pixels q ′ x < p ′ x + 1 Overlap projected on p ′ and q ′ Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 15 / 43
JPF: For rectified views Process direction Ref. View p q Overlap Disocclusion New View q ′ p ′ q ′ x = p ′ x + 1 No artifact Consider p and q two pixels q ′ x < p ′ x + 1 Overlap projected on p ′ and q ′ q ′ x > p ′ x + 1 Disocclusion Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 15 / 43
JPF Generalized: For non rectified views Process direction Disocclusion P q ′ q ′ p ′ � x ≤ P q ′ P q ′ : the last pixel projected q ′ x + 1 No artifact x > P q ′ on row q ′ q ′ x + 1 Disocclusion y Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 16 / 43
JPF: Results Process direction Forward Proj. Navier-Strokes Directional inpaint. JPF Proj. JPF method well synthesize sharp boundaries and thin fingers Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 17 / 43
JPF: Conclusion Advantages One-step projection, without post-processing Handles cracks and disocclusions during the projection Fills disocclusions with background Preserves geometrical structures Limitations Hard to implement on GPU Introduce stretching artifacts if used for texture projection Should be used as a part of a full view synthesis method Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 18 / 43
View Synthesis: Proposed scheme 1 JPF 2 Projection 3 Reference Virtual Backward Depth-Aided Warping Inpainting Backward Projection Depth-aided Inpainting JPF method replaces for: Synthesized Depth used for: Forward Projection Backward Warping Depth Filtering Depth-Aided Inpainting Depth Inpainting Vincent Jantet (ENS-Cachan – FR) Layered Depth Images for Multi-View Coding Ph.D. defense, 2012 19 / 43
Recommend
More recommend
