Incremental-LDI for Multi-View Coding Generation & - PowerPoint PPT Presentation

Incremental-LDI for Multi-View Coding Generation & Representation Vincent JANTET IRISA Rennes - TEMICS Team FRANCE Supervisors: Christine Guillemot - IRISA Luce Morin - IETR - INSA Gaël Sourimant - IRISA April 1, 2010 Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 1 / 19

Context Multi-view videos Desired functionalities: 3DTV: Depth feeling by stereovision simulation. Figure: Multi-view acquisition FVV: (for Free Viewpoint Video) Live viewpoint selection. Problems Acquisition: Synchronization, calibration. . . Compression: Compact representation of the huge amount of data. Rendering: Photo-realistic virtual view generation. Figure: 3D rendering Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 2 / 19

Context Multi-view videos Desired functionalities: 3DTV: Depth feeling by stereovision simulation. Figure: Multi-view acquisition FVV: (for Free Viewpoint Video) Live viewpoint selection. Problems Acquisition: Synchronization, calibration. . . Compression: Compact representation of the huge amount of data. Rendering: Photo-realistic virtual view generation. Figure: 3D rendering Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 2 / 19

Table on contents Introduction 1 Incremental-LDI construction scheme 2 Ghosting 3 Results 4 Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 3 / 19

Outline Introduction 1 Incremental-LDI construction scheme 2 Ghosting 3 Results 4 Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 4 / 19

View generation (Warping algorithm) Image-based rendering Depth i View i Input : View and associated Depth map Output : New viewpoint (texture & depth). Problems Warping Sampling: Visual artifacts ⇒ Inpainting Disocclusion: Unknown texture. ⇒ Extra information ( LDI ). Figure: Disocclusion Figure: Warping algorithm Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 5 / 19

LDI (Layered Depth Image) - [SGHS98, YLKH07] A Layered Depth Image is a set of many layers constituted by depth pixels. Contains texture of occluded area. · · · 1 st layer 2 nd layer 3 rd layer 4 th layer Figure: First layers of an LDI frame. Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 6 / 19

Classical LDI construction [CSY07] Merging policy Every input views are warped onto a reference viewpoint, Eliminate pixels and then merged together. described twice. View i Reference . . . Merging LDI . viewpoint View j Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 7 / 19

Advantages and limits Advantages Disocclusion: Filled by real texture. Camera freedom: Virtual camera can move inside a large area. Compactness: Eliminate some correlated pixels and reduce data size. Limits Compression: Many layers, partially empty with scattered pixels distribution. Visual artifacts: Ghosting, Bluring, . . . Figure: Ghosting artifacts. Figure: Scattered pixels distribution. Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 8 / 19

Outline Introduction 1 Incremental-LDI construction scheme 2 Ghosting 3 Results 4 Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 9 / 19

Incremental-LDI construction scheme (I-LDI) Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view. This information is warped back into the I-LDI. Viewpoint i Warp Ref to i . I-LDI Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 10 / 19

Incremental-LDI construction scheme (I-LDI) Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view. This information is warped back into the I-LDI. Warp i to Ref View i Exclusion difference Viewpoint i Warp Ref to i . I-LDI Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 10 / 19

Incremental-LDI construction scheme (I-LDI) Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view. This information is warped back into the I-LDI. Warp i to Ref View i Exclusion difference Insert Viewpoint i Warp Ref to i . I-LDI Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 10 / 19

Incremental-LDI construction scheme (I-LDI) Current I-LDI is warped iteratively on every other viewpoint. Residual information is computed by exclusion difference with the real view. This information is warped back into the I-LDI. Warp j to Ref View j Exclusion difference Insert Viewpoint j Warp Ref to j . I-LDI Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 10 / 19

Outline Introduction 1 Incremental-LDI construction scheme 2 Ghosting 3 Results 4 Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 11 / 19

Ghosting Artifacts Ghosting is due to pixels with blended color between background and foreground. Detect depth discontinuity [Canny]. Classify background and foreground pixels near each boundaries. Ignore background blended pixels from data. First layer of an I-LDI frame. Figure: Ghosting Depth Map Canny Boundaries Ghost artifacts removal edges removal Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 12 / 19

Outline Introduction 1 Incremental-LDI construction scheme 2 Ghosting 3 Results 4 Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 13 / 19

First layers of an LDI and an I-LDI Comparison. LDI frame: many pixels, scattered pixels distribution, ... ... (a) 1 st layer (b) 2 nd layer (c) 3 rd layer (d) 4 th layer (e) I-LDI frame: less layer and less pixels, compact distribution, ... ... (f) 1 st layer (g) 2 nd layer (h) 3 rd layer (i) 4 th layer (j) Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 14 / 19

LDI & I-LDI comparison. (a) Layers completion rate. (b) Pixels ratio taken from different views. Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 15 / 19

PSNR Original LDI Original 30.23 I-LDI 30.22 46.26 SSIM Original LDI Original 79.62% I-LDI 79.68% 99.52% Figure: PSNR & SSIM Figure: LDI rendering. (a) LDI. (b) I-LDI. Figure: Differences. Figure: I-LDI rendering. Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 16 / 19

Rendering result. Figure: Rendering result of an I-LDI. Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 17 / 19

Rendering result. Figure: Rendering result of an I-LDI. Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 17 / 19

Rendering result. Figure: Rendering result of an I-LDI. Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 17 / 19

Conclusion Advantages Limits 80% less pixels for the same Sampling artifacts. quality. Some textures will never Less correlation between layers. be inserted into the I-LDI. Compact pixels distribution. Future works. Look for an efficient I-LDI compression algorithm. Questions? Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 18 / 19

References I X. Cheng, L. Sun, and S. Yang. Generation of layered depth images from multi-view video. Image Processing, 2007. ICIP 2007. IEEE International Conference on , 5:V –225–V –228, 16 2007-Oct. 19 2007. J. Shade, S. Gortler, L. He, and R. Szeliski. Abstract layered depth images. 1998. S.-U. Yoon, E.-K. Lee, S.-Y. Kim, and Y.-S. Ho. A framework for representation and processing of multi-view video using the concept of layered depth image. Journal of VLSI Signal Processing Systems for Signal Image and Video Technology , 46:87–102, 2007. Vincent JANTET (IRISA - France) I-LDI for Multi-View Coding April 1, 2010 19 / 19