Bayesian View Synthesis and Image-Based Rendering Principles 1 1 - PowerPoint PPT Presentation

Bayesian View Synthesis and Image-Based Rendering Principles 1 1 2 Sergi Pujades, Frédéric Devernay, Bastian Goldluecke CVPR 2014 1 2 University of Konstanz

Image Based Rendering Input views Input views v 2 v 1 INRIA Grenoble, France CVPR 2014 - 27 June 2014 2

Image Based Rendering Input views Input views v 2 ? v 1 Target view u INRIA Grenoble, France CVPR 2014 - 27 June 2014 2

Image Based Rendering Scene Geometry Input views Input views v 2 ? v 1 Target view u INRIA Grenoble, France CVPR 2014 - 27 June 2014 2

Image Based Rendering Scene Geometry Input views Input views v 2 x v 1 Target view u INRIA Grenoble, France CVPR 2014 - 27 June 2014 2

Image Based Rendering Scene Geometry Input views Input views v 2 x v 1 Target view u INRIA Grenoble, France CVPR 2014 - 27 June 2014 2

Image Based Rendering Scene Geometry Input views Input views v 2 x v 1 Target view u INRIA Grenoble, France CVPR 2014 - 27 June 2014 2

State of the art IBR Continum Scene Geometry less more Light field Lumigraph Texture-mapped models INRIA Grenoble, France CVPR 2014 - 27 June 2014 3

State of the art Unstructured Lumigraph Rendering C. Buehler et al. - SIGGRAPH 2001 8 Desirable Properties • Use of geometric proxies • Unstructured input • Minimal angular deviation • Epipole consistency • Equivalent ray consistency • Resolution sensitivity • Continuity • Real-time INRIA Grenoble, France CVPR 2014 - 27 June 2014 4

State of the art Unstructured Lumigraph Rendering C. Buehler et al. - SIGGRAPH 2001 8 Desirable Properties • Use of geometric proxies • Unstructured input • Minimal angular deviation • Epipole consistency • Equivalent ray consistency • Resolution sensitivity • Continuity • Real-time INRIA Grenoble, France CVPR 2014 - 27 June 2014 4

Minimal angular deviation Input views Input views v 2 v 1 INRIA Grenoble, France CVPR 2014 - 27 June 2014 5

Minimal angular deviation Input views Input views v 2 v 1 u Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 5

Minimal angular deviation Input views Input views v 2 v 1 u Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 5

Minimal angular deviation Input views Input views v 2 v 1 u Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 5

Resolution Sensitivity Input views v 2 Input views v 1 INRIA Grenoble, France CVPR 2014 - 27 June 2014 6

Resolution Sensitivity Input views v 2 u Target view Input views v 1 INRIA Grenoble, France CVPR 2014 - 27 June 2014 6

Resolution Sensitivity Input views v 2 u Target view Input views v 1 INRIA Grenoble, France CVPR 2014 - 27 June 2014 6

Resolution Sensitivity Input views v 2 u Target view Input views v 1 INRIA Grenoble, France CVPR 2014 - 27 June 2014 6

State of the art limitations For both properties: • Minimal angular deviation • Resolution sensitivity No formal deduction of heuristics Manual parameter tuning depending on the scene INRIA Grenoble, France CVPR 2014 - 27 June 2014 7

New properties proposed • Use of geometric proxies • Unstructured input • Minimal angular deviation • Epipole consistency • Equivalent ray consistency • Resolution sensitivity • Formal deduction of heuristics • Physics-based parameters • Continuity • Real-time INRIA Grenoble, France CVPR 2014 - 27 June 2014 8

New properties proposed • Use of geometric proxies • Unstructured input • Minimal angular deviation • Epipole consistency • Equivalent ray consistency • Resolution sensitivity • Formal deduction of heuristics • Physics-based parameters • Continuity • Real-time INRIA Grenoble, France CVPR 2014 - 27 June 2014 8

State of the art Formal deduction Resolution Minimal angular Method Physics-Based sensitivity deviation Parameters Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering INRIA Grenoble, France CVPR 2014 - 27 June 2014 9

State of the art Formal deduction Resolution Minimal angular Method Physics-Based sensitivity deviation Parameters Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy INRIA Grenoble, France CVPR 2014 - 27 June 2014 9

State of the art Formal deduction Resolution Minimal angular Method Physics-Based sensitivity deviation Parameters Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy INRIA Grenoble, France CVPR 2014 - 27 June 2014 9

State of the art Formal deduction Resolution Minimal angular Method Physics-Based sensitivity deviation Parameters Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy Wanner and Goldluecke ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields INRIA Grenoble, France CVPR 2014 - 27 June 2014 9

State of the art Formal deduction Resolution Minimal angular Method Physics-Based sensitivity deviation Parameters Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy Wanner and Goldluecke ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields INRIA Grenoble, France CVPR 2014 - 27 June 2014 9

State of the art Formal deduction Resolution Minimal angular Method Physics-Based sensitivity deviation Parameters Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy Wanner and Goldluecke ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields Our method CVPR 2014 INRIA Grenoble, France CVPR 2014 - 27 June 2014 9

State of the art Formal deduction Resolution Minimal angular Method Physics-Based sensitivity deviation Parameters Buehler et al. SIGGRAPH 2001 Unstructured Lumigraph Rendering Keita Takahashi ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy Wanner and Goldluecke ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields Our method CVPR 2014 INRIA Grenoble, France CVPR 2014 - 27 June 2014 9

Bayesian Approach: Inverse Problem ? v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 10

Bayesian Approach: Inverse Problem ? ? v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 10

Bayesian Approach: Inverse Problem x v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 11

Bayesian Approach: Inverse Problem Scene Geometry x v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 11

Bayesian Approach: Inverse Problem Scene Geometry x ˜ τ i v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 11

Bayesian Approach: Inverse Problem Perfect image Scene Geometry v i ( x ) = ( u � ˜ τ i )( x ) ˜ ˜ τ i x ˜ τ i v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 11

Bayesian Approach: Inverse Problem Perfect image Scene Geometry v i ( x ) = ( u � ˜ τ i )( x ) ˜ ˜ τ i Generative Model Perfect image formation description x ˜ τ i v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 11

Bayesian Approach: Inverse Problem Perfect image Scene Geometry v i ( x ) = ( u � ˜ τ i )( x ) ˜ ˜ τ i Generative Model Perfect image formation description x ˜ τ i v 1 u Input view assuming Lambertian model Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 11

Bayesian Approach: Inverse Problem Perfect image Scene Geometry v i ( x ) = ( u � ˜ τ i )( x ) ˜ ˜ τ i x ˜ τ i v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 12

Bayesian Approach: Inverse Problem Perfect image Scene Geometry v i ( x ) = ( u � ˜ τ i )( x ) ˜ ˜ τ i Observed image v i ( x ) = x ˜ τ i v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 12

Bayesian Approach: Inverse Problem Perfect image Scene Geometry v i ( x ) = ( u � ˜ τ i )( x ) ˜ ˜ τ i Observed image v i ( x ) = ) = ˜ v i ( x ) + x ˜ τ i v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 12

Bayesian Approach: Inverse Problem Perfect image Scene Geometry v i ( x ) = ( u � ˜ τ i )( x ) ˜ ˜ τ i Observed image Sensor noise v i ( x ) = ) = ˜ v i ( x ) + ) + e s ( x ) x ˜ τ i v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 12

Bayesian Approach: Inverse Problem Perfect image Scene Geometry v i ( x ) = ( u � ˜ τ i )( x ) ˜ ˜ τ i Observed image Sensor noise v i ( x ) = ) = ˜ v i ( x ) + ) + e s ( x ) x ˜ τ i Gaussian distribution v 1 u Input view Target view INRIA Grenoble, France CVPR 2014 - 27 June 2014 12