Photometric 3D-reconstruction Jean-Denis D UROU , Yvain Q UAU , and - PowerPoint PPT Presentation

Photometric 3D-reconstruction Jean-Denis D UROU , Yvain Q UÉAU , and Jean M ÉLOU IRIT, Toulouse, France Computational Methods for Inverse Problems in Imaging Como, July 17, 2018 Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 1 / 35

Outline Introduction 1 Shape-from-shading (SfS) 2 Photometric stereo 3 Multi-view shape-from-shading 4 Conclusion and perspectives 5 Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 2 / 35

Introduction Outline Introduction 1 Shape-from-shading (SfS) 2 Photometric stereo 3 Multi-view shape-from-shading 4 Conclusion and perspectives 5 Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 3 / 35

Introduction 3D-scanning ≡ 3D-reconstruction + Color estimation Some applications of 3D-scanning Quality control Architecture Cultural heritage Augmented reality Metrology Different kinds of 3D-reconstruction techniques Palpation ≡ Mechanical process Kinect V1 ≡ Projection of an infra-red pattern Kinect V2 ≡ Time of flight of laser pulses Photographic techniques ≡ Shape-from-X Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 4 / 35

Introduction 3D-scanning ≡ 3D-reconstruction + Color estimation Some applications of 3D-scanning Quality control Architecture Cultural heritage Augmented reality Metrology Different kinds of 3D-reconstruction techniques Palpation ≡ Mechanical process Kinect V1 ≡ Projection of an infra-red pattern Kinect V2 ≡ Time of flight of laser pulses Photographic techniques ≡ Shape-from-X Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 4 / 35

Introduction Main shape-from-X techniques Geometric techniques Photometric techniques N = 1 view Structured light Shape-from-shading (SfS) N = 1 image Shape-from-texture Shape-from-contour Shape-from-shadow N = 1 view Shape-from-defocus Photometric stereo N > 1 images N > 1 views Stereoscopy Multi-view SfS N > 1 images Shape-from-silhouettes Structure-from-motion Multi-view stereo Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 5 / 35

Introduction Main shape-from-X techniques Geometric techniques Photometric techniques N = 1 view Structured light Shape-from-shading (SfS) N = 1 image Shape-from-texture Shape-from-contour Shape-from-shadow N = 1 view Shape-from-defocus Photometric stereo N > 1 images N > 1 views Stereoscopy Multi-view SfS N > 1 images Shape-from-silhouettes Structure-from-motion Multi-view stereo Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 5 / 35

Introduction Geometric techniques vs. Photometric techniques Geometric techniques Goal: match feature points in several images Tools: linear algebra Pros: simple tools; robust Cons: sparse 3D-reconstruction; no color estimation Photometric techniques Goal: explain the color of each pixel in each image Tools: variational methods Pros: dense 3D-reconstruction; color can be estimated Cons: nonlinear models; heavy computations Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 6 / 35

Shape-from-shading (SfS) Outline Introduction 1 Shape-from-shading (SfS) 2 Photometric stereo 3 Multi-view shape-from-shading 4 Conclusion and perspectives 5 Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 7 / 35

Shape-from-shading (SfS) SfS: An intuitive technique to infer 3D-shape This image suffices to infer that the wall is not perfectly flat Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 8 / 35

Shape-from-shading (SfS) SfS for a Lambertian surface (1/2) = × Image Albedo Shading I = × s · n ρ ⇓ Shape-from-shading ���� ���� ���� Image Shading Albedo s : lighting vector n : normal to the surface (3D-shape) n s 3D-shape Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 9 / 35

Shape-from-shading (SfS) SfS for a Lambertian surface (2/2) Classical assumptions Data: graylevel I Albedo ρ ≡ 1 Unknown: normal n such that � n � = 1 Lighting s = [ 0 , 0 , 1 ] ⊤ I = s · n = cos θ s θ The model specifies only one out of the two degrees of freedom of n → Impossible to locally estimate the shape Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 10 / 35

Shape-from-shading (SfS) Non-differential approach vs. differential approach Non-differential approach: unknown normal n �� �� [ I ( u , v ) − s · n ( u , v )] 2 d u d v + λ min R ( n ( u , v )) d u d v n : Ω → R 3 Ω Ω Pros: boundary condition not required; easily discretized Cons: approximate solution; n integrated afterwards Differential approach: unknown depth z 1 [ −∇ z ( u , v ) ⊤ , 1 ] ⊤ I ( u , v ) = s · � �∇ z ( u , v ) � 2 + 1 � �� � ≡ n ( u , v ) under orthographic projection Pros: exact solution; direct estimation of the 3D-shape Cons: boundary condition required; solution not always realistic Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 11 / 35

Shape-from-shading (SfS) Shape-from-shading: Three major drawbacks Knowledge of the albedo ρ ρ ≡ 1 → Arbitrary Difficult to get ρ Knowledge of the lighting vector s s = [ 0 , 0 , 1 ] ⊤ → Arbitrary Possible to get s by calibration SfS is still ill-posed Non-differential approach: Regularization → Approximate solution Differential approach: Boundary condition → Arbitrary Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 12 / 35

Photometric stereo Outline Introduction 1 Shape-from-shading (SfS) 2 Photometric stereo 3 Multi-view shape-from-shading 4 Conclusion and perspectives 5 Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 13 / 35

Photometric stereo A well-posed extension of SfS + − → Albedo ρ Depth z N > 1 lighting vectors s 1 , . . . s N Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 14 / 35

Photometric stereo Claude Monet: a precursor? N ≈ 30 paintings of the cathedral of Rouen (1892-1894) Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 15 / 35

Photometric stereo A linear and local problem [Woodham, 1980] I 1 = ρ s 1 · n = s 1 · m I 2 = ρ s 2 · n = s 2 · m I 3 = ρ s 3 · n = s 3 · m In each pixel, denoting m = ρ n : I i = s i · m , i = 1 . . . N If lighting vectors s i are non-coplanar: Unique exact solution if N = 3 Unique approximate solution if N > 3 Eventually: ρ = � m � and n = m � m � Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 16 / 35

Photometric stereo Photometric stereo: Three major advantages vs. SfS No assumption on the albedo ρ → Realistic ρ is now an additional unknown No other shape-from-X technique can estimate the albedo Well-posed problem → Accurate Requires that the N vectors s i are known (by calibration) Neither regularization, nor boundary condition Linear and local problem → Quick resolution N � 2 � � I i − s i · m Problem to solve in each point: min m i = 1 Integrating the normal field n into a depth map z is global, yet Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 17 / 35

Photometric stereo First example: N = 3 images of a plaster bust I 1 = s 1 · m I 2 = s 2 · m I 3 = s 3 · m Albedo ρ Normal field n Depth map z Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 18 / 35

Photometric stereo Second example: N = 2812 images of a ball of wire 3 images, out of N = 2812 [Wu et al., 2006] Least-squares normal estimation Robust normal estimation Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 19 / 35

Photometric stereo Historical landmarks [Van Diggelen, 1951] First paper on photoclinometry (premise of shape-from-shading) [Horn, 1970] First explicit mention of shape-from-shading [Woodham, 1980] First paper on photometric stereo [Rouy, Tourin, Lions, 1988] First applied mathematics contribution to shape-from-shading (viscosity solutions) [Hayakawa, 1994] Renewed interest for photometric stereo (linked to the popularization of digital photography?) Since 2000: More papers on photometric stereo than on shape-from-shading Since 2010: First convincing applications of photometric stereo Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 20 / 35

Photometric stereo Historical landmarks [Van Diggelen, 1951] First paper on photoclinometry (premise of shape-from-shading) [Horn, 1970] First explicit mention of shape-from-shading [Woodham, 1980] First paper on photometric stereo [Rouy, Tourin, Lions, 1988] First applied mathematics contribution to shape-from-shading (viscosity solutions) [Hayakawa, 1994] Renewed interest for photometric stereo (linked to the popularization of digital photography?) Since 2000: More papers on photometric stereo than on shape-from-shading Since 2010: First convincing applications of photometric stereo Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 20 / 35

Photometric stereo Application to 3D-scanning of faces (1/2) Experimental setup in our lab Two images of a face, out of N = 8 Jean-Denis D UROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 21 / 35