Photometric*Stereo* October*8,*2013* Dr.*Grant*Schindler* * - PowerPoint PPT Presentation

Photometric*Stereo* October*8,*2013* Dr.*Grant*Schindler* * schindler@gatech.edu*

Mul@ple*Images:*Different*Ligh@ng*

Mul@ple*Images:*Different*Ligh@ng*

Mul@ple*Images:*Different*Ligh@ng*

Religh@ng*the*Scene*

Credit:*Alex*Powell*

Credit:*Alex*Powell*

Credit:*Alex*Powell*

Credit:*Alex*Powell*

Credit:*Alex*Powell*

Credit:*Alex*Powell*

Demo* • Froggies*

Geometry*of*Ligh@ng* Lamber@an*Surface*

Infinitely*Distant*Light*Source*

Acquiring)the)Reflectance)Field)of)a) Human)Face:)Paul)Debevec)et)al * hOp://gl.ict.usc.edu/LightStages/*

More*Info* • hOp://www.pauldebevec.com/* • hOp://www.pauldebevec.com/Research/LS/* • hOp://gl.ict.usc.edu/Research/DigitalEmily/* • hOp://gl.ict.usc.edu/Research/RHL/*

RealTTime*Photometric*Stereo * Surface*Normals* Height*Field* (Eigenvectors*via*Power*Itera@on)* (Integrate*Surface*Normals*via*GaussTSeidel)* 6*ms* 32*ms*

RealTTime*Photometric*Stereo*

Demo* • Processing*Demo*

Reading ** • Szeliski*Book:*Sec@on*12.1.1*(page*580T583)*

3D*Scanner*in*Your*Pocket*

How*does*it*work?* Camera*(FrontTFacing)* Reflect*Light* Emit*Light*

How*does*it*work?* Camera*(FrontTFacing)* Reflect*Light* Emit*Light*

How*does*it*work?* Camera*(FrontTFacing)* Reflect*Light* Emit*Light*

How*does*it*work?* Camera*(FrontTFacing)* Reflect*Light* Emit*Light*

How*does*it*work?* 3D* 2D*

Lambertian Surface ^ n The intensity of a pixel I(u,v) is: ^ s a I(u,v) I ( u , v ) = [ a ( u , v ) ˆ ( u , v )] • [ s 0 ˆ n s ] = b ( u , v ) • s a(u,v) is the albedo of the surface projecting to (u,v). n (u,v) is the direction of the surface normal. s 0 is the light source intensity. s is the direction to the light source. Slide courtesy of David Kriegman

Orthographic Projection • Simplification for light sources that are sufficiently far away b from an object. s • All incoming light rays s b are parallel. s • Thus, while b vectors vary over the surface, b s vector is constant . T s , b 2 T s , b 3 T s ,... ⇒ Bs Pixels: b 1

Images Live in a 3-D Subspace of All Possible Images Light Surface Directions Normals Images 3 = #pixels 4 #pixels 4 3

Images Live in a 3-D Subspace of All Possible Images Light Surface Directions Normals Images 3 = #pixels 4 #pixels 4 3

Background:*Linear*Algebra* • hOp://www.cc.gatech.edu/~phlosob/ transforms/* • Keywords:*Singular*Value*Decomposi@on* • Related:*Image*Compression*

Synthetic Sphere Images Five different lighting conditions Slide courtesy of David Forsyth

Recovered Albedo Slide courtesy of David Forsyth

Recovered Surface Normals Slide courtesy of David Forsyth

Recovered Surface Shape Recovery up to a constant depth error (not absolute depth) Slide courtesy of David Forsyth

Integra@on*Methods* TFrankotTChellappa*(FFT)* TGaussTSeidel*(Itera@ve*Condi@onal*Modes)* TPath*Integra@on*

High Quality Results George Vogiatzis and Carlos Hernandez

Photometric Stereo

Photometric Stereo s 2 s 1 s 3 b 2 b 3 b 1 • Given multiple images of the same surface under different known lighting conditions, can we recover the surface shape? – Yes! (Woodham, 1978)

Photometric Stereo • Assume: – A set of point sources that are infinitely distant – A set of pictures of an object, obtained in exactly the same camera/object configuration but using different sources I 1 = Bs 1 ; I 2 = Bs 2 ; I 3 = Bs 3 ... – A Lambertian object (or the specular component has been identified and removed)

Stereo for a Pixel s 2 I s 1 s 3 b For a pixel ( x,y ) we have n measurements: T b ( x , y ); T b ( x , y ) ... ⇒ I ( x , y ) = Sb ( x , y ) I 1 ( x , y ) = s 1 I 2 ( x , y ) = s 2 Solve an over-constrained linear system for b (with n>3)

What About Shadows? • Shadowed pixels (e.g. attached shadows for a given light source position) are outliers. • � Max � trick can be adapted for this case too: " % T " % " % " % I 1 ( x , y ) 0 0 I 1 ( x , y ) I 1 ( x , y ) 0 0 s 1 $ ' $ ' $ ' $ '     0 0 0 0 b ( x , y ) = $ ' $ ' $ ' $ ' $ T ' $ 0 0 I n ( x , y ) ' $ I n ( x , y ) ' $ 0 0 I n ( x , y ) ' s n # & # & # & # & Pre-multiplying by a thresholded weight matrix zeros the contributions from shadowed pixels

Recovering the Albedo Recall that b ( x , y ) = a ( x , y )ˆ n ( x , y ) ⇒ a ( x , y ) = b ( x , y ) • This gives a check on the normal recovery at a pixel – If the magnitude of a ( x, y ) is greater than 1, there � s a problem Then ˆ n ( x , y ) = b ( x , y )/ a ( x , y )

Recovering the Surface Shape Depth map model (also called Monge patch): Graphic courtesy of David Forsyth z = f ( x , y ) Surface is set of points { x , y , f ( x , y )}

Recovering a surface from normals - 1 • If we write the known • Recall the surface vector g as is written as g 1 ( x , y ) " % $ ' g ( x , y ) = g 2 ( x , y ) ( x , y , f ( x , y )) $ ' g 3 ( x , y ) # & • Then we obtain values • This means the for the partial normal has the derivatives of the form: surface: − f x " % " % ( ) f x ( x , y ) = g 1 ( x , y ) g 3 ( x , y ) 1 $ ' N ( x , y ) = − f y $ ' 2 + f y 2 + 1 f x $ ' ( ) f y ( x , y ) = g 2 ( x , y ) g 3 ( x , y ) # & 1 # & Slide courtesy of David Forsyth

Recovering a surface from normals - 2 • Recall that mixed • We can now recover second partials are the surface height at equal --- this gives us any point by a check. We must integration along have: some path, e.g. ( ) ∂ g 1 ( x , y ) g 3 ( x , y ) x = f ( x , y ) = ∫ f x ( s , y ) ds + ∂ y 0 ( ) ∂ g 2 ( x , y ) g 3 ( x , y ) y f y ( x , t ) dt + c ∫ ∂ x • (or they should be similar, at least) 0 Slide courtesy of David Forsyth

Light Sources

Lambertian Surface ^ n The intensity of a pixel I(u,v) is: ^ s a I(u,v) I ( u , v ) = [ a ( u , v ) ˆ ( u , v )] • [ s 0 ˆ n s ] = b ( u , v ) • s a(u,v) is the albedo of the surface projecting to (u,v). n (u,v) is the direction of the surface normal. s 0 is the light source intensity. s is the direction to the light source. Slide courtesy of David Kriegman

Orthographic Projection • Simplification for light sources that are sufficiently far away b from an object. s • All incoming light rays s b are parallel. s • Thus, while b vectors vary over the surface, b s vector is constant . T s , b 2 T s , b 3 T s ,... ⇒ Bs Pixels: b 1