Light, Camera and Shading CS 543 / ECE 549 Saurabh Gupta Spring - PowerPoint PPT Presentation

Light, Camera and Shading CS 543 / ECE 549 – Saurabh Gupta Spring 2020, UIUC http://saurabhg.web.illinois.edu/teaching/ece549/sp2020/ Many slides adapted from S. Seitz, L. Lazebnik, D. Hoiem, D. Forsyth

Recap 𝐸 - + 1 1 𝐸 = 1 𝑔 𝑒 𝜒 = 𝑢𝑏𝑜 &' 2𝑔

Recap

Recap é ù 2 æ ö p d ç ÷ = ê a ú 4 E cos L ç ÷ 4 f ê ú è ø ë û

Recap transparent sub-surface reflection phosphoresence fluoroscence

Overview • Lambertian reflection model • Shape from shading • Color

Most surfaces have both Specularity = spot where specular reflection dominates (typically reflects light source) Typically, specular component is small Slide from D. Hoiem Photo: northcountryhardwoodfloors.com

Specular reflection Picture source Slide from L. Lazebnik

When light hits a typical surface • Some light is absorbed absorption • Some light is reflected diffusely – Independent of viewing direction diffuse reflection • Some light is reflected specularly – Light bounces off (like a mirror), specular depends on viewing direction reflection Θ Θ Slide from D. Hoiem

Bidirectional Reflectance Distribution Function (BRDF) • How bright a surface appears when viewed from one direction when light falls on it from another • Definition: ratio of the radiance in the emitted direction to irradiance in the incident direction Source: Steve Seitz

Lambertian reflectance model Some light is absorbed (function of albedo 𝜍 ) Remaining light is scattered, equally in all directions. Examples: soft cloth, concrete, matte paints light light source source diffuse reflection 𝜍 absorption (1 − 𝜍) Slide from D. Hoiem

Intensity and Surface Orientation Intensity depends on illumination angle because less light comes in at oblique angles. 𝜍 = albedo 𝑻 = directional source 𝑶 = surface normal I = reflected intensity 𝐽 𝑦 = 𝜍 𝑦 𝑻 ⋅ 𝑶(𝑦) Slide: Forsyth

Photometric stereo (shape from shading) • Can we reconstruct the shape of an object based on shading cues? Slide from L. Lazebnik

Photometric stereo Assume: • A Lambertian object • A local shading model (each point on a surface receives light only from sources visible at that point) • A set of known light source directions • A set of pictures of an object, obtained in exactly the same camera/object configuration but using different sources • Orthographic projection Goal: reconstruct object shape and albedo S 2 S n S 1 ??? F&P 2 nd ed., sec. 2.2.4 Slide from L. Lazebnik

Example 1 Recovered Recovered normal Recovered surface albedo field model F&P 2 nd ed., sec. 2.2.4 Slide from L. Lazebnik

Example 2 Input … Recovered Recovered surface model albedo Recovered normal field x y z Slide from L. Lazebnik

Image model • Known: source vectors S j and pixel values I j ( x , y ) • Unknown: surface normal N ( x , y ) and albedo ρ ( x , y ) F&P 2 nd ed., sec. 2.2.4 Slide from L. Lazebnik

Image model • Known: source vectors S j and pixel values I j ( x , y ) • Unknown: surface normal N ( x , y ) and albedo ρ ( x , y ) • Assume that the response function of the camera is a linear scaling by a factor of k • Lambert’s law: ( ) ( ) ( ) = r × I ( x , y ) k x , y N x , y S j j ( ( ) ( ) ) = r × x , y N x , y ( k S ) j = × g ( x , y ) V j F&P 2 nd ed., sec. 2.2.4 Slide from L. Lazebnik

Least squares problem • For each pixel, set up a linear system: ! $ ! $ T V I 1 ( x , y ) # & 1 # & # & T I 2 ( x , y ) # & V 2 g ( x , y ) = # & # & ! ! # & # & # & I n ( x , y ) T # & V " % " % n ( n × 1) ( n × 3) (3 × 1) known known unknown • Obtain least-squares solution for g ( x,y ) (which we defined as N ( x , y ) r ( x,y ) ) • Since N ( x,y ) is the unit normal, r ( x,y ) is given by the magnitude of g ( x,y ) Slide from L. Lazebnik • Finally, N ( x,y ) = g ( x,y ) / r ( x,y ) F&P 2 nd ed., sec. 2.2.4

Synthetic example Recovered albedo Recovered normal field F&P 2 nd ed., sec. 2.2.4 Slide from L. Lazebnik

Recovering a surface from normals Recall the surface is If we write the estimated written as vector g as æ ö g ( x , y ) ( x , y , f ( x , y )) ç 1 ÷ = g ( x , y ) g ( x , y ) ç ÷ 2 ç ÷ This means the normal g ( x , y ) è ø 3 has the form: Then we obtain values for the partial derivatives of æ ö f ç x ÷ the surface: 1 = N ( x , y ) f ç ÷ y + + 2 2 f f 1 = ç ÷ f ( x , y ) g ( x , y ) / g ( x , y ) 1 x y è ø x 1 3 = f ( x , y ) g ( x , y ) / g ( x , y ) y 2 3 F&P 2 nd ed., sec. 2.2.4 Slide from L. Lazebnik

Recovering a surface from normals We can now recover the Integrability : for the surface height at any point surface f to exist, the by integration along some mixed second partial path, e.g. derivatives must be equal: ¶ x = ∫ ( g ( x , y ) / g ( x , y )) f ( x , y ) = f x ( s ,0) ds + 1 3 ¶ y 0 ¶ y ∫ ( g ( x , y ) / g ( x , y )) f y ( x , t ) dt + C 2 3 ¶ x 0 (for robustness, should (in practice, they should take integrals over many at least be similar) different paths and average the results) F&P 2 nd ed., sec. 2.2.4 Slide from L. Lazebnik

Surface recovered by integration F&P 2 nd ed., sec. 2.2.4 Slide from L. Lazebnik

Limitations • Orthographic camera model • Simplistic reflectance and lighting model • No shadows • No interreflections • No missing data • Integration is tricky Slide from L. Lazebnik

Finding the direction of the light source I ( x,y ) = N ( x,y ) · S ( x,y ) Full 3D case: N S P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001 Slide by L. Lazebnik

Finding the direction of the light source Consider points on the occluding contour : N z = 0 N z positive N z negative Image Projection direction ( z ) P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001 Slide by L. Lazebnik

Finding the direction of the light source I ( x,y ) = N ( x,y ) · S ( x,y ) Full 3D case: N S For points on the occluding contour, N z = 0: P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001 Slide by L. Lazebnik

Finding the direction of the light source P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001 Slide by L. Lazebnik

Application: Detecting composite photos Real photo Fake photo M. K. Johnson and H. Farid, Exposing Digital Forgeries by Detecting Inconsistencies in Lighting, ACM Multimedia and Security Workshop, 2005. Slide by L. Lazebnik

Overview • Lambertian reflection model • Shape from shading • Color