Polarimetric 3D Reconstruction and Image Separation Zhaopeng Cui - PowerPoint PPT Presentation

Polarimetric 3D Reconstruction and Image Separation Zhaopeng Cui ETH Zurich 12.19.2019 | |

Outline  Polarization and Polarizer  Polarimetric 3D Reconstruction  Polarimetric Multiple- View Stereo [CVPR’2017]  Poalrimetric Dense Monocular SLAM [CVPR’2018]  Poalrimetric Relative Pose Estimation [ICCV’2019]  Polarimetric Reflection Separation [NeurIPS’2019]  Conclusion Zhaopeng Cui | | 12/20/2019 2

Polarization  Polarization is a characteristic of all transverse waves.  Oscillation which take places in a transverse wave in many different directions is said to be unpolarized.  In an unpolarized transverse wave oscillations may take place in any direction at right angles to the direction in which the wave travels. Direction of propagation of wave Zhaopeng Cui | | 12/20/2019 3

Polarization by Reflection  Unpolarized light can be polarized, either partially or completely, by reflection.  The amount of polarization in the reflected beam depends on the angle of incidence. Zhaopeng Cui | | 12/20/2019 4

Polarizer  Polarizer is made from long chain molecules oriented with their axis perpendicular to the polarizing axis;  These molecules preferentially absorb light that is polarized along their length. Polarizing axis Polarizing axis Zhaopeng Cui | | 12/20/2019 5

Polarimetric Imaging  Images with a Rotating Polarizer  Pixel intensity varies with polarizer angles  We can recover geometric information from polarized images Camera Polarizer ∅ Object Zhaopeng Cui | | 12/20/2019 6

Surface Normal from Polarization Camera Normal Light  Estimation of the azimuth angle 𝜒 (diffuse reflection): 𝜚 𝑞𝑝𝑚 𝐽 𝜚 𝑞𝑝𝑚 = 𝐽 𝑛𝑏𝑦 + 𝐽 𝑛𝑗𝑜 + 𝐽 𝑛𝑏𝑦 − 𝐽 𝑛𝑗𝑜 cos 2 𝜚 𝑞𝑝𝑚 − 𝜚 2 2 Polarizer 𝜒 = 𝜚 or 𝜒 = 𝜚 + 𝜌  Estimation of the zenith angle 𝜄 (diffuse reflection): 𝑜 − 1/𝑜 2 sin 2 𝜄 𝜍 = 𝐽 𝑛𝑏𝑦 − 𝐽 𝑛𝑗𝑜 = 2 + 2𝑜 2 − 𝑜 + 1/𝑜 sin 2 𝜄 + 4 cos 𝜄 𝑜 2 − sin 2 𝜄 𝐽 𝑛𝑏𝑦 + 𝐽 𝑛𝑗𝑜  Estimation of the surface normal v : T = cos 𝜒 sin 𝜄 , − sin 𝜒 sin 𝜄 , − cos 𝜄 T v = 𝑤 𝑦 , 𝑤 𝑧 , 𝑤 𝑨 Zhaopeng Cui | | 12/20/2019 7

Polarimetric 3D Reconstruction Polarimetric Dense Monocular SLAM Polarimetric Multiple-View Stereo [R, t] c ′ c Polarimetric Relative Pose Estimation Zhaopeng Cui | | 12/20/2019 8

Traditional Multi-View Stereo  Given several images of the same object or scene, compute a representation of its 3D shape.  Traditional methods usually failed for featureless objects. Zhaopeng Cui | | Input Sample 12/20/2019 9 Traditional MVS Results

Shape from Surface Normal [Xie et al. CVPR’19] Zhaopeng Cui | | 12/20/2019 10

Challenges Surface normal estimation from polarization is hard:  Refractive distortion: Zenith angle estimation requires the knowledge of the refractive index.  Azimuthal ambiguity: The estimation of the azimuthal angle has 𝜌 -ambiuity. 𝐽 𝜚 𝑞𝑝𝑚 = 𝐽 𝑛𝑏𝑦 + 𝐽 𝑛𝑗𝑜 + 𝐽 𝑛𝑏𝑦 − 𝐽 𝑛𝑗𝑜 cos 2 𝜚 𝑞𝑝𝑚 − 𝜚 2 2 𝜒 = 𝜚 or 𝜒 = 𝜚 + 𝜌  Mixed reflection in real environment. Zhaopeng Cui | | 12/20/2019 11

Mixed Reflection Polarized Unpolarized Polarized Specularly Diffusely Diffusely Incident light reflected light reflected light reflected light Air Object Zhaopeng Cui | | 12/20/2019 12

Polarimetric Multiple View Stereo [CVPR’17] Proposition 1. Under unpolarized illumination, the measured scene radiance from a reflective surface through a linear polarizer at a polarization angle 𝜔 𝑞𝑝𝑚 is 𝐽 ∅ 𝑞𝑝𝑚 = 𝐽 𝑛𝑏𝑦 + 𝐽 𝑛𝑗𝑜 + 𝐽 𝑛𝑏𝑦 − 𝐽 𝑛𝑗𝑜 cos 2 ∅ 𝑞𝑝𝑚 − ∅ , 2 2 where 𝐽 𝑛𝑏𝑦 and 𝐽 𝑛𝑗𝑜 are the maximum and minimum measured radiance. The phase angle ∅ is related to the azimuth angle 𝜒 as follows: 𝜒 𝑗𝑔 𝑞𝑝𝑚𝑏𝑠𝑗𝑨𝑓𝑒 𝑒𝑗𝑔𝑔𝑣𝑡𝑓 𝑠𝑓𝑔𝑚𝑓𝑑𝑢𝑗𝑝𝑜 𝑒𝑝𝑛𝑗𝑜𝑏𝑢𝑓𝑡 𝜒 − 𝜌 ∅ = ቐ 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓 2 𝜌/2 -ambiguity * The azimuthal ( 𝜌 ) ambiguity still holds. Zhaopeng Cui | | 12/20/2019 13

Polarimetric Multiple View Stereo [CVPR’17]  Exploit polarimetric information for dense reconstruction:  Use geometric information to help resolve ambiguities of polarimetric information … View N Structure-from-Motion Initialization (Depth Estimation) View 1 … … Polarized Images from Multiple Resolving π /2 - Ambiguity Phase Angle Estimation Viewpoints (Per-View) Zhaopeng Cui | | 12/20/2019 14

Polarimetric Multiple View Stereo [CVPR’17]  Use geometric information to help resolve 𝜌/2 -ambiguity Initial Depth after depth consistency Azimuth angle map check (after solving 𝜌/2 - ambiguity) Phase angle map 𝑞 = ቊ0, 𝑒𝑗𝑔𝑔𝑣𝑡𝑓𝑒 𝑒𝑝𝑛𝑗𝑜𝑏𝑢𝑓𝑡 𝑔 𝐹 𝑔 = ෍ 𝐸 𝑔 𝑞 + 𝜇 ෍ 𝑇(𝑔 𝑞 , 𝑔 𝑟 ) 𝑞 1, 𝑡𝑞𝑓𝑑𝑣𝑚𝑏𝑠 𝑒𝑝𝑛𝑗𝑜𝑏𝑢𝑓𝑡 𝑞∈𝒬 𝑞,𝑟∈𝒪 𝐸 𝑔 𝑞 enforces consistency with MVS at well-textured regions. 𝑇(𝑔 𝑞 , 𝑔 𝑟 ) enforces neighboring pixels to have similar azimuth angles. Zhaopeng Cui | | 12/20/2019 15

Polarimetric Multiple View Stereo [CVPR’17]  Exploit polarimetric information for dense reconstruction:  Use geometric information to help resolve ambiguities of polarimetric information  Use polarimetric information to improve geometric information … View N Structure-from-Motion Initialization … (Depth Estimation) View 1 Depth Depth Fusion Optimization … … Polarized Images from Multiple Resolving π /2 - Ambiguity Phase Angle Estimation Viewpoints (Per-View) Zhaopeng Cui | | 12/20/2019 16

Polarimetric Multiple View Stereo [ CVPR’17 ]  Iso-depth contour tracing: Propagate reliable depth values along iso-depth contour 1. Phase angle determine the projected surface normal direction (with 𝜌 -ambiguity) 2. From the normal, we can get iso-depth contour on which the pixels have with the same depth 3. Propagate sparse depth values along iso-depth contour 𝜔 Projected Contour surface normals Zhaopeng Cui | | 12/20/2019 17

Polarimetric Multiple View Stereo [CVPR’17]  Per-frame depth optimization ෍ 𝐹 𝑞 𝑒 𝑦, 𝑧 + 𝛿𝐹 𝑒 𝑒 𝑦, 𝑧 + |∆𝑒(𝑦, 𝑧)| (𝑦,𝑧)∈𝒬 constraint from smoothness constraint from azimuth angles known 3D points constraint Zhaopeng Cui | | 12/20/2019 18

Polarimetric multiple view stereo [CVPR17] Zhaopeng Cui | | 12/20/2019 19

Polarimetric Dense Monocular SLAM [CVPR’18] DSLR + Polarizer Filters Polarization camera Sensor Structure Rotate the polarizer filter video with multiple polarized image manually Zhaopeng Cui | | 12/20/2019 20

Polarimetric Dense Monocular SLAM [CVPR’18] Zhaopeng Cui | | 12/20/2019 21

Polarimetric Dense Monocular SLAM [CVPR’18]  Phase angle disambiguation: Using rough depth to solve the 𝜌/2 -ambiguity  Intuition: The correct iso-contour should have less depth variation.  Strategy: Trace two local contours, select the one with less depth variance. Captured Disambiguation Results Phase Angle Map Polarized Images Zhaopeng Cui | | 12/20/2019 22

Polarimetric Dense Monocular SLAM [CVPR’18]  Depth propagation along contours  Issue: wrong propagation caused by noisy 3D points  Solution: Two-View propagation and validation Phase map Inlier Points Propagated Points (Using Single-View) Zhaopeng Cui | | 12/20/2019 23

Polarimetric Dense Monocular SLAM [CVPR’18] Propagate depth in the current Keyframe ′ Φ 𝑢 Consistency ′ Φ 𝑠 Check Inlier New Inlier points 𝒀 𝒋−𝟐 points 𝒀 𝒋 iteration i -1 iteration i Propagate depth in the reference Keyframe Zhaopeng Cui | | 12/20/2019 24

Zhaopeng Cui | | 12/20/2019 25

Traditional Relative Pose Estimation  5-point algorithm: [R, t] c ′ c Zhaopeng Cui | | 12/20/2019 26

Challenges Surface normal estimation from polarization is hard:  Refractive distortion: Zenith angle estimation requires the knowledge of the refractive index  Azimuthal ambiguity: The estimation of the azimuthal angle has 𝜌 -ambiuity 𝐽 𝜚 𝑞𝑝𝑚 = 𝐽 𝑛𝑏𝑦 + 𝐽 𝑛𝑗𝑜 + 𝐽 𝑛𝑏𝑦 − 𝐽 𝑛𝑗𝑜 4 𝑜 possibilities given n cos 2 𝜚 𝑞𝑝𝑚 − 𝜚 2 2 pairs of correspondences. 𝜒 = 𝜚 or 𝜒 = 𝜚 + 𝜌  Mixed reflection in real environment. Zhaopeng Cui | | 12/20/2019 27

Polarimetric Relative Pose Estimation [ICCV’19] Two-point relative pose estimation:  Step 1. Solve the relative rotation: 2 + Rv 2 − v 2 ′ ′ 2 R∈𝑇𝑃(3) Rv 1 − v 1 m𝑗𝑜 R = U diag 1,1, det UV T V T T + v 2 ′ v 2 UΣV T = v 1 ′ v 1 T  Step 2. Solve the relative translation: ′ ∙ t × Rx 𝑗 = t ∙ Rx 𝑗 × x 𝑗 ′ = 0, 𝑗 = 1,2 x 𝑗 ′ ) × (Rx 2 × x 2 ′ ) [R, t] c ′ t = (Rx 1 × x 1 c  Step 3. Hypothesis validation to choose the one which has the largest consensus. Zhaopeng Cui | | 12/20/2019 28