X Disparity Determination in Stereo Vision Lu Sang, Michael Haberl, - PowerPoint PPT Presentation

X Disparity Determination in Stereo Vision Lu Sang, Michael Haberl, Raphael Ullmann 22.07.2017 Lu Sang, Michael Haberl, Raphael Ullmann

Overview Problem Description Algorithms Results Lu Sang, Michael Haberl, Raphael Ullmann

Overview Problem Description Algorithms Results Lu Sang, Michael Haberl, Raphael Ullmann

Problem Description Lu Sang, Michael Haberl, Raphael Ullmann

Problem Description Lu Sang, Michael Haberl, Raphael Ullmann

Problem Description • What is the distance to an object? • How to determine the distance by 2D images? 1https://upload.wikimedia.org/wikipedia/commons/4/49/Roboterhand.mit.Gluehbirne.png Lu Sang, Michael Haberl, Raphael Ullmann

Problem Description • What is the distance to an object? • How to determine the distance by 2D images? Applications • Autonomous driving • Robotics • Object recognition 1 1https://upload.wikimedia.org/wikipedia/commons/4/49/Roboterhand.mit.Gluehbirne.png Lu Sang, Michael Haberl, Raphael Ullmann

Stereo Cameras Lu Sang, Michael Haberl, Raphael Ullmann

Corresponding Pixels Figure: Left Picture [1] Figure: Right Picture Lu Sang, Michael Haberl, Raphael Ullmann

Corresponding Pixels Figure: Left Picture [1] Figure: Right Picture Lu Sang, Michael Haberl, Raphael Ullmann

Corresponding Pixels Figure: Left Picture [1] Figure: Right Picture 1 Calculate for the pixels in the left image costs in the right image Lu Sang, Michael Haberl, Raphael Ullmann

Corresponding Pixels Figure: Left Picture [1] Figure: Right Picture 1 Calculate for the pixels in the left image costs in the right image 2 Pixel with minimal cost is the corresponding pixel Lu Sang, Michael Haberl, Raphael Ullmann

Disparity Lu Sang, Michael Haberl, Raphael Ullmann

Disparity Disparity Pixel distance of related pixels. Lu Sang, Michael Haberl, Raphael Ullmann

Getting distance from Disparity Lu Sang, Michael Haberl, Raphael Ullmann

Getting distance from Disparity Lu Sang, Michael Haberl, Raphael Ullmann

Getting distance from Disparity Lu Sang, Michael Haberl, Raphael Ullmann

Getting distance from Disparity Lu Sang, Michael Haberl, Raphael Ullmann

Distance z ✏ f ☎ b d • Distance z • Focal length of the camera f • Disparity d Lu Sang, Michael Haberl, Raphael Ullmann

Overview Problem Description Algorithms Results Lu Sang, Michael Haberl, Raphael Ullmann

Overview Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter Lu Sang, Michael Haberl, Raphael Ullmann

Overview Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter Lu Sang, Michael Haberl, Raphael Ullmann

Pre-processing: Undistortion Original Picture Undistorted Picture Lu Sang, Michael Haberl, Raphael Ullmann

Pre-processing: Undistortion Original Picture Undistorted Picture Lu Sang, Michael Haberl, Raphael Ullmann

Pre-processing: Rectification Lu Sang, Michael Haberl, Raphael Ullmann

Pre-processing: Rectification Lu Sang, Michael Haberl, Raphael Ullmann

Pre-processing: Rectification Lu Sang, Michael Haberl, Raphael Ullmann

Pre-processing: Rectification Left Picture Right Picture Lu Sang, Michael Haberl, Raphael Ullmann

Pre-processing: Rectification Left Picture Right Picture Lu Sang, Michael Haberl, Raphael Ullmann

Modelling: Energy Function Energy Function argmin d C ♣ p , d q � S ♣ p , d q • Cost C ♣ p , d q for every pixel p and disparities d ✏ 1 , ..., D • Regularization S ♣ p , d q • E.g. penalty for deviation of neighbouring pixels Lu Sang, Michael Haberl, Raphael Ullmann

Overview Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter Lu Sang, Michael Haberl, Raphael Ullmann

Cost Calculation: Comparing Windows Lu Sang, Michael Haberl, Raphael Ullmann

Cost Calculation: Comparing Windows Lu Sang, Michael Haberl, Raphael Ullmann

❞ ✏ ñ ✏ ✁ ❞ ✏ ñ ✏ ✁ r✁ s Cost Calculation: Cross Correlation Lu Sang, Michael Haberl, Raphael Ullmann

ñ ✏ ✁ ❞ ✏ ñ ✏ ✁ r✁ s Cost Calculation: Cross Correlation ❞ ✏ Lu Sang, Michael Haberl, Raphael Ullmann

ñ ✏ ✁ ❞ ✏ ñ ✏ ✁ r✁ s Cost Calculation: Cross Correlation ❞ ✏ Lu Sang, Michael Haberl, Raphael Ullmann

❞ ✏ ñ ✏ ✁ r✁ s Cost Calculation: Cross Correlation ❞ ✏ Total sum = 2 ñ Costs ✏ ✁ 2 Lu Sang, Michael Haberl, Raphael Ullmann

❞ ✏ ñ ✏ ✁ r✁ s Cost Calculation: Cross Correlation ❞ ✏ Total sum = 2 ñ Costs ✏ ✁ 2 Lu Sang, Michael Haberl, Raphael Ullmann

ñ ✏ ✁ r✁ s Cost Calculation: Cross Correlation ❞ ✏ Total sum = 2 ñ Costs ✏ ✁ 2 ❞ ✏ Lu Sang, Michael Haberl, Raphael Ullmann

r✁ s Cost Calculation: Cross Correlation ❞ ✏ Total sum = 2 ñ Costs ✏ ✁ 2 ❞ ✏ Total sum = 4 ñ Costs ✏ ✁ 4 Lu Sang, Michael Haberl, Raphael Ullmann

Cost Calculation: Cross Correlation ❞ ✏ Total sum = 2 ñ Costs ✏ ✁ 2 ❞ ✏ Total sum = 4 ñ Costs ✏ ✁ 4 Normalization and Zero Mean: Values in r✁ 1 , 1 s Lu Sang, Michael Haberl, Raphael Ullmann

Cost Calculation: Result Lu Sang, Michael Haberl, Raphael Ullmann

Cost Calculation: Result Lu Sang, Michael Haberl, Raphael Ullmann

Cost Calculation: Result • Error Rate of NCC: 33 . 09% Lu Sang, Michael Haberl, Raphael Ullmann

Test Data Figure: Left Image Figure: Right Image Lu Sang, Michael Haberl, Raphael Ullmann

Test Data • Middlebury Dataset • Ground Truth • Leaderboard Lu Sang, Michael Haberl, Raphael Ullmann

Algorithm Defect Lu Sang, Michael Haberl, Raphael Ullmann

Overview Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter Lu Sang, Michael Haberl, Raphael Ullmann

Pyramid Scheme Lu Sang, Michael Haberl, Raphael Ullmann

Pyramid Scheme Lu Sang, Michael Haberl, Raphael Ullmann

Pyramid Scheme Lu Sang, Michael Haberl, Raphael Ullmann

Pyramid Scheme: Results Figure: Results of NCC Figure: Results of Pyramid Scheme • Error Rate of NCC: 33 . 09% • Error Rate of Pyramid Scheme: 28 . 1% Lu Sang, Michael Haberl, Raphael Ullmann

Overview Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter Lu Sang, Michael Haberl, Raphael Ullmann

Force Local Consistency • Interpret Cross Correlation as confidence indicator • Use only pixels with high confidence • Replace low confidence by values with high confidence in the window Lu Sang, Michael Haberl, Raphael Ullmann

Force Local Consistency: Confidence Map Figure: Results of NCC Figure: Confidence Map • Black point: trustworthy pixel with correct disparity ( C ♣ p q ➙ T ). • Red point: unreliable pixel with violated disparity ( C ♣ p q ➔ T ). Lu Sang, Michael Haberl, Raphael Ullmann

Force Local Consistency Original C ♣ p ✶ q P W Disparity all violated Violated Unique p ✶ Disparity unique ➔ T max p ✶ P W C ♣ p ✶ q C ♣ p q C ♣ p ✶ q not unique ➙ T Correct Closest p ✶ Disparity 1 Mark all violated pixels. Lu Sang, Michael Haberl, Raphael Ullmann

Force Local Consistency Original C ♣ p ✶ q P W Disparity all violated Violated Unique p ✶ Disparity unique ➔ T max p ✶ P W C ♣ p ✶ q C ♣ p q C ♣ p ✶ q not unique ➙ T Correct Closest p ✶ Disparity 1 Mark all violated pixels. 2 Find the pixel with max NCC coefficient. Lu Sang, Michael Haberl, Raphael Ullmann

Force Local Consistency Original C ♣ p ✶ q P W Disparity all violated Violated Unique p ✶ Disparity unique ➔ T max p ✶ P W C ♣ p ✶ q C ♣ p q C ♣ p ✶ q not unique ➙ T Correct Closest p ✶ Disparity 1 Mark all violated pixels. 2 Find the pixel with max NCC coefficient. 3 Replace the disparity value by using the disparity of new pixel. Lu Sang, Michael Haberl, Raphael Ullmann

Force Local Consistency: Results Figure: Results of NCC Figure: Results of FLC • Error Rate of NCC: 33 . 09% • Error Rate of FLC: 26 . 90% Lu Sang, Michael Haberl, Raphael Ullmann

Overview Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter Lu Sang, Michael Haberl, Raphael Ullmann

Local Penalty Local Penalty 1 Smoothness check 2 Search four directions 3 Punish on NCC coefficient 4 Iterate Lu Sang, Michael Haberl, Raphael Ullmann

Overview Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter Lu Sang, Michael Haberl, Raphael Ullmann

Post Processing: Median Filter ñ Lu Sang, Michael Haberl, Raphael Ullmann

Final Results Figure: Results of FLC Figure: After Post-processing • Error Rate of FLC: 26 . 90% • Error Rate of FLC + Median filter: 21 . 10% Lu Sang, Michael Haberl, Raphael Ullmann

Overview Problem Description Algorithms Results Lu Sang, Michael Haberl, Raphael Ullmann