image matching presented by Dmytro Mishkin joint work with - PowerPoint PPT Presentation

Crafting and learning for image matching presented by Dmytro Mishkin joint work with Anastasia Mishchuk, Milan Pultar, Filip Radenovic, Daniel Barath, Michal Perdoch, Jiri Matas 1 2019.07.06, Odesa, EECVC

What is image matching? • The task is to find the correspondences between pixels in two images and/or find a geometrical relations between camera poses • It`s special version is also known as “ wide baseline stereo ”: • large change in viewpoint, illumination, time & occlusions, modality 2019.07.06, Odesa, EECVC 2

Where is image matching useful? 3D rec & SLAM L. Schonberger and J.-M. Frahm, R. Mur-Artal, and J. D. Tardós. Daniel DeTone, Tomasz Malisiewicz, Andrew Rabinovich ORB-SLAM2: an Open-Source SLAM System for Structure-from-Motion Revisited , 2016 MagicLeap SLAM Monocular, Stereo and RGB-D Cameras , arXiv 2016 COLMAP 2019.07.06, Odesa, EECVC 3

Where is image matching useful? Image retrieval Google Landmark Retrieval challenge 2019 winner Image retrieval 2.0 F. Radenovic, series of works 2019.07.06, Odesa, EECVC 4

What is NOT topic of my talk • Semantic correspondences (the object/scene is not the same) Images from Aberman et al. “Neural Best -Buddies: Sparse Cross- Domain Correspondence”, SIGGRAPH 2018 2019.07.06, Odesa, EECVC 5

What is NOT topic of my talk • Short baseline stereo (wait for Anastasiia Mishchuk talk at 17:20) • Optimization methods for stereo • see talks from previous EECVCs: Alexander Shekhovtsov (2017) and Tolga Birdal (2018) 2019.07.06, Odesa, EECVC 6

Wide baseline stereo pipeline Measurement Detector Descriptor region selector Geometrical Matching verification (RANSAC) Measurement Detector Descriptor region selector Single feature visualization 2019.07.06, Odesa, EECVC 7 Image credit: Andrea Vedaldi, ICCVW 2017

Toy example for illustration: matching with OpenCV SIFT Try yourself: https://github.com/ducha-aiki/matching-strategies-comparison 2019.07.06, Odesa, EECVC 8

Toy example for illustration: matching with OpenCV SIFT Recovered 1 st to 2 nd image projection, ground truth 1 st to 2 nd image project, inlier correspondences 2019.07.06, Odesa, EECVC 9

Measurement Detector Descriptor region selector Geometrical Matching verification (RANSAC) Measurement Detector Descriptor region selector Geometric verification (RANSAC) 2019.07.06, Odesa, EECVC 10

Homography: planar surface/static camera Planar surface or static camera → use homography Image with dominant plane → use homography Not sure what to use? → try homography first. Image credit: forums.fast.ai 2019.07.06, Odesa, EECVC 11

Fundamental matrix: general two-view case • General two view geometry in static scene. A corresponding point lies somewhere on a line in the other image. Where on the line - depends on the (unknown) depth • Weaker constraint than homography • Still rigid (no motion in scene assumed) Image credit: https://en.wikipedia.org/wiki/Epipolar_geometry 2019.07.06, Odesa, EECVC 12

RANSAC: fitting the data with gross outliers • What is it OpenCV functions: cv2.findHomography() cv2.findFundamentalMatrix() We will publish soon a python package, which is 2..5 times faster and have an additional tricks inside https://github.com/ducha-aiki/pyransac (save this link, the repo is private now, will clean-up and open next week) Image credit: https://scipy-cookbook.readthedocs.io/items/RANSAC.html 2019.07.06, Odesa, EECVC 13

Pitfails and solutions: homography • Wrong geometry case because of dirty correspondences OpenCV finds 31 wrong inliers in 0.018s. See the same pattern in img1: 3 corrs in line + group. RANSAC H is prone to such case CMP RANSAC finds 6 wrong inliers in 0.004s. 2019.07.06, Odesa, EECVC 14

Pitfails and solutions: homography −1 • Solution: 2-way error metric 𝐼 2−1 = 𝐼 1−2 Check #inliers consistent in opposite direction CMP RANSAC + transfer check: finds 48 correct inliers in 0.005s. D. Mishkin, J. Matas and M. Perdoch. MODS: Fast and Robust Method for Two-View Matching , CVIU 2015, Python CMP RANSAC package will be available soon 2019.07.06, Odesa, EECVC 15

Pitfails and solutions: fundamental matrix • F is too permissive (point to line) 2019.07.06, Odesa, EECVC 16

Pitfails and solutions: fundamental matrix • LAF-check: remember that local feature is oriented circle or ellipse, not just a point. • Check if additional points on circle are consistent with geometry D. Mishkin, J. Matas and M. Perdoch. MODS: Fast and Robust Method for Two-View Matching , CVIU 2015, 2019.07.06, Odesa, EECVC 17

Measurement Detector Descriptor region selector Geometrical Matching verification (RANSAC) Measurement Detector Descriptor region selector Matching strategies 2019.07.06, Odesa, EECVC 18

Nearest neighbor (1NN) strategy Features from img1 are matched to features from img2 You can see, that it is asymmetric and allowing “many -to- one” matches 2019.07.06, Odesa, EECVC 20

Nearest neighbor (NN) strategy Features from img1 are matched to features from img2 OpenCV RANSAC failed to find a good model with NN matching Found 1 st image projection: blue, ground truth: green, Inlier correspondences: yellow 2019.07.06, Odesa, EECVC 21

Mutual nearest neighbor (MNN) strategy Features from img1 are matched to features from img2 Only cross-consistent (mutual NNs) matches are retained. 2019.07.06, Odesa, EECVC 22

Mutual nearest neighbor (MNN) strategy Features from img1 are matched to features from img2 Only cross-consistent (mutual NNs) matches are retained. OpenCV RANSAC failed to find a good model with MNN matching No one-to-many connections, but still bad Found 1 st image projection: blue, ground truth: green , inlier correspondences: yellow 2019.07.06, Odesa, EECVC 23

Second nearest neighbor ratio (SNN) strategy 1 st NN/2 nd NN < 0.8, keep Features from img1 are matched to features from img2 1 st NN - we look for 2 nearest neighbors If both are too similar (1 st NN/2 nd NN ratio > 0.8) → - 2 nd NN 1 st NN discard If 1 st NN is much closer (1 st NN/2 nd NN ratio ≤ 0.8) → - 2 nd NN keep 2 nd NN 1 st NN 1 st NN/2 nd NN > 0.8, drop 2019.07.06, Odesa, EECVC 24 D. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, IJCV 2004

Second nearest neighbor ratio (SNN) strategy NN/SNN < 0.8, keep NN SNN NN SNN SNN OpenCV RANSAC found a model roughly correct No one-to-many connections, but still bad Found 1 st image projection: blue, ground truth: green , NN inlier correspondences: yellow NN/SNN > 0.8, drop 2019.07.06, Odesa, EECVC 25

1st geometrically inconsistent nearest neighbor ratio (FGINN) strategy SNN ratio is cool, but what about symmetrical, or too closely detected features? Ratio test will kill them. Solution: look for 2 nd nearest neighbor, which is far enough from 1 st nearest. 2019.07.06, Odesa, EECVC Mishkin et al., “MODS: Fast and Robust Method for Two - View Matching”, CVIU 2015 26

1st geometrically inconsistent nearest neighbor ratio (FGINN) strategy SNN ratio is cool, but what about symmetrical, or too closely detected features? Ratio test will kill them. Solution: look for 2 nd nearest neighbor, which is far enough from 1 st nearest. 2019.07.06, Odesa, EECVC Mishkin et al., “MODS: Fast and Robust Method for Two - View Matching”, CVIU 2015 27

SNN vs FGINN SNN: roughly correct FGINN: more correspondences, better geometry found 2019.07.06, Odesa, EECVC Mishkin et al., “MODS: Fast and Robust Method for Two - View Matching”, CVIU 2015 28

Symmetrical FGINN Recall, that FGINN is still asymmetric: Matching (Img1 → Img2) ≠ (Img2 → Img1) We can do both (Img1 → Img2) and (Img2 → Img1) and keep all FGINNs (union) or only cross-consistent FGINNs 2019.07.06, Odesa, EECVC 29

Learned filtering strategy (CVPR 2018) Input: matches (x1, y1, x2, y2) [ N x 4 ] Output: scores [ N x 1 ] 2019.07.06, Odesa, EECVC Yi et al. Learning to Find Good Correspondences https://arxiv.org/abs/1711.05971 30

Evaluation on IMW2019 data CVPR 2019 competition https://image-matching-workshop.github.io/ Evaluation Stereo: features ⇨ matching ⇨ Participants ⇨ OpenCV RANSAC ⇨ pose estimation organizers Metric: # of precise enough recovered camera poses (mAP @ 15°) 2019.07.06, Odesa, EECVC 31

Evaluation on IMW2019 data • NN – are you kidding? Never use it alone • SNN is simple and good • FGINN is always a bit better • Symmetrical FGINN rocks • Learning is not that powerful (yet?) 2019.07.06, Odesa, EECVC 32

Measurement Detector Descriptor region selector Geometrical Matching verification (RANSAC) Measurement Detector Descriptor region selector Descriptor: HardNet (NIPS, 2017) Mishchuk et.al. Working hard to know your neighbor’s margins: Local descriptor learning loss. NIPS 2017 2019.07.06, Odesa, EECVC 33

34 Classical way to select good correspondences • Q1: How to find correct correspondences? • Q2: How to filter out features, which do not have a correspondence? • A1: Nearest neighbor by descriptor distance • A2: Threshold the second-to-first nearest ratio (SNN) HardNet: lets use it for training CNNs! 2019.07.06, Odesa, EECVC