Robust Feature Matching and Fast GMS Solution Singapore University - PowerPoint PPT Presentation

Robust Feature Matching and Fast GMS Solution Singapore University of Technology and Design (SUTD) Advanced Digital Sciences Center (ADSC) Speaker ： JiaWang Bian( 边佳旺 ) http://jwbian.net/ June 14,2017 6/14/2017 Robust Feature Matching and Fast GMS Solution 1/57

Content • Feature Matching Introduction • Feature Matching • Feature Detector & Descriptor • Matching • RANSAC-based Geometry Estimation (or Verification) • Recent Robust Matchers • CODE (PAMI,2016) • RepMatch (ECCV,2016) • Fast and Robust GMS Solution(CVPR,2017) • Video Demo • Methodology • Algorithm • Share (Material Links) 6/14/2017 Robust Feature Matching and Fast GMS Solution 2/57

Feature Matching Introduction 6/14/2017 Robust Feature Matching and Fast GMS Solution 3/57

Feature Matching Introduction • Feature Matching • Pipeline Detection Description Matching Geometry 6/14/2017 Robust Feature Matching and Fast GMS Solution 4/57

Feature Matching Introduction • Applications Correct Correspondences Geometry between 2 views Similarity(Number of matches) Estimate Camera Pose Image retrieval Localization (SFM) Object Recognition Tracking (SLAM) Loop Closing (SLAM) … Re-localization (SLAM) … 6/14/2017 Robust Feature Matching and Fast GMS Solution 5/57

Sparse Feature Matching • Feature detector & descriptor SIFT Faster Better SURF, PCA-SIFT, ORB, ASIFT, AKAZE, LIFT, … … 6/14/2017 Robust Feature Matching and Fast GMS Solution 6/57

Feature Matching Introduction • Matching Matching Nearest-Neighbor Optimization Others Brute-Force Approximate(FLANN) Matching Algorithms Graph Matching CODE, RepMatch , GMS… 6/14/2017 Robust Feature Matching and Fast GMS Solution 7/57

Feature Matching Introduction • RANSAC-based Geometry Estimation (or Verification) • An example for RANSAC framework (fitting a line) 6/14/2017 Robust Feature Matching and Fast GMS Solution 8/57

Feature Matching Introduction • RANSAC-based Geometry Estimation (or Verification) • Fundamental Matrix (for 3D scenes) • Point to Line (weak, general) • Homography (for 2D scenes) • Point to Point (strong, narrow range) 6/14/2017 Robust Feature Matching and Fast GMS Solution 9/57

Recent Robust Matchers 6/14/2017 Robust Feature Matching and Fast GMS Solution 10/57

Recent Robust Matchers • CODE[1] • For wide-baseline matching. • RepMatch[2] • Based on CODE[1]. • Solve the repeated structure problem. [1] CODE: Coherence Based Decision Boundaries for Feature Correspondence, IEEE TPAMI,2016, Lin et. al. [2] RepMatch: Robust Feature Matching and Pose for Reconstructing Modern Cities, ECCV, 2016,, Lin et. al. 6/14/2017 Robust Feature Matching and Fast GMS Solution 11/57

Recent Robust Matchers (CODE) • Wide-baseline matching 6/14/2017 Robust Feature Matching and Fast GMS Solution 12/57

Recent Robust Matchers (CODE) • Idea 6/14/2017 Robust Feature Matching and Fast GMS Solution 13/57

Recent Robust Matchers (CODE) • Regression models • Likelihood Regression • Affine motion regression -> x • Affine motion regression -> y 6/14/2017 Robust Feature Matching and Fast GMS Solution 14/57

Recent Robust Matchers (CODE) • Likelihood Regression • Train Data • Selected distinctive correspondences(after ratio-test). • Test Data • All feature correspondences. • Features of a correspondence • 𝑌 𝑗 = [𝑦, 𝑧, 𝑒𝑦, 𝑒𝑧, 𝑈 1 , 𝑈 2 , 𝑈 3 , 𝑈 4 ] . • T is a transformation matrix of [s1, r1] to [s2, r2]. • s means scale, r represents rotation. • Labels • 1 for all correspondences • Cost function • Huber function • Non-linear Optimization • Construct Gaussian Similar Matrix • X(Matrix with n x n elements), Y(Matrix with nx1 elements(1) ) • n is the number of train data 6/14/2017 Robust Feature Matching and Fast GMS Solution 15/57

Recent Robust Matchers (CODE) • Affine motion regression • Train Data • The inliers of train data in the likelihood model • Test Data • Correspondences filtered by the likelihood model • Feature Space • Same as the likelihood model • Label • X2, and y2.(x,y represents pixel position, 2 means the second image) • Cost function • Huber function • Non-linear Optimization • Same as before(Gaussian Similar Matrix). 6/14/2017 Robust Feature Matching and Fast GMS Solution 16/57

Recent Robust Matchers (CODE) • Insight (likelihood model) 6/14/2017 Robust Feature Matching and Fast GMS Solution 17/57

Recent Robust Matchers (CODE) • Matching samples 6/14/2017 Robust Feature Matching and Fast GMS Solution 18/57

Recent Robust Matchers (CODE) • Structure from Motion C. Wu, “ VisualSfM: A visual structure from motion system,” 2011[Online]. Available: http://ccwu.me/vsfm/ 6/14/2017 Robust Feature Matching and Fast GMS Solution 19/57

Recent Robust Matchers (CODE) • Run time comparison 6/14/2017 Robust Feature Matching and Fast GMS Solution 20/57

Recent Robust Matchers (RepMatch) • RepMatch 6/14/2017 Robust Feature Matching and Fast GMS Solution 21/57

Recent Robust Matchers (RepMatch) • Repetitive Structure 6/14/2017 Robust Feature Matching and Fast GMS Solution 22/57

Recent Robust Matchers (RepMatch) • Idea 6/14/2017 Robust Feature Matching and Fast GMS Solution 23/57

Recent Robust Matchers (RepMatch) • Structure from Motion 6/14/2017 Robust Feature Matching and Fast GMS Solution 24/57

Recent Robust Matchers (RepMatch) • Structure from Motion 6/14/2017 Robust Feature Matching and Fast GMS Solution 25/57

Recent Robust Matchers (RepMatch) • Structure from Motion 6/14/2017 Robust Feature Matching and Fast GMS Solution 26/57

Recent Robust Matchers (RepMatch) • Structure from Motion 6/14/2017 Robust Feature Matching and Fast GMS Solution 27/57

Fast and Robust GMS Solution 6/14/2017 Robust Feature Matching and Fast GMS Solution 28/57

Video Demo • ORB with GMS vs SIFT with Ratio 6/14/2017 Robust Feature Matching and Fast GMS Solution 29/57

Motivation: Trade-off of quality and speed • Trade-off Matching Nearest-Neighbor Optimization GMS Ratio test Current Methods Graph Matching Popular, Fast, Fast, Robust Slow, Robust Non-Robust 6/14/2017 Robust Feature Matching and Fast GMS Solution 30/57

Methodology: Motion Smoothness • Observation • True matches(green) are visually smooth while false matches(cyan) are not. 6/14/2017 Robust Feature Matching and Fast GMS Solution 31/57

Methodology: Key idea • Inference • According to the Bayesian rule, as true matches are smooth in motion space, consistent matches are thus more likely to be true. • Key idea • Find smooth matches from noisy data as our proposals. • Method Motion Statistics Grid Framework Motion Kernels 6/14/2017 Robust Feature Matching and Fast GMS Solution 32/57

Methodology: Motion Statistics • Motion Statistics Model 6/14/2017 Robust Feature Matching and Fast GMS Solution 33/57

Methodology: Motion Statistics • Distribution • Let 𝑔 𝑏 be one of the n supporting features in region 𝑏 • Let 𝑞 𝑢 , 𝑞 𝑔 be the probability that, feature fa’s nearest neighbor is in region 𝑐 , given {𝑏, 𝑐} view the same and different location, respectively, 6/14/2017 Robust Feature Matching and Fast GMS Solution 34/57

Methodology: Motion Statistics • Event • Assumption Here, 𝑛 is the number of features in region 𝑐 and 𝑁 is the number of features in second image. 𝛾 is a factor added to accommodate violations of assumption caused by repeated patterns. 6/14/2017 Robust Feature Matching and Fast GMS Solution 35/57

Methodology: Motion Statistics • Probability 𝑐 occurs Explanation: If {𝑏 𝑐} view the same location, event 𝑔 𝑏 when 𝑔 𝑏 matches correctly or it matches wrongly but coincidentally lands in region 𝑐 . Explanation: If {𝑏 𝑐} view the different location, event 𝑐 occurs only when 𝑔 𝑏 matches wrongly and coincidentally 𝑔 𝑏 lands in region 𝑐 . 6/14/2017 Robust Feature Matching and Fast GMS Solution 36/57

Methodology: Motion Statistics • Multi-region Generalization 6/14/2017 Robust Feature Matching and Fast GMS Solution 37/57

Methodology: Motion Statistics • Distribution • Mean & Variance 6/14/2017 Robust Feature Matching and Fast GMS Solution 38/57

Methodology: Motion Statistics • Analysis • Partionability • Quantity-Quality equivalence: • Relationship to Descriptors: 6/14/2017 Robust Feature Matching and Fast GMS Solution 39/57

Methodology: Motion Statistics • Experiments on real data: The model is evaluated on Oxford Affine Dataset. Here, we run SIFT matching and label all matches as inlier or outlier according to the ground truth. we count the supporting score for each match in a small region. 6/14/2017 Robust Feature Matching and Fast GMS Solution 40/57

Algorithm: Grid Framework O(N)  O(1)! • Grid Framework • Both images are segmented by a pre-defined grid. • Calculating the Motion Statistics for cell-pairs instead of each feature correspondence. 6/14/2017 Robust Feature Matching and Fast GMS Solution 41/57

Algorithm: Motion Kernels • Basic Motion Kernel 6/14/2017 Robust Feature Matching and Fast GMS Solution 42/57