Consistency-driven multiple graph matching 一致性驱动的多图匹配模型和算法 Junchi Yan 严骏驰 IBM Research – China (Shanghai) East China Normal University
Outline Introduction on Graph Matching Reference graph based alternating approach, TIP ’15 • Consistency-driven Alternating Optimization for Multi- graph Matching: a Unified Approach, IEEE Transactions on Image Processing, 2015, 24 (3), 994-1009 More ‘distributed’ approach, TPAMI 2015 • Multi-Graph Matching via Affinity Optimization with Graduated Consistency Regularization, IEEE Transactions on Pattern Analysis and Machine Intelligence, accepted on Sep.1 2015, in press Summary
Outline Introduction on Graph Matching Reference graph based alternating approach, TIP’15 • Consistency-driven Alternating Optimization for Multi- graph Matching: a Unified Approach, IEEE Transactions on Image Processing, 2015, 24 (3), 994-1009 More ‘distributed’ approach, TPAMI 2015 • Multi-Graph Matching via Affinity Optimization with Graduated Consistency Regularization, IEEE Transactions on Pattern Analysis and Machine Intelligence, accepted on Sep.1 2015, in press Summary
Graph matching vs. registration Point registration: Often use parametric transformation model between point sets, e.g. RPM, ICP Transformation->correspondence RPM / ICP Node sets ->Transformation->correspondence Graph matching: non-parametric form, no prior transform model between graphs Graph matching RPM: Chui, H., Rangarajan, A.: A new point matching algorithm for non-rigid registration. Computer Vision and Image Understanding 89 (2003) 114 – 141 ICP: Z. Zhang. Iterative point matching for registration of free-form curves and surfaces. IJCV , 1994. Graph matching: Thirty years of graph matching in pattern recognition. IJPRAI, 2004.
Node correspondence by an assignment matrix Assume one-to-one correspondence Solution: assignment matrix, i.e. a partial permutation matrix 4 b 5 detected features 4 detected features Sift feature CNN feature …
Node-wise linear assignment problem Linear Assignment Problem f 4 T f b Node-to-node Hungarian Algorithm affinity/cost matrix (Kuhn & Munkres, 1955) Global optimality is ensured by O(n^3) time complexity where n is the number of nodes
Edge-wise graph matching Build graph by Delaunay Triangulation Edge Similarity Edge Similarity 5 3 1st-order Feature (eg. Local Texture) Feature Matching (linear) c b 2nd-order Feature (eg, Edge Length) Graph Matching (quadratic)
Graph matching: a combinatorial optimization formulation Node Similarity Edge Similarity Node Compatibility Edge Compatibility Affinity maximization Node-Edge Index Steven Gold A. Rangarajan S. Gold and A. Rangarajan, “A graduated assignment algorithm for graph matching,” IEEE Transaction on PAMI , 1996
A more clean writing by an Affinity matrix Affinity matrix: (Leordeanu & Hebert, 2005) Edge-to-edge relations M. Leordeanu M. Hebert M. Leordeanu and M. Hebert, “A spectral technique for correspondence problems using pairwise constraints,” in ICCV, 2005
Node-to-node affinity (Diagonal) 3 b 3 b
Edge-to-edge affinity (Off-Diagonal) 3 b c 5
Quadratic Assignment Problem NP-hard Branch-Bound Koopmans & Beckmann, 1955 Lawler, 1963 Loiola et al, 2007
Spectral Approximation Spectral Method M. Leordeanu M. Hebert is Faster Faster Not Tight Not Discrete T. Cour J. Shi M. Leordeanu and M. Hebert, “A spectral technique for correspondence problems using pairwise constraints,” in ICCV , 2005 P. S. T. Cour and J. Shi, “Balanced graph matching,” in NIPS, 2006
Double-stochastic Approximation Not Convex • S. Gold & Rangarajan, 1996 (K is Indefinite) • Cho et al, 2010 • Leordeanu et al, 2009 • … Not Discrete Slower Gradient Method is More Accurate S. Gold A. Rangarajan M. Cho M. Leordeanu M. Hebert R. Sukthankar K. Lee S. Gold and A. Rangarajan, “A graduated assignment algorithm for graph matching,” IEEE Transaction on PAMI , 1996 M. Cho, J. Lee, and K. M. Lee, “Reweighted random walks for graph matching,” in ECCV , 2010 M. Leordeanu, M. Hebert, and R. Sukthankar, “An integer projected fixed point method for graph matching and map inference,” i n NIPS , 2009
Beyond: Higher-order models 2-Order is Rotation / Scale Invariant 3-Order is Similarity Transformation Invariant 4-Order is Affine Transformation Invariant ?-Order is Non-rigid Invariant Combinatorial Explosion K Complexity Memory for 100 nodes Pair-wise Matrix (381 MB) Triple-wise Tensor (3.7 GB)
Factorized model Node Similarity ? Sparse Edge Similarity Block- Structured F. Zhou and F. D. Torre, “Factorized graph matching,” in CVPR, 2012. Feng Zhou Fernando De la Torre
Factorize the edge affinity F. Zhou and F. D. Torre, “Factorized graph matching,” in CVPR, 2012. Fernando De la Torre F. Zhou
Path-following optimization New objective Original objective Factorization Assuming X is binary Assuming X is orthogonal Interpolation Interpolation Interpolation Concave relaxation Convex relaxation Frank-Wolfe Initialize Frank-Wolfe Always discrete Optimal, Continuous F. Zhou and F. D. Torre, “Factorized graph matching,” in CVPR, 2012. Fernando De la Torre F. Zhou
Two paradigms for two-graph matching Non-factorized paradigm Factorized paradigm Minsu Cho Factorization Spectral / Gradient Feng Zhou Concave Relaxation Convex Relaxation Discrete Rounding M. Leordeanu Fernando De la Torre T. Cour
Matching more than two graphs • More practical problem, with more information to use Infra-red line- Optical image scan image Cartographic data Graphical object query Info fusion PRL’97 3- D weak reconstruction ICCV’15 and indexing, shape analysis SIGGRAPH’12 Exploring collections of 3D models using fuzzy correspondences, SIGGRAPH’12 Multiple Graph Matching with Bayesian Inference , Pattern recognition letters’97 Multi-Image Matching via Fast Alternating Minimization ICCV’15
Motivating illustration G2 G1 Father Mother G3 Interpolating graph (son) G1->G2 G1->G3->G2
Existing multiple GM methods Main categories Designate one of the graphs as the reference, and match all the others to the reference graph • A. Sole-Ribalta, F. Serratosa, Models and algorithms for computing the common labelling of a set of attributed graphs, CVIU 2011 Compute pairwise matchings, based on which improve overall accuracy • D. Pachauriy, R. Kondorx, V. Singh, Solving the multi-way matching problem by permutation synchronization, in NIPS 2013 • Y. Chen, G. Leonidas, and Q. Huang. Matching partially similar objects via matrix completion. In ICML , 2014 One-shot multiple feature set (not graph) matching • Z. Zeng, T. H. Chan, K. Jia, and D. Xu. Finding correspondence from multiple images via sparse and low-rank decomposition. In ECCV , 2012 • X. Zhou, M. Zhu, and K. Daniilidis. Multi-image matching via fast alternating minimization. In ICCV , 2015
Outline Introduction on Graph Matching Reference graph based alternating approach, TIP’15 • Consistency-driven Alternating Optimization for Multi- graph Matching: a Unified Approach, IEEE Transactions on Image Processing, 2015, 24 (3), 994-1009 More ‘distributed’ approach, TPAMI 2015 • Multi-Graph Matching via Affinity Optimization with Graduated Consistency Regularization, IEEE Transactions on Pattern Analysis and Machine Intelligence, accepted on Sep.1 2015, in press Summary
From two-graph to multi-graph (I) • Junchi Yan et al. 2013&2015 • Journal extension • Consistency-driven Alternating Optimization for Multi- graph Matching: a Unified Approach, IEEE Transactions on Image Processing, 2015, 24 (3), 994-1009 • Conference preliminary version • Joint optimization for consistent multiple graph matching, in ICCV 2013
Adding up pairwise affinities Redundancy Basis Assumption: 1) All graphs are of equal size, can be realized by adding dummy nodes or outliers 2) We are matching a collection of related graphs with common structures
Use a basis set of pairwise matchings Basis set: fixed updating Graph: r,f1,f2,f3,f4,u new Alternating updating Iteration 5 Iteration 1 Iteration 2 Iteration 3 Iteration 4 Fix X f1r , X f2r , X f3r , X f4r Fix X ur , X f2r , X f3r , X f4r Fix X ur , X f1r , X f3r , X f4r Fix X ur , X f2r , X f1r , X f4r Fix X ur , X f2r , X f3r , X f1r Update X ur Update X f1r Update X f2r Update X f3r Update X f4r
Two challenges • How to select the reference graph? • How to decide the updating order? Consistency implies accuracy Inconsistent matchings Consistent matchings G1 G1 1 G2 G3 G3 G2 1’’ 1’
How to select the reference graph? Yan et al. 2015 First compute all X ij G 1 G 2 G 6 Find the reference graph by: Consistent i,j=1,2,… G K Then have the basis set: G 3 X 1u , X 2u , X 3u , …, X Nu G 5 Less consistent G 4
How to decide the updating order? G 1 G j K=1,2,… G i G 2 Decide the updating order of X ur in ascending order of C p (X ij , X )
Algorithm (Non-factorized model)
How consistency helps alternating optimization? Order gain Reference graph gain Outlier test Deform test Density test
Pass Factorized formulation convex concave
Recommend
More recommend
