Partial Functional Correspondence Emanuele Rodol` a USI Lugano - PowerPoint PPT Presentation

Partial Functional Correspondence Emanuele Rodol` a USI Lugano Joint work with A. T orsello M.M. Bronstein L. Cosmo D. Cremers SGP, Berlin, 21 June 2016 1/26

Shape correspondence problem Isometric 2/26

Shape correspondence problem Isometric Partial 2/26

Shape correspondence problem Isometric Partial Different representation 2/26

Shape correspondence problem Isometric Partial Different representation Non-isometric 2/26

Point-wise maps t y j x i X Y Point-wise maps t : X → Y 3/26

Functional maps g f F ( X ) F ( Y ) T Functional maps T : F ( X ) → F ( Y ) Ovsjanikov et al., 2012 3/26

Functional correspondence f ↓ T ↓ g Ovsjanikov et al., 2012 4/26

Functional correspondence + a 2 + · · · + a k f ≈ a 1 ↓ T ↓ g ≈ b 1 + b 2 + + b k · · · Ovsjanikov et al., 2012 4/26

Functional correspondence + a 2 + · · · + a k f ≈ a 1 ↓ ↓ C ⊤ Translates Fourier coefficients from Φ to Ψ T ↓ ↓ g ≈ b 1 + b 2 + + b k · · · Ovsjanikov et al., 2012 4/26

Functional correspondence + a 2 + · · · + a k f ≈ a 1 ↓ ↓ C ⊤ Φ ⊤ Translates Fourier coefficients from Φ to Ψ T ≈ Ψ k k ↓ ↓ g ≈ b 1 + b 2 + + b k · · · g ⊤ Ψ k = f ⊤ Φ k C where Φ k = ( φ 1 , . . . , φ k ) , Ψ k = ( ψ 1 , . . . , ψ k ) are Laplace-Beltrami eigenbases Ovsjanikov et al., 2012 4/26

Functional correspondence + a 2 + · · · + a k f ≈ a 1 ↓ ↓ C ⊤ Φ ⊤ Translates Fourier coefficients from Φ to Ψ T ≈ Ψ k k ↓ ↓ g ≈ b 1 + b 2 + + b k · · · G ⊤ Ψ k = F ⊤ Φ k C where Φ k = ( φ 1 , . . . , φ k ) , Ψ k = ( ψ 1 , . . . , ψ k ) are Laplace-Beltrami eigenbases Ovsjanikov et al., 2012 4/26

Functional correspondence in Laplacian eigenbases For isometric simple spectrum shapes C is diagonal since ψ i = ± T φ i 5/26

Our setting Full model Partial query 6/26

Partial Laplacian eigenvectors ζ 2 ζ 3 ζ 4 ζ 5 ζ 6 ζ 7 ζ 8 ζ 9 ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 φ 8 φ 9 Laplacian eigenvectors of a shape with missing parts (Neumann boundary conditions) 7/26

Partial Laplacian eigenvectors Functional correspondence matrix C 8/26

Perturbation analysis: intuition ∆ X φ 1 φ 2 φ 3 X ¯ X ∆ X φ 1 φ 2 φ 3 ¯ ¯ ¯ ∆ ¯ φ 1 φ 2 φ 3 X Ignoring boundary interaction: disjoint parts (block-diagonal matrix) Eigenvectors = Mixture of eigenvectors of the parts 9/26

Perturbation analysis: eigenvalues 8 . 00 · 10 − 2 X 6 . 00 4 . 00 r k Y 2 . 00 0 . 00 10 20 30 40 50 eigenvalue number k ≈ area( X ) Slope r area( Y ) (depends on the area of the cut) Consistent with Weyl’s law 10/26

Perturbation analysis: details ∆ X X t E ¯ ∆ X + t D X X t E ⊤ ∆ ¯ X + t D ¯ X ∆ ¯ X 11/26

Perturbation analysis: details ∆ X X t E ¯ ∆ X + t D X X t E ⊤ ∆ ¯ X + t D ¯ X ∆ ¯ X “How would the Laplacian eigenvalues and eigenvectors of the red part change if we attached a blue part to it?” 11/26

Perturbation analysis: boundary interaction strength 20 10 Value of f Eigenvector perturbation depends on length and position of the boundary Perturbation strength ≤ c � ∂X f ( x ) dx , where n � φ i ( x ) φ j ( x ) � 2 � f ( x ) = λ i − λ j i,j =1 j � = i 12/26

Partial functional maps Given full model shape Y and query shape X corresponding to an unknown approximately isometric part Y ′ ⊂ Y , the partial functional map T : F ( X ) → F ( Y ) is given by Tf = diag( v ) g where v ∈ F ( Y ) is an indicator function of the part solve ⇒ 13/26

Partial functional maps Given full model shape Y and query shape X corresponding to an unknown approximately isometric part Y ′ ⊂ Y , the partial functional map T : F ( X ) → F ( Y ) is given by Tf = diag( v ) g where v ∈ F ( Y ) is an indicator function of the part Optimization problem w.r.t. correspondence and part C ,v � F ⊤ ΦC − G ⊤ diag( η ( v )) Ψ � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min where η ( t ) = 1 2 (tanh(2 t − 1) + 1) saturates the part membership function 14/26

Partial functional maps C ,v � F ⊤ ΦC − G ⊤ diag( η ( v )) Ψ � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min 15/26

Partial functional maps C ,v � F ⊤ ΦC − G ⊤ diag( η ( v )) Ψ � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min Part regularization � � 2 � � ρ part ( v ) = µ 1 area( X ) − η ( v ) dx + µ 2 �∇ Y η ( v ) � dx Y Y Bronstein and Bronstein 2008 15/26

Partial functional maps C ,v � F ⊤ ΦC − G ⊤ diag( η ( v )) Ψ � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min Part regularization � � 2 � � ρ part ( v ) = µ 1 area( X ) − η ( v ) dx + µ 2 �∇ Y η ( v ) � dx Y Y � �� � � �� � area preservation Mumford − Shah Bronstein and Bronstein 2008 15/26

Partial functional maps C ,v � F ⊤ ΦC − G ⊤ diag( η ( v )) Ψ � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min Part regularization � � 2 � � ρ part ( v ) = µ 1 area( X ) − η ( v ) dx + µ 2 �∇ Y η ( v ) � dx Y Y � �� � � �� � area preservation Mumford − Shah Correspondence regularization � � ρ corr ( C ) = µ 3 � C ◦ W � 2 ( C ⊤ C ) 2 (( C ⊤ C ) ii − d i ) 2 F + µ 4 ij + µ 5 i � = j i Bronstein and Bronstein 2008 15/26

Partial functional maps C ,v � F ⊤ ΦC − G ⊤ diag( η ( v )) Ψ � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min Part regularization � � 2 � � ρ part ( v ) = µ 1 area( X ) − η ( v ) dx + µ 2 �∇ Y η ( v ) � dx Y Y � �� � � �� � area preservation Mumford − Shah Correspondence regularization � � ρ corr ( C ) = µ 3 � C ◦ W � 2 ( C ⊤ C ) 2 (( C ⊤ C ) ii − d i ) 2 + µ 4 + µ 5 F ij � �� � i � = j i slant � �� � � �� � rank ≈ r ≈ orthogonality Bronstein and Bronstein 2008 15/26

Partial functional maps C ,v � F ⊤ ΦC − G ⊤ diag( η ( v )) Ψ � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min Part regularization � � 2 � � ρ part ( v ) = µ 1 area( X ) − η ( v ) dx + µ 2 �∇ Y η ( v ) � dx Y Y � �� � � �� � area preservation Mumford − Shah Correspondence regularization � � ρ corr ( C ) = µ 3 � C ◦ W � 2 ( C ⊤ C ) 2 (( C ⊤ C ) ii − d i ) 2 + µ 4 + µ 5 F ij � �� � i � = j i slant � �� � � �� � rank ≈ r ≈ orthogonality F , G = dense SHOT descriptor in all our experiments Bronstein and Bronstein 2008; Tombari et al. 2010 15/26

Structure of partial functional correspondence 4 2 0 0 20 40 60 80 100 C ⊤ C C W singular values 16/26

Alternating minimization C -step: fix v ∗ , solve for correspondence C C � F ⊤ ΦC − G ⊤ diag( η ( v ∗ )) Ψ � 2 , 1 + ρ corr ( C ) min v -step: fix C ∗ , solve for part v � F ⊤ ΦC ∗ − G ⊤ diag( η ( v )) Ψ � 2 , 1 + ρ part ( v ) min v 17/26

Alternating minimization C -step: fix v ∗ , solve for correspondence C C � F ⊤ ΦC − G ⊤ diag( η ( v ∗ )) Ψ � 2 , 1 + ρ corr ( C ) min v -step: fix C ∗ , solve for part v � F ⊤ ΦC ∗ − G ⊤ diag( η ( v )) Ψ � 2 , 1 + ρ part ( v ) min v Iteration 1 2 3 4 17/26

Example of convergence Time (sec.) 0 5 10 15 20 25 10 10 C -step 10 9 ν -step 10 8 Energy 10 7 10 6 10 5 10 4 0 20 40 60 80 100 Iteration 18/26

Examples of partial functional maps 19/26

Examples of partial functional maps 19/26

Examples of partial functional maps 19/26

Examples of partial functional maps 19/26

Partial functional maps vs Functional maps 100 150 100 80 50 % Correspondences PFM 60 Func. maps 40 50 100 20 150 0 0 0.05 0.1 0.15 0.2 0.25 Geodesic error Correspondence performance for different rank values k 20/26

Partial correspondence performance (SHREC’16) Cuts Holes 100 % Correspondences 80 60 40 20 0 0 0.05 0.1 0.15 0.2 0.25 0 0.05 0.1 0.15 0.2 0.25 Geodesic Error Geodesic Error PFM RF IM EN GT SHREC’16 Partial Matching benchmark Rodol` a et al. 2016; Methods: Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 ( PFM ); Sahillio˘ glu and Yemez 2012 (IM); Rodol` a et al. 2012 (GT); Rodol` a et al. 2013 (EN); Rodol` a et al. 2014 (RF) 21/26

Conclusions Partial deformable shape matching is a challenging problem, much less investigated than the full case 22/26

Conclusions Partial deformable shape matching is a challenging problem, much less investigated than the full case Functional correspondence has a slanted diagonal structure, with slant equal to area ratio 22/26

Conclusions Partial deformable shape matching is a challenging problem, much less investigated than the full case Functional correspondence has a slanted diagonal structure, with slant equal to area ratio Spectral shape analysis still yields good results! 22/26