Matching Deformable Objects in Clutter Emanuele Rodol` a USI - PowerPoint PPT Presentation

Matching Deformable Objects in Clutter Emanuele Rodol` a USI Lugano Joint work with L. Cosmo A. Torsello J. Masci M.M. Bronstein ICSEE 2016, Eilat, 18 November 2016 1/35

Shape correspondence problem Isometric 2/35

Shape correspondence problem Isometric Partial 2/35

Shape correspondence problem Isometric Partial Different representation 2/35

Point-wise maps t y j x i X Y Point-wise maps t : X → Y 3/35

Functional maps g f F ( X ) F ( Y ) T Functional maps T : F ( X ) → F ( Y ) Ovsjanikov et al., 2012 3/35

Functional correspondence f ↓ T ↓ g Ovsjanikov et al., 2012 4/35

Functional correspondence f ≈ a 1 + a 2 + · · · + a k ↓ T ↓ g ≈ b 1 + b 2 + · · · + b k Ovsjanikov et al., 2012 4/35

Functional correspondence f ≈ a 1 + a 2 + · · · + a k ↓ ↓ T C Translates Fourier coefficients from Φ to Ψ ↓ ↓ g ≈ b 1 + b 2 + · · · + b k where Φ k = ( φ 1 , . . . , φ k ) , Ψ k = ( ψ 1 , . . . , ψ k ) are Laplace-Beltrami eigenbases Ovsjanikov et al., 2012 4/35

Functional correspondence f ≈ a 1 + a 2 + · · · + a k ↓ ↓ Φ ⊤ T ≈ Ψ k C Translates Fourier coefficients from Φ to Ψ k ↓ ↓ g ≈ b 1 + b 2 + · · · + b k Ψ ⊤ k g = CΦ ⊤ k f where Φ k = ( φ 1 , . . . , φ k ) , Ψ k = ( ψ 1 , . . . , ψ k ) are Laplace-Beltrami eigenbases Ovsjanikov et al., 2012 4/35

Laplacian eigenbases The Laplacian is invariant to isometries φ 1 φ 2 φ 3 φ 4 ψ 1 ψ 2 ψ 3 ψ 4 5/35

Functional correspondence in Laplacian eigenbases C = Ψ ⊤ k TΦ k ⇒ c ij = � ψ i , Tϕ j � For isometric simple spectrum shapes, C is diagonal since ψ i = ± T φ i 6/35

Part-to-full correspondence Full model Partial query 7/35

Partial Laplacian eigenvectors ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 8/35

Partial Laplacian eigenvectors ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 φ 8 φ 9 Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 8/35

Partial Laplacian eigenvectors ζ 2 ζ 3 ζ 4 ζ 5 ζ 6 ζ 7 ζ 8 ζ 9 ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 φ 8 φ 9 Laplacian eigenvectors of a shape with missing parts (Neumann boundary conditions) Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 9/35

Partial Laplacian eigenvectors Functional correspondence matrix C Diagonal angle ≈ area ratio of surfaces Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 10/35

Our setting: Objects in clutter Full model Cluttered partial view 11/35

Functional correspondence with clutter C M S 1 C ⊤ C 12/35

Functional correspondence with clutter C M S 1 S 2 C ⊤ C 12/35

Functional correspondence with clutter C M S 1 S 2 C ⊤ C S 3 12/35

Functional correspondence with clutter C M S 1 S 2 C ⊤ C S 3 S 4 12/35

Laplacian eigenvectors with clutter � Tϕ i , ψ j � S ϕ 5 ϕ 6 ϕ 8 ψ 23 ψ 25 ψ 31 13/35

Functional object-in-clutter T diag( u ) f = diag( v ) g u : M → [0 , 1] v : S → [0 , 1] 14/35

Functional object-in-clutter T diag( u ) f = diag( v ) g T − → 14/35

Functional object-in-clutter C ,θ,u,v � CΦ ⊤ diag( u ) F − Ψ ⊤ diag( v ) G � 2 , 1 + � CΦ ⊤ u − Ψ ⊤ v � 2 min 2 + ρ corr ( C , θ ) + ρ part ( u, v ) 15/35

Functional object-in-clutter C ,θ,u,v � CΦ ⊤ diag( u ) F − Ψ ⊤ diag( v ) G � 2 , 1 + � CΦ ⊤ u − Ψ ⊤ v � 2 min 2 + ρ corr ( C , θ ) + ρ part ( u, v ) Part regularization �� � � 2 �� � � 2 ρ part ( u, v ) = µ 1 udx − − µ 2 udx + vdx vdx M S M S �� � � + µ 3 �∇ M u � dx + �∇ S v � dx M S 15/35

Functional object-in-clutter C ,θ,u,v � CΦ ⊤ diag( u ) F − Ψ ⊤ diag( v ) G � 2 , 1 + � CΦ ⊤ u − Ψ ⊤ v � 2 min 2 + ρ corr ( C , θ ) + ρ part ( u, v ) Part regularization �� � � 2 �� � � 2 ρ part ( u, v ) = µ 1 udx − − µ 2 udx + vdx vdx M S M S � �� � � �� � area preservation part size �� � � + µ 3 �∇ M u � dx + �∇ S v � dx M S � �� � Mumford − Shah 15/35

Functional object-in-clutter C ,θ,u,v � CΦ ⊤ diag( u ) F − Ψ ⊤ diag( v ) G � 2 , 1 + � CΦ ⊤ u − Ψ ⊤ v � 2 min 2 + ρ corr ( C , θ ) + ρ part ( u, v ) Correspondence regularization � � ρ corr ( C , θ ) = µ 4 � C ◦ W ( θ ) � 2 ( C ⊤ C ) 2 | C ⊤ C | ii F + µ 5 ij + µ 6 i � = j i 1 θ W ( θ ) = 0 16/35

Functional object-in-clutter C ,θ,u,v � CΦ ⊤ diag( u ) F − Ψ ⊤ diag( v ) G � 2 , 1 + � CΦ ⊤ u − Ψ ⊤ v � 2 min 2 + ρ corr ( C , θ ) + ρ part ( u, v ) Correspondence regularization � � ρ corr ( C , θ ) = µ 4 � C ◦ W ( θ ) � 2 ( C ⊤ C ) 2 | C ⊤ C | ii + µ 5 + µ 6 F ij � �� � i � = j i slant � �� � � �� � sparsity ≈ orthogonality 1 θ W ( θ ) = 0 16/35

Learning descriptors C ,θ,u,v � CΦ ⊤ diag( u ) F − Ψ ⊤ diag( v ) G � 2 , 1 + · · · min For the data term we use dense descriptor fields. 17/35

Learning descriptors C ,θ,u,v � CΦ ⊤ diag( u ) F − Ψ ⊤ diag( v ) G � 2 , 1 + · · · min For the data term we use dense descriptor fields. Existing isometry-invariant descriptors (HKS, WKS) are affected by clutter and boundary effects Sun et al. 2009; Aubry et al. 2011 17/35

Learning descriptors C ,θ,u,v � CΦ ⊤ diag( u ) F − Ψ ⊤ diag( v ) G � 2 , 1 + · · · min For the data term we use dense descriptor fields. Existing isometry-invariant descriptors (HKS, WKS) are affected by clutter and boundary effects Local descriptors (FPFH, SHOT) are not isometry invariant and sensitive to sampling Sun et al. 2009; Aubry et al. 2011; Rusu et al. 2009; Tombari et al. 2010 17/35

Learning descriptors C ,θ,u,v � CΦ ⊤ diag( u ) F − Ψ ⊤ diag( v ) G � 2 , 1 + · · · min For the data term we use dense descriptor fields. Existing isometry-invariant descriptors (HKS, WKS) are affected by clutter and boundary effects Local descriptors (FPFH, SHOT) are not isometry invariant and sensitive to sampling Our solution: Perform metric learning upon 544-dim SHOT to derive 32-dim descriptors that are robust to clutter, missing parts, and near-isometries Sun et al. 2009; Aubry et al. 2011; Rusu et al. 2009; Tombari et al. 2010; Hadsell et al. 2006; Masci, Boscaini, Bronstein, Vandergheynst 2015 17/35

Performance of learned descriptors ROC CMC 1 80 0.8 Ours 60 True Positive Rate SHOT HKS Hit rate (%) 0.6 WKS 40 0.4 Ours 20 SHOT 0.2 HKS WKS 0 0 0 2 10 3 4 10 3 6 10 3 8 10 3 10 4 0 0.2 0.4 0.6 0.8 1 . . . . False Positive Rate Best matches Tombari et al. 2010 (SHOT); Sun et al. 2009 (HKS); Aubry et al. 2011 (WKS) 18/35

Comparisons 100 80 % Correspondences Ours 60 CPD GTM PFM 40 FM 20 0 0 0.05 0.1 0.15 0.2 0.25 Geodesic Error Methods: Myronenko et al. 2010 (CPD); Rodol` a, Bronstein, Albarelli, Bergamasco, Torsello 2013 (GTM); Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 (PFM); Ovsjanikov et al. 2012 (FM) 19/35

Examples with clutter 20/35

Examples with clutter 20/35

Examples with clutter 20/35

Failure case 21/35

Conclusions Deformable object-in-clutter has been much less investigated than its rigid counterpart, and there is a lack of data and benchmarks. 22/35

Conclusions Deformable object-in-clutter has been much less investigated than its rigid counterpart, and there is a lack of data and benchmarks. We presented a spectral approach that works remarkably well despite the realistic setting. 22/35

Conclusions Deformable object-in-clutter has been much less investigated than its rigid counterpart, and there is a lack of data and benchmarks. We presented a spectral approach that works remarkably well despite the realistic setting. Existing descriptors do not behave well in this setting; we need new descriptors! 22/35

Conclusions Deformable object-in-clutter has been much less investigated than its rigid counterpart, and there is a lack of data and benchmarks. We presented a spectral approach that works remarkably well despite the realistic setting. Existing descriptors do not behave well in this setting; we need new descriptors! Thank you! 22/35

23/35

Examples (no clutter) 24/35

Perturbation analysis: intuition ∆ X φ 1 φ 2 φ 3 X ¯ X ∆ X φ 1 φ 2 φ 3 ¯ ¯ ¯ ∆ ¯ φ 1 φ 2 φ 3 X Ignoring boundary interaction: disjoint parts (block-diagonal matrix) Eigenvectors = Mixture of eigenvectors of the parts 25/35

Perturbation analysis: eigenvalues 8 . 00 · 10 − 2 X 6 . 00 4 . 00 r k Y 2 . 00 0 . 00 10 20 30 40 50 eigenvalue number k ≈ area( X ) Slope r area( Y ) (depends on the area of the cut) Consistent with Weyl’s law for 2-manifolds Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 26/35