Computing and Processing Correspondences with Functional Maps Maks - PowerPoint PPT Presentation

Joint diagonalization problem off( P ⊤ Λ M ,k ′ P ) + off( Q ⊤ Λ N ,k ′ Q ) + µ � P ⊤ A − Q ⊤ B � 2 , 1 min P , Q P ⊤ P = I Q ⊤ Q = I s . t . i � = j x 2 Off-diagonal elements penalty off( X ) = � ij Dirichlet energy off( X ) = trace( X ) for k ′ > k If Frobenius norm is used and k ′ = k , due to rotation invariance C = QP ⊤ is the functional correspondence matrix Robust norm � X � 2 , 1 = � j � x j � 2 allows coping with outliers Kovnatsky, Bronstein 2 , Glashoff, Kimmel 2013 7/52

Example of joint diagonalization Mesh with 8.5K vertices Mesh with 850 vertices Kovnatsky, Bronstein 2 , Glashoff, Kimmel 2013 8/52

Example of joint diagonalization Mesh with 8.5K vertices Point cloud with 850 vertices Kovnatsky, Bronstein 2 , Glashoff, Kimmel 2013 9/52

Choice of the basis Functional correspondence matrix C expressed in standard Laplacian eigenbases Kovnatsky, Bronstein 2 , Glashoff, Kimmel 2013 10/52

Choice of the basis Functional correspondence matrix C expressed in coupled approximate eigenbases Kovnatsky, Bronstein 2 , Glashoff, Kimmel 2013 10/52

Multiple shapes C ij A i ≈ A j M j M p M i M 1 M 2 Kovnatsky, Bronstein 2 , Glashoff, Kimmel 2013; Kovnatsky, Glashoff, Bronstein 2016 11/52

Multiple shapes P ⊤ i A i ≈ P ⊤ j A j M j M p M i M 1 M 2 Kovnatsky, Bronstein 2 , Glashoff, Kimmel 2013; Kovnatsky, Glashoff, Bronstein 2016 11/52

Multiple shapes P ⊤ i A i ≈ P ⊤ j A j P j M j P i P p M p M i P 1 P 2 M 1 M 2 Kovnatsky, Bronstein 2 , Glashoff, Kimmel 2013; Kovnatsky, Glashoff, Bronstein 2016 11/52

Multiple shapes p � trace( P ⊤ � � P ⊤ i A i − P ⊤ min i Λ M i P i ) + µ j A j � P 1 ,..., P p i =1 i � = j P ⊤ s . t . i P i = I ‘Synchronization problem’ Matrices P 1 , . . . , P p orthogonally align the p eigenbases Kovnatsky, Bronstein 2 , Glashoff, Kimmel 2013; Kovnatsky, Glashoff, Bronstein 2016 12/52

Computing Functional Maps with Manifold Optimization 13/52

trace( P ⊤ ΛP ) + µ � PA − B � P ⊤ P = I min s . t . P 14/52

trace( P ⊤ ΛP ) + µ � PA − B � P ⊤ P = I min s . t . P Optimization on the Stiefel manifold of orthogonal matrices 14/52

Manifold optimization toy example: eigenvalue problem x ∈ R 3 x ⊤ Ax x ⊤ x = 1 min s . t . Minimization of a quadratic function on the sphere 15/52

Manifold optimization toy example: eigenvalue problem x ∈ S (3 , 1) x ⊤ Ax min Minimization of a quadratic function on the sphere 15/52

Optimization on the manifold: main idea X ( k ) X ( k +1) S Absil et al. 2009 16/52

Optimization on the manifold: main idea ∇ f ( X ( k ) ) X ( k ) P X ( k ) ∇ S f ( X ( k ) ) T X ( k ) S S Absil et al. 2009 16/52

Optimization on the manifold: main idea ∇ f ( X ( k ) ) X ( k ) P X ( k ) α ( k ) ∇ S f ( X ( k ) ) T X ( k ) S S Absil et al. 2009 16/52

Optimization on the manifold: main idea ∇ f ( X ( k ) ) X ( k ) P X ( k ) α ( k ) ∇ S f ( X ( k ) ) T X ( k ) S R X ( k ) X ( k +1) S Absil et al. 2009 16/52

Optimization on the manifold repeat Compute extrinsic gradient ∇ f ( X ( k ) ) Projection: ∇ S f ( X ( k ) ) = P X ( k ) ( ∇ f ( X ( k ) )) Compute step size α ( k ) along the descent direction −∇ S f ( X ( k ) ) Retraction: X ( k +1) = R X ( k ) ( − α ( k ) ∇ S f ( X ( k ) )) k ← k + 1 until convergence ; Absil et al. 2009; Boumal et al. 2014 17/52

Optimization on the manifold repeat Compute extrinsic gradient ∇ f ( X ( k ) ) Projection: ∇ S f ( X ( k ) ) = P X ( k ) ( ∇ f ( X ( k ) )) Compute step size α ( k ) along the descent direction −∇ S f ( X ( k ) ) Retraction: X ( k +1) = R X ( k ) ( − α ( k ) ∇ S f ( X ( k ) )) k ← k + 1 until convergence ; Projection P and retraction R operators are manifold-dependent Absil et al. 2009; Boumal et al. 2014 17/52

Optimization on the manifold repeat Compute extrinsic gradient ∇ f ( X ( k ) ) Projection: ∇ S f ( X ( k ) ) = P X ( k ) ( ∇ f ( X ( k ) )) Compute step size α ( k ) along the descent direction −∇ S f ( X ( k ) ) Retraction: X ( k +1) = R X ( k ) ( − α ( k ) ∇ S f ( X ( k ) )) k ← k + 1 until convergence ; Projection P and retraction R operators are manifold-dependent Typically expressed in closed form Absil et al. 2009; Boumal et al. 2014 17/52

Optimization on the manifold repeat Compute extrinsic gradient ∇ f ( X ( k ) ) Projection: ∇ S f ( X ( k ) ) = P X ( k ) ( ∇ f ( X ( k ) )) Compute step size α ( k ) along the descent direction −∇ S f ( X ( k ) ) Retraction: X ( k +1) = R X ( k ) ( − α ( k ) ∇ S f ( X ( k ) )) k ← k + 1 until convergence ; Projection P and retraction R operators are manifold-dependent Typically expressed in closed form “Black box”: need to provide only f ( X ) and gradient ∇ f ( X ) Absil et al. 2009; Boumal et al. 2014 17/52

Partial Functional Maps 18/52

Partial Laplacian eigenvectors ζ 2 ζ 3 ζ 4 ζ 5 ζ 6 ζ 7 ζ 8 ζ 9 ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 φ 8 φ 9 Laplacian eigenvectors of a shape with missing parts (Neumann boundary conditions) Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 19/52

Partial Laplacian eigenvectors Functional correspondence matrix C Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 20/52

Perturbation analysis: intuition ¯ ¯ ¯ ∆ ¯ φ 1 φ 2 φ 3 M ∆ M φ 1 φ 2 φ 3 ¯ ¯ ¯ λ 1 λ 2 λ 3 M λ 1 λ 2 λ 3 ¯ M φ 1 φ 2 φ 3 ∆ M ¯ ¯ ¯ ∆ ¯ φ 1 φ 2 φ 3 M ¯ ¯ ¯ ≤ ≤ ≤ ≤ ≤ λ 1 λ 1 λ 2 λ 3 λ 2 λ 3 Ignoring boundary interaction: disjoint parts (block-diagonal matrix) Eigenvectors = Mixture of eigenvectors of the parts Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 21/52

Perturbation analysis: eigenvalues 8 . 00 · 10 − 2 M 6 . 00 4 . 00 r k N 2 . 00 0 . 00 10 20 30 40 50 eigenvalue number k ≈ |M| Slope r |N | (depends on the area of the cut) Consistent with Weyl’s law Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 22/52

Perturbation analysis: details ∆ M M t E ¯ ∆ M + t D M M t E ⊤ ∆ ¯ M + t D ¯ M ∆ ¯ M Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 23/52

Perturbation analysis: boundary interaction strength 20 10 Value of f Eigenvector perturbation depends on length and position of the boundary Perturbation strength ≤ c � ∂ M f ( m ) dm , where n � 2 � φ i ( m ) φ j ( m ) � f ( m ) = λ i − λ j i,j =1 j � = i Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 24/52

Partial functional maps Model shape M Query shape N T F Part M ⊆ M ≈ isometric to N Part Data f 1 , . . . , f q ∈ L 2 ( N ) M Query N g 1 , . . . , g q ∈ L 2 ( M ) Partial functional map ( T F f i )( m ) ≈ g i ( m ) , m ∈ M Model M Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 25/52

Partial functional maps Model shape M Query shape N T F Part M ⊆ M ≈ isometric to N Part Data f 1 , . . . , f q ∈ L 2 ( N ) v Query N g 1 , . . . , g q ∈ L 2 ( M ) Partial functional map T F f i ≈ g i · v, v : M → [0 , 1] Model M Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 25/52

Partial functional maps Model shape M Query shape N C Part M ⊆ M ≈ isometric to N Part Data f 1 , . . . , f q ∈ L 2 ( N ) v Query N g 1 , . . . , g q ∈ L 2 ( M ) Partial functional map ≈ B ( v ) , v : M → [0 , 1] CA � φ N � � A = i , f j � L 2 ( N ) � φ M � � Model M B ( v ) = i , g j · v � L 2 ( M ) Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 25/52

Partial functional maps Model shape M Query shape N C Part M ⊆ M ≈ isometric to N Part Data f 1 , . . . , f q ∈ L 2 ( N ) v Query N g 1 , . . . , g q ∈ L 2 ( M ) Partial functional map ≈ B ( v ) , v : M → [0 , 1] CA � φ N � � A = i , f j � L 2 ( N ) � φ M � � Model M B ( v ) = i , g j · v � L 2 ( M ) Optimization problem w.r.t. correspondence C and part v min C ,v � CA − B ( v ) � 2 , 1 + ρ corr ( C ) + ρ part ( v ) Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 25/52

Partial functional maps min C ,v � CA − B ( v ) � 2 , 1 + ρ corr ( C ) + ρ part ( v ) Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 26/52

Partial functional maps min C ,v � CA − B ( v ) � 2 , 1 + ρ corr ( C ) + ρ part ( v ) Part regularization Area preservation � M v ( m ) dx ≈ |N| Spatial regularity = small boundary length (Mumford-Shah) a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein 2 2008 Rodol` 26/52

Partial functional maps min C ,v � CA − B ( v ) � 2 , 1 + ρ corr ( C ) + ρ part ( v ) Part regularization Area preservation � M v ( m ) dx ≈ |N| Spatial regularity = small boundary length (Mumford-Shah) Correspondence regularization Slanted diagonal structure Approximate ortho-projection ( C ⊤ C ) i � = j ≈ 0 rank( C ) ≈ r a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein 2 2008 Rodol` 26/52

Alternating minimization C -step: fix v ∗ , solve for correspondence C C � CA − B ( v ∗ ) � 2 , 1 + ρ corr ( C ) min v -step: fix C ∗ , solve for part v � C ∗ A − B ( v ) � 2 , 1 + ρ part ( v ) min v Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 27/52

Alternating minimization C -step: fix v ∗ , solve for correspondence C C � CA − B ( v ∗ ) � 2 , 1 + ρ corr ( C ) min v -step: fix C ∗ , solve for part v � C ∗ A − B ( v ) � 2 , 1 + ρ part ( v ) min v Iteration 1 2 3 4 Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 27/52

Example of convergence Time (sec.) 0 5 10 15 20 25 10 10 C -step 10 9 v -step 10 8 Energy 10 7 10 6 10 5 10 4 0 20 40 60 80 100 Iteration Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 28/52

Examples of partial functional maps Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 29/52

Examples of partial functional maps Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 29/52

Examples of partial functional maps Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 29/52

Examples of partial functional maps Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 29/52

Partial functional maps vs Functional maps 100 150 100 80 50 % Correspondences PFM 60 Func. maps 40 50 100 20 150 0 0 0.05 0.1 0.15 0.2 0.25 Geodesic error Correspondence performance for different basis size k Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 30/52

Partial correspondence performance Cuts Holes 100 % Correspondences 80 60 40 20 0 0 0.05 0.1 0.15 0.2 0.25 0 0.05 0.1 0.15 0.2 0.25 Geodesic Error Geodesic Error PFM RF IM EN GT SHREC’16 Partial Matching benchmark Rodol` a et al. 2016; Methods: Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 ( PFM ); Sahillio˘ glu, Yemez 2012 (IM); Rodol` a, Bronstein, Albarelli, Bergamasco, Torsello 2012 (GT); Rodol` a et al. 2013 (EN); Rodol` a et al. 2014 (RF) 31/52

Partial correspondence performance Cuts Holes 1 Mean geodesic error 0 . 8 0 . 6 0 . 4 0 . 2 0 20 40 60 80 20 40 60 80 Partiality (%) Partiality (%) PFM RF IM EN GT SHREC’16 Partial Matching benchmark Rodol` a et al. 2016; Methods: Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 ( PFM ); Sahillio˘ glu, Yemez 2012 (IM); Rodol` a, Bronstein, Albarelli, Bergamasco, Torsello 2012 (GT); Rodol` a et al. 2013 (EN); Rodol` a et al. 2014 (RF) 32/52

Partial correspondence (part-to-full) Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 33/52

Partial correspondence (part-to-part) a, Bronstein 2 , Cremers 2016 Litany, Rodol` 33/52

Key observation M N N M C N N C M M slant ∝ | N | slant ∝ | M | |N| |M| a, Bronstein 2 , Cremers 2016 Litany, Rodol` 34/52

Key observation M N N M C NM = C M M C N M C N N slant ∝ | N | |M| |N| | M | a, Bronstein 2 , Cremers 2016 Litany, Rodol` 34/52

Key observation M N N M C NM = C M M C N M C N N slant ∝ | N | |M| | M | = |M| |N| |N| a, Bronstein 2 , Cremers 2016 Litany, Rodol` 34/52

Partial correspondence (part-to-part) a, Bronstein 2 , Cremers 2016 Litany, Rodol` 35/52

Non-rigid puzzle (multi-part) a, Bronstein 2 , Cremers 2016 Litany, Rodol` 35/52

Litany, Bronstein 2 2012

Non-rigid puzzles problem formulation Input M 1 T F 1 Model M N 1 N c Parts N 1 , . . . , N p 2 T F 2 M 2 N c 1 N 2 Output Part N 1 Segmentation M i ⊆ M Located parts N i ⊆ N i M 0 Clutter N c i Missing parts M 0 Correspondences T F i Model M Part N 2 a, Bronstein 2 , Cremers 2016 Litany, Rodol` 37/52

Non-rigid puzzles problem formulation Input u 1 C 1 Model M v 1 Parts N 1 , . . . , N p C 2 u 2 v 2 Output Part N 1 Segmentation u i : M→ [0 , 1] Located parts v i : N i → [0 , 1] u 0 Clutter 1 − v i Missing parts u 0 Correspondences C i Model M Part N 2 a, Bronstein 2 , Cremers 2016 Litany, Rodol` 37/52

Non-rigid puzzles problem formulation p p p � � � min � C i A i ( v i ) − B ( u i ) � 2 , 1 + ρ part ( u i , v i ) + ρ corr ( C i ) C i ,u i ,v i i =1 i =0 i =1 p � s.t. u i = 1 i =0 a, Bronstein 2 , Cremers 2016 Litany, Rodol` 38/52

Convergence example Outer iteration 1 a, Bronstein 2 , Cremers 2016 Litany, Rodol` 39/52

Convergence example Outer iteration 2 a, Bronstein 2 , Cremers 2016 Litany, Rodol` 39/52

Convergence example Outer iteration 3 a, Bronstein 2 , Cremers 2016 Litany, Rodol` 39/52

Convergence example Time (sec) 30 32 34 36 38 40 42 44 46 48 80 90 100 110 120 130 140 150 160 Iteration number a, Bronstein 2 , Cremers 2016 Litany, Rodol` 40/52

Example: “Perfect puzzle” Model/Part Synthetic (TOSCA) Transformation Isometric Clutter No Missing part No Data term Dense (SHOT) a, Bronstein 2 , Cremers 2016 Litany, Rodol` 41/52

Example: “Perfect puzzle” Model/Part Synthetic (TOSCA) Transformation Isometric Clutter No Missing part No Data term Dense (SHOT) Segmentation a, Bronstein 2 , Cremers 2016 Litany, Rodol` 41/52

Example: “Perfect puzzle” Model/Part Synthetic (TOSCA) Transformation Isometric Clutter No Missing part No Data term Dense (SHOT) Correspondence a, Bronstein 2 , Cremers 2016 Litany, Rodol` 41/52

Example: Overlapping parts Model/Part Synthetic (FAUST) Transformation Near-isometric Clutter Yes (overlap) Missing part No Data term Dense (SHOT) Segmentation a, Bronstein 2 , Cremers 2016 Litany, Rodol` 42/52

Example: Overlapping parts Model/Part Synthetic (FAUST) Transformation Near-isometric Clutter Yes (overlap) Missing part No Data term Dense (SHOT) Correspondence a, Bronstein 2 , Cremers 2016 Litany, Rodol` 42/52

Example: Missing parts Model/Part Synthetic (TOSCA) Transformation Isometric Clutter Yes (extra part) Missing part Yes Data term Dense (SHOT) a, Bronstein 2 , Cremers 2016 Litany, Rodol` 43/52

Example: Missing parts Model/Part Synthetic (TOSCA) Transformation Isometric Clutter Yes (extra part) Missing part Yes Data term Dense (SHOT) Segmentation a, Bronstein 2 , Cremers 2016 Litany, Rodol` 43/52

Example: Missing parts Model/Part Synthetic (TOSCA) Transformation Isometric Clutter Yes (extra part) Missing part Yes Data term Dense (SHOT) Correspondence a, Bronstein 2 , Cremers 2016 Litany, Rodol` 43/52

Partial functional correspondence with spatial part model M φ M φ M φ M φ M φ M φ M 1 2 3 4 5 6 T F N N φ N φ N φ N φ N φ N φ N 1 2 3 4 5 6 π j ≈ j |N | � T F φ M i , v · φ N Slanted diagonal: j � L 2 ( N ) ≈ ± δ i,π j |M| Complicated alternating optimization w.r.t. v and C Explicit spatial model v of the part ⇒ O ( n ) complexity! a, Bronstein 2 2016 Litany, Rodol` 44/52

Spectral partial functional correspondence M φ M φ M φ M φ M φ M φ M 1 2 3 4 5 6 T F N φ N ˆ φ N ˆ φ N ˆ φ N ˆ φ N ˆ φ N ˆ 1 2 3 4 5 6 Find a new basis { ˆ i , ˆ φ N i } k � T F φ M φ N i =1 such that j � L 2 ( N ) ≈ δ ij a, Bronstein 2 2016 Litany, Rodol` 45/52

Spectral partial functional correspondence M φ M φ M φ M φ M φ M φ M 1 2 3 4 5 6 T F N φ N ˆ φ N ˆ φ N ˆ φ N ˆ φ N ˆ φ N ˆ 1 2 3 4 5 6 Find a new basis { ˆ i , � k φ N i } k � T F φ M l =1 q lj φ N i =1 such that l � L 2 ( N ) ≈ δ ij New basis functions { ˆ φ N i } k i =1 are localized on N Optimization over coefficients Q = ( q ij ) ⇒ O ( k 2 ) complexity! a, Bronstein 2 2016 Litany, Rodol` 45/52