madmm a generic algorithm for non smooth optimization on
play

MADMM: a generic algorithm for non-smooth optimization on manifolds - PowerPoint PPT Presentation

MADMM: a generic algorithm for non-smooth optimization on manifolds Michael Bronstein Faculty of Informatics Perceptual Computing Group University of Lugano Intel Corporation Switzerland Israel Louvain-la-Neuve, 25 September 2015 1/54


  1. Manifold ADMM Non-smooth manifold optimization problem equivalently written as X ∈M ,Z f ( X ) + g ( Z ) s . t . Z = AX min introducing an artificial variable Z and a linear constraint Hestenes 1969; Powell 1969; Kovnatsky, Glashoff, B 2015 11/54

  2. Manifold ADMM Non-smooth manifold optimization problem equivalently written as X ∈M ,Z f ( X ) + g ( Z ) s . t . Z = AX min introducing an artificial variable Z and a linear constraint Apply the method of multipliers only to the constraint Z = AX X ∈M ,Z f ( X ) + g ( Z ) + ρ 2 ∥ AX − Z + U ∥ 2 min F Solve alternating w.r.t. X and Z and updating U ← U + AX − Z Hestenes 1969; Powell 1969; Kovnatsky, Glashoff, B 2015 11/54

  3. Manifold ADMM Non-smooth manifold optimization problem equivalently written as X ∈M ,Z f ( X ) + g ( Z ) s . t . Z = AX min introducing an artificial variable Z and a linear constraint Apply the method of multipliers only to the constraint Z = AX X ∈M ,Z f ( X ) + g ( Z ) + ρ 2 ∥ AX − Z + U ∥ 2 min F Solve alternating w.r.t. X and Z and updating U ← U + AX − Z Problem breaks into Smooth manifold optimization sub-problem w.r.t. X , and Non-smooth unconstrained sub-problem w.r.t. Z Hestenes 1969; Powell 1969; Kovnatsky, Glashoff, B 2015 11/54

  4. MADMM Algorithm 2 MADMM for non-smooth optimization on manifold M Initialize k ← 1 , Z ( 1 ) = AX ( 1 ) , U ( 1 ) = 0 . repeat 2 ∥ AX − Z ( k ) + U ( k ) ∥ 2 X -step: X ( k + 1 ) = argmin f ( X ) + ρ F X ∈M Z -step: Z ( k + 1 ) = argmin 2 ∥ AX ( k + 1 ) − Z + U ( k ) ∥ 2 g ( Z ) + ρ F Z Update U ( k + 1 ) = U ( k ) + AX ( k + 1 ) − Z ( k + 1 ) k ← k + 1 until convergence ; Kovnatsky, Glashoff, B 2015 12/54

  5. MADMM Algorithm 2 MADMM for non-smooth optimization on manifold M Initialize k ← 1 , Z ( 1 ) = AX ( 1 ) , U ( 1 ) = 0 . repeat 2 ∥ AX − Z ( k ) + U ( k ) ∥ 2 X -step: X ( k + 1 ) = argmin f ( X ) + ρ F X ∈M Z -step: Z ( k + 1 ) = argmin 2 ∥ AX ( k + 1 ) − Z + U ( k ) ∥ 2 g ( Z ) + ρ F Z Update U ( k + 1 ) = U ( k ) + AX ( k + 1 ) − Z ( k + 1 ) k ← k + 1 until convergence ; Solver/number of optimization iterations in X - and Z -steps Kovnatsky, Glashoff, B 2015 12/54

  6. MADMM Algorithm 2 MADMM for non-smooth optimization on manifold M Initialize k ← 1 , Z ( 1 ) = AX ( 1 ) , U ( 1 ) = 0 . repeat 2 ∥ AX − Z ( k ) + U ( k ) ∥ 2 X -step: X ( k + 1 ) = argmin f ( X ) + ρ F X ∈M Z -step: Z ( k + 1 ) = argmin 2 ∥ AX ( k + 1 ) − Z + U ( k ) ∥ 2 g ( Z ) + ρ F Z Update U ( k + 1 ) = U ( k ) + AX ( k + 1 ) − Z ( k + 1 ) k ← k + 1 until convergence ; Solver/number of optimization iterations in X - and Z -steps X -step and Z -step in some problems have a closed form Kovnatsky, Glashoff, B 2015 12/54

  7. MADMM Algorithm 2 MADMM for non-smooth optimization on manifold M Initialize k ← 1 , Z ( 1 ) = AX ( 1 ) , U ( 1 ) = 0 . repeat 2 ∥ AX − Z ( k ) + U ( k ) ∥ 2 X -step: X ( k + 1 ) = argmin f ( X ) + ρ F X ∈M Z -step: Z ( k + 1 ) = argmin 2 ∥ AX ( k + 1 ) − Z + U ( k ) ∥ 2 g ( Z ) + ρ F Z Update U ( k + 1 ) = U ( k ) + AX ( k + 1 ) − Z ( k + 1 ) k ← k + 1 until convergence ; Solver/number of optimization iterations in X - and Z -steps X -step and Z -step in some problems have a closed form Parameter ρ > 0 can be chosen fixed or adapted Kovnatsky, Glashoff, B 2015 12/54

  8. Compressed modes 13/54

  9. Laplacian eigenfunctions The first k eigenfunctions of some Laplacian are used in... Spectral clustering Dimensionality Spectral distances reduction Ng et al. 2001; Belkin, Nyogi 2001; Coifman, Lafon 2006 14/54

  10. Laplacian eigenfunctions Find the first k eigenfunctions of an n × n Laplacian matrix ∆ Φ ∈ R n × k tr ( Φ ⊺ ∆Φ ) s . t . Φ ⊺ Φ = I min 15/54

  11. Laplacian eigenfunctions Find the first k eigenfunctions of an n × n Laplacian matrix ∆ Φ ∈ R n × k tr ( Φ ⊺ ∆Φ ) s . t . Φ ⊺ Φ = I min tr ( Φ ⊺ ∆Φ ) = ∑ ij w ij ∥ φ i − φ j ∥ 2 F a.k.a. Dirichlet energy in physics 15/54

  12. Laplacian eigenfunctions Find the first k eigenfunctions of an n × n Laplacian matrix ∆ Φ ∈ R n × k tr ( Φ ⊺ ∆Φ ) s . t . Φ ⊺ Φ = I min tr ( Φ ⊺ ∆Φ ) = ∑ ij w ij ∥ φ i − φ j ∥ 2 F a.k.a. Dirichlet energy in physics Many efficient solvers with global optimality guarantees 15/54

  13. Laplacian eigenfunctions Find the first k eigenfunctions of an n × n Laplacian matrix ∆ Φ ∈ R n × k tr ( Φ ⊺ ∆Φ ) s . t . Φ ⊺ Φ = I min tr ( Φ ⊺ ∆Φ ) = ∑ ij w ij ∥ φ i − φ j ∥ 2 F a.k.a. Dirichlet energy in physics Many efficient solvers with global optimality guarantees 1D Euclidean Laplacian eigenfunctions = Fourier basis ∆ e − iωx = − ω 2 e − iωx 15/54

  14. Laplacian eigenfunctions Find the first k eigenfunctions of an n × n Laplacian matrix ∆ Φ ∈ R n × k tr ( Φ ⊺ ∆Φ ) s . t . Φ ⊺ Φ = I min tr ( Φ ⊺ ∆Φ ) = ∑ ij w ij ∥ φ i − φ j ∥ 2 F a.k.a. Dirichlet energy in physics Many efficient solvers with global optimality guarantees 1D Euclidean Laplacian eigenfunctions = Fourier basis ∆ e − iωx = − ω 2 e − iωx Globally supported! 15/54

  15. Laplacian eigenfunctions: 1D example 0.2 0.2 0 0 −0.2 −0.2 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 φ 1 φ 2 0.2 0.2 0 0 −0.2 −0.2 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 φ 3 φ 4 0.2 0.2 0 0 −0.2 −0.2 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 φ 5 φ 6 First eigenfunctions of a 1D Euclidean Laplacian 16/54

  16. Laplacian eigenfunctions: non-Euclidean example max 0 min First Laplacian eigenfunctions of a Laplacian on a triangular mesh Neumann et al. 2014 17/54

  17. Compressed modes Φ ∈ R n × k tr ( Φ ⊺ ∆Φ ) + µ ∥ Φ ∥ 1 s . t . Φ ⊺ Φ = I min Ozoli¸ nˇ s et al. 2013 18/54

  18. Compressed modes Φ ∈ R n × k tr ( Φ ⊺ ∆Φ ) + µ ∥ Φ ∥ 1 s . t . Φ ⊺ Φ = I min Dirichlet energy = smoothness Ozoli¸ nˇ s et al. 2013 18/54

  19. Compressed modes Φ ∈ R n × k tr ( Φ ⊺ ∆Φ ) + µ ∥ Φ ∥ 1 s . t . Φ ⊺ Φ = I min Dirichlet energy = smoothness L 1 -norm = sparsity Ozoli¸ nˇ s et al. 2013 18/54

  20. Compressed modes Φ ∈ R n × k tr ( Φ ⊺ ∆Φ ) + µ ∥ Φ ∥ 1 s . t . Φ ⊺ Φ = I min Dirichlet energy = smoothness L 1 -norm = sparsity Smoothness + sparsity = localization Ozoli¸ nˇ s et al. 2013 18/54

  21. Compressed modes: 1D example 6 5 4 2 0 0 −5 −2 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 φ 1 φ 2 5 5 0 0 −5 −5 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 φ 3 φ 4 5 5 0 0 −5 −5 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 φ 5 φ 6 First compressed modes of a 1D Euclidean Laplacian Ozoli¸ nˇ s et al. 2013 19/54

  22. Compressed modes: non-Euclidean example max 0 min First compressed modes Neumann et al. 2014 20/54

  23. Compressed modes: non-Euclidean example max 0 min First Laplacian eigenfunctions Neumann et al. 2014 20/54

  24. Wannier functions Maximally-localized Wannier functions in Si and GaAs crystals Wannier 1937; Mostofi 2008 21/54

  25. Splitting method for orthogonality constraints (SOC) Φ ∈ R n × k tr ( Φ ⊺ ∆Φ ) + µ ∥ Φ ∥ 1 Φ ⊺ Φ = I min s . t . Lai, Osher 2014 22/54

  26. Splitting method for orthogonality constraints (SOC) Φ ,P,Q ∈ R n × k tr ( Φ ⊺ ∆Φ ) + µ ∥ Q ∥ 1 P = Φ , Q = Φ , P ⊺ P = I min s . t . Lai, Osher 2014 22/54

  27. Splitting method for orthogonality constraints (SOC) Φ ,P,Q ∈ R n × k tr ( Φ ⊺ ∆Φ ) + µ ∥ Q ∥ 1 P = Φ , Q = Φ , P ⊺ P = I min s . t . Algorithm 3 SOC method for computing compressed modes Initialize k ← 1 , Φ ( 1 ) , P ( 1 ) = Q ( 1 ) = Φ ( 1 ) , U ( 1 ) = V ( 1 ) = 0 repeat Φ ( k + 1 ) = argmin F + ρ ′ tr ( Φ ⊺ ∆Φ )+ ρ 2 ∥ Φ − Q ( k ) + U ( k ) ∥ 2 2 ∥ Φ − P ( k ) + V ( k ) ∥ 2 F Φ Q ( k + 1 ) = argmin 2 ∥ Φ ( k + 1 ) − Q + U ( k ) ∥ 2 µ ∥ Q ∥ 1 + ρ F Q P ( k + 1 ) = argmin 2 ∥ Φ ( k + 1 ) − P + V ( k ) ∥ 2 ρ ′ P ⊺ P = I s . t . F P U ( k + 1 ) = U ( k ) + Φ ( k + 1 ) − Q ( k + 1 ) V ( k + 1 ) = V ( k ) + Φ ( k + 1 ) − P ( k + 1 ) k ← k + 1 until convergence ; Lai, Osher 2014 22/54

  28. Compressed modes as manifold optimization Φ ∈ S ( n,k ) tr ( Φ ⊺ ∆Φ ) + µ ∥ Φ ∥ 1 min Stiefel manifold S ( n,k ) = { X ∈ R n × k ∶ X ⊺ X = I } Kovnatsky, Glashoff, B 2015 23/54

  29. Compressed modes as manifold optimization Φ ∈ S ( n,k ) ,Z tr ( Φ ⊺ ∆Φ ) + µ ∥ Z ∥ 1 + ρ 2 ∥ Φ − Z + U ∥ 2 min F Stiefel manifold S ( n,k ) = { X ∈ R n × k ∶ X ⊺ X = I } Kovnatsky, Glashoff, B 2015 23/54

  30. Compressed modes as manifold optimization Φ ∈ S ( n,k ) ,Z tr ( Φ ⊺ ∆Φ ) + µ ∥ Z ∥ 1 + ρ 2 ∥ Φ − Z + U ∥ 2 min F Stiefel manifold S ( n,k ) = { X ∈ R n × k ∶ X ⊺ X = I } Sub-problem w.r.t. Φ : smooth manifold-constrained minimization of a quadratic function Φ ∈ S ( n,k ) tr ( Φ ⊺ ∆Φ ) + ρ 2 ∥ Φ − Z + U ∥ 2 min F Kovnatsky, Glashoff, B 2015 23/54

  31. Compressed modes as manifold optimization Φ ∈ S ( n,k ) ,Z tr ( Φ ⊺ ∆Φ ) + µ ∥ Z ∥ 1 + ρ 2 ∥ Φ − Z + U ∥ 2 min F Stiefel manifold S ( n,k ) = { X ∈ R n × k ∶ X ⊺ X = I } Sub-problem w.r.t. Φ : smooth manifold-constrained minimization of a quadratic function Φ ∈ S ( n,k ) tr ( Φ ⊺ ∆Φ ) + ρ 2 ∥ Φ − Z + U ∥ 2 min F Sub-problem w.r.t. Z : sparse coding (Lasso) problem ∥ Z ∥ 1 + ρ 2 ∥ Φ + U − Z ∥ 2 min F Z Kovnatsky, Glashoff, B 2015; Chen et al. 1995; Tibshirani 1996 23/54

  32. Compressed modes by MADMM Algorithm 4 MADMM for computing compressed modes Input n × n Laplacian matrix ∆ , parameter µ > 0 Output k first compressed modes of ∆ Initialize k ← 1 , Φ ( 1 ) ← some orthonormal matrix, Z ( 1 ) = Φ ( 1 ) , U ( 1 ) = 0 repeat 2 ∥ Φ − Z ( k ) + U ( k ) ∥ 2 Φ ( k + 1 ) = argmin tr ( Φ ⊺ ∆Φ ) + ρ F Φ ∈ S ( n,k ) ρ ( Φ ( k + 1 ) + U ( k ) ) Z ( k + 1 ) = Shrink µ Update U ( k + 1 ) = U ( k ) + Φ ( k + 1 ) − Z ( k + 1 ) k ← k + 1 until convergence ; where Shrink α ( x ) = sign ( x ) max { 0 , ∣ x ∣ − α } is the shrinkage operator Kovnatsky, Glashoff, B 2015 24/54

  33. Convergence Convergence of MADMM with different random initializations (compressed modes problem of size n = 500 ,k = 10 ) 10 3 10 2 Cost 10 1 10 − 1 10 0 10 1 10 2 Time (sec) Kovnatsky, Glashoff, B 2015 25/54

  34. Convergence Convergence of MADMM with different X -step solvers (compressed modes problem of size n = 500 ,k = 10 ) Trust regions 10 3 2 Conjugate gradients 3 3 2 10 2 Cost 5 5 10 1 10 − 1 10 0 10 1 10 2 Time (sec) Kovnatsky, Glashoff, B 2015; Manopt: Boumal et al. 2014 26/54

  35. Convergence Example of convergence of different methods (compressed modes problem of size n = 8 × 10 3 ,k = 10 ) 10 3 Lai & Osher Neumann et al. MADMM 10 2 Cost 10 1 10 0 0 1 , 000 2 , 000 3 , 000 4 , 000 5 , 000 Time (sec) Kovnatsky, Glashoff, B 2015; Lai, Osher 2014; Neumann et al. 2014 27/54

  36. Scalability Complexity of different methods (compressed modes problem of size n,k = 10 ) 10 2 Lai & Osher Neumann MADMM 10 1 Time/iter (sec) 10 0 10 − 1 1 , 000 2 , 000 3 , 000 4 , 000 5 , 000 Problem size n Kovnatsky, Glashoff, B 2015; Lai, Osher 2014; Neumann et al. 2014 28/54

  37. Functional correspondence 29/54

  38. Applications of shape correspondence Texture mapping Pose transfer B 2 , Kimmel 2007; Sumner et al. 2004 30/54

  39. Shape correspondence t q s S Q Point-wise map t ∶ S → Q 31/54

  40. Shape correspondence t q s q ′ s ′ S Q Minimum-distortion point-wise map t ∶ S → Q B 2 , Kimmel 2006 31/54

  41. Shape correspondence g f F(S) F(Q) linear T Functional map T ∶F( S ) → F ( Q ) Ovsjanikov et al. 2012 31/54

  42. Functional correspondence f ↓ T ↓ g Ovsjanikov et al. 2012 32/54

  43. Functional correspondence ≈ a 1 + a 2 + ⋯ + a k f φ 1 φ 2 φ k ↓ T ↓ g ≈ b 1 + b 2 + ⋯ + b k ψ 1 ψ 2 ψ k Ovsjanikov et al. 2012 32/54

  44. Functional correspondence ≈ a 1 + a 2 + ⋯ + a k f φ 1 φ 2 φ k ↓ ↓ T C ↓ ↓ g ≈ b 1 + b 2 + ⋯ + b k ψ 1 ψ 2 ψ k Representation in truncated Laplacain eigenbasis, T ≈ Ψ C Φ ⊺ Ovsjanikov et al. 2012 32/54

  45. Functional correspondence ≈ a 1 + a 2 + ⋯ + a k f φ 1 φ 2 φ k ↓ ↓ T C ↓ ↓ g ≈ b 1 + b 2 + ⋯ + b k ψ 1 ψ 2 ψ k Representation in truncated Laplacain eigenbasis, T ≈ Ψ C Φ ⊺ If T is area-preserving, C ⊺ C = I Ovsjanikov et al. 2012 32/54

  46. Functional correspondence ≈ a 1 + a 2 + ⋯ + a k f φ 1 φ 2 φ k ↓ ↓ T C ↓ ↓ g ≈ b 1 + b 2 + ⋯ + b k ψ 1 ψ 2 ψ k Representation in truncated Laplacain eigenbasis, T ≈ Ψ C Φ ⊺ If T is area-preserving, C ⊺ C = I Φ ⊺ = Ψ X ( Φ Y ) ⊺ (rotation of bases) Represent C = XY ⊺ , then T ≈ ˆ Ψˆ Ovsjanikov et al. 2012 32/54

  47. Functional correspondence ≈ a 1 + a 2 + ⋯ + a k f φ 1 φ 2 φ k ↓ ↓ T C ↓ ↓ g ≈ b 1 + b 2 + ⋯ + b k ψ 1 ψ 2 ψ k Representation in truncated Laplacain eigenbasis, T ≈ Ψ C Φ ⊺ If T is area-preserving, C ⊺ C = I Φ ⊺ = Ψ X ( Φ Y ) ⊺ (rotation of bases) Represent C = XY ⊺ , then T ≈ ˆ Ψˆ Given known corresponding functions F = ( f 1 , ⋯ ,f q ) and G = ( g 1 , ⋯ ,g q ) , find C by solving linear system C Φ ⊺ F = Ψ ⊺ G Ovsjanikov et al. 2012 32/54

  48. Functional correspondence ≈ a 1 + a 2 + ⋯ + a k f φ 1 φ 2 φ k ↓ ↓ T C ↓ ↓ g ≈ b 1 + b 2 + ⋯ + b k ψ 1 ψ 2 ψ k Representation in truncated Laplacain eigenbasis, T ≈ Ψ C Φ ⊺ If T is area-preserving, C ⊺ C = I Φ ⊺ = Ψ X ( Φ Y ) ⊺ (rotation of bases) Represent C = XY ⊺ , then T ≈ ˆ Ψˆ Given known corresponding Fourier coefficients A = Φ ⊺ F and B = Ψ ⊺ G , find C by solving linear system CA = B Ovsjanikov et al. 2012 32/54

  49. Functional correspondence in shape collection A ij C ij ≈ B ij S j S L ⋱ S i S 1 S 2 Kovnatsky, B 2 , Glashoff, Kimmel 2013; Kovnatsky, Glashoff, B 2015 33/54

  50. Functional correspondence in shape collection A ij X i ≈ B ij X j X j S j X i X L S L ⋱ S i X 1 X 2 S 1 S 2 Kovnatsky, B 2 , Glashoff, Kimmel 2013; Kovnatsky, Glashoff, B 2015 33/54

  51. Functional correspondence as manifold optimization L ∥ A ij X i − B ij X j ∥ 2 , 1 + µ tr ( X ⊺ i Λ i X i ) ( X 1 , ⋯ ,X L )∈ S L ( k,k ) ∑ ∑ min i ≠ j i = 1 where A ij ,B ij are Fourier coefficients of given corresponding functions on shapes S i , S j , and Λ i are the first k eigenvalues of Laplacian ∆ S i Eynard, Kovnatsky, B 2 , Glashoff 2012; Kovnatsky, Glashoff, B 2015 34/54

  52. Functional correspondence as manifold optimization L ∥ A ij X i − B ij X j ∥ 2 , 1 + µ tr ( X ⊺ i Λ i X i ) ( X 1 , ⋯ ,X L )∈ S L ( k,k ) ∑ ∑ min i ≠ j i = 1 where A ij ,B ij are Fourier coefficients of given corresponding functions on shapes S i , S j , and Λ i are the first k eigenvalues of Laplacian ∆ S i Joint diagonalization of Laplacians: find new bases ˆ Φ i = Φ i X i that approximately diagonalize ∆ i and are coupled Optimization on product of Stiefel manifolds S L ( k,k ) Eynard, Kovnatsky, B 2 , Glashoff 2012; Kovnatsky, Glashoff, B 2015 34/54

  53. Functional correspondence as manifold optimization L ∥ A ij X i − B ij X j ∥ 2 , 1 + µ tr ( X ⊺ i Λ i X i ) ( X 1 , ⋯ ,X L )∈ S L ( k,k ) ∑ ∑ min i ≠ j i = 1 where A ij ,B ij are Fourier coefficients of given corresponding functions on shapes S i , S j , and Λ i are the first k eigenvalues of Laplacian ∆ S i Joint diagonalization of Laplacians: find new bases ˆ Φ i = Φ i X i that approximately diagonalize ∆ i and are coupled Optimization on product of Stiefel manifolds S L ( k,k ) L 2 , 1 -norm allows to cope with outliers in correspondence data Eynard, Kovnatsky, B 2 , Glashoff 2012; Kovnatsky, Glashoff, B 2015 34/54

  54. Functional correspondence as manifold optimization L ∥ A ij X i − B ij X j ∥ 2 , 1 + µ tr ( X ⊺ i Λ i X i ) ( X 1 , ⋯ ,X L )∈ S L ( k,k ) ∑ ∑ min i ≠ j i = 1 where A ij ,B ij are Fourier coefficients of given corresponding functions on shapes S i , S j , and Λ i are the first k eigenvalues of Laplacian ∆ S i Joint diagonalization of Laplacians: find new bases ˆ Φ i = Φ i X i that approximately diagonalize ∆ i and are coupled Optimization on product of Stiefel manifolds S L ( k,k ) L 2 , 1 -norm allows to cope with outliers in correspondence data X -step: manifold-constrained minimization of a quadratic function Eynard, Kovnatsky, B 2 , Glashoff 2012; Kovnatsky, Glashoff, B 2015 34/54

  55. Functional correspondence as manifold optimization L ∥ A ij X i − B ij X j ∥ 2 , 1 + µ tr ( X ⊺ i Λ i X i ) ( X 1 , ⋯ ,X L )∈ S L ( k,k ) ∑ ∑ min i ≠ j i = 1 where A ij ,B ij are Fourier coefficients of given corresponding functions on shapes S i , S j , and Λ i are the first k eigenvalues of Laplacian ∆ S i Joint diagonalization of Laplacians: find new bases ˆ Φ i = Φ i X i that approximately diagonalize ∆ i and are coupled Optimization on product of Stiefel manifolds S L ( k,k ) L 2 , 1 -norm allows to cope with outliers in correspondence data X -step: manifold-constrained minimization of a quadratic function Z -step: one iteration of shrinkage Eynard, Kovnatsky, B 2 , Glashoff 2012; Kovnatsky, Glashoff, B 2015 34/54

  56. Correspondence data Example of correspondence data (10% of outliers shown in red) Kovnatsky, Glashoff, B 2015 35/54

  57. Correspondence quality Least squares Robust (MADMM) Kovnatsky, Glashoff, B 2015 36/54

  58. Correspondence quality Correspondence quality evaluated using Princeton protocol 1 0 . 8 % of correspondence 0 . 6 0 . 4 LS 0 . 2 MADMM 0 0 5 ⋅ 10 − 2 0 . 1 0 . 15 0 . 2 0 . 25 % geodesic diameter Kovnatsky, Glashoff, B 2015; data: B 2 , Kimmel 2008, benchmark: Kim et al. 2011 37/54

  59. Convergence Convergence of different methods Smoothing 10 0 . 4 MADMM Cost 10 -8 10 0 . 2 10 -6 10 -4 0 2 4 6 8 10 Time (sec) Kovnatsky, Glashoff, B 2015 38/54

  60. Multimodal spectral clustering Uncoupled No outliers 100% Kovnatsky, Glashoff, B 2015; Eynard, Kovnatsky, B 2 , Glashoff 2012 39/54

  61. Multimodal spectral clustering Uncoupled Coupled ( L 2 ) No outliers No outliers 100% 53% Kovnatsky, Glashoff, B 2015; Eynard, Kovnatsky, B 2 , Glashoff 2012 39/54

  62. Multimodal spectral clustering Coupled ( L 2 ) Uncoupled Coupled ( L 2 ) 10% outliers No outliers No outliers 100% 53% 72% Kovnatsky, Glashoff, B 2015; Eynard, Kovnatsky, B 2 , Glashoff 2012 39/54

  63. Multimodal spectral clustering Coupled ( L 2 ) Coupled ( L 2 , 1 ) Uncoupled Coupled ( L 2 ) 10% outliers 10% outliers No outliers No outliers 100% 53% 72% 82% Kovnatsky, Glashoff, B 2015; Eynard, Kovnatsky, B 2 , Glashoff 2012 39/54

  64. Multidimensional scaling 40/54

  65. Multidimensional scaling 3 2 9 10 7 3 14 1 2 1 2 14 2 7 9 D = X = 9 1 2 2 2 10 2 7 2 13 7 1 9 2 13 MDS problem: given an n × n (squared) distance matrix D , find a k -dimensional configuration of points X ∈ R n × k such that ∥ x i − x j ∥ 2 2 ≈ d ij Cayton, Dasgupta 2006 41/54

  66. Similarity vs Distance Equivalence between distances and similarities dist ( B ) = ( b ii + b jj − 2 b ij ) PSD EDM B = − 1 2 HDH (Squared) distances Similarities n 11 ⊺ is the double-centering matrix where H = I − 1 Sch¨ onberg 1938; Dattoro 2005; Cayton, Dasgupta 2006 42/54

  67. Similarity vs Distance Equivalence between distances and similarities PSD EDM B ∗ = U Λ + U ⊺ B = − 1 2 HDH (Squared) distances Similarities n 11 ⊺ is the double-centering matrix where H = I − 1 Sch¨ onberg 1938; Dattoro 2005; Cayton, Dasgupta 2006 42/54

  68. Classical MDS Algorithm 5 Classical MDS Input squared distance matrix D Compute similarity by double centering: B = − 1 2 HDH Perform eigendecomposition B = U Λ U ⊺ and take the largest k positive eigenvalues Λ k and corresponding eigenvectors U k Output X = U k Λ 1 / 2 k Young, Householder 1938 43/54

  69. Classical MDS Algorithm 5 Classical MDS Input squared distance matrix D Compute similarity by double centering: B = − 1 2 HDH Perform eigendecomposition B = U Λ U ⊺ and take the largest k positive eigenvalues Λ k and corresponding eigenvectors U k Output X = U k Λ 1 / 2 k Classical MDS as optimization problem: minimize the strain X ∈ R n × k ∥ HDH − XX ⊺ ∥ 2 min F Young, Householder 1938 43/54

  70. Sensitivity to outliers Error dispersion by double-centering ǫ ǫ / n ǫ B = − 1 2 HDH ǫ / n 2 (Squared) distance matrix Similarity matrix Cayton, Dasgupta 2006 44/54

Recommend


More recommend