Its Applications to Rotation Synchronization Yunpeng Shi Joint work - PowerPoint PPT Presentation

Message Passing Least Squares Framework and Its Applications to Rotation Synchronization Yunpeng Shi Joint work with Prof. Gilad Lerman School of Mathematics University of Minnesota

Rotation Synchronization 3 2 1 5 4 Given a graph 𝐻 𝑜 , 𝐹 𝑜 : = 1,2,3, … , 𝑜 , 𝐹 is the set of edges

Rotation Synchronization ∗ = ? 𝑆 3 ∗ = ? 𝑆 2 ∗ = ? 𝑆 1 ∗ = ? ∗ = ? 𝑆 4 𝑆 5 ∗ ∈ 𝑇𝑃(3) Each node 𝑗 ∈ [𝑜] is assigned an unknown ground truth rotation 𝑆 𝑗

Rotation Synchronization 𝑆 23 ∗ = ? 𝑆 12 𝑆 3 ∗ = ? 𝑆 2 ∗ = ? 𝑆 1 𝑆 25 𝑆 35 𝑆 14 ∗ = ? ∗ = ? 𝑆 4 𝑆 5 𝑆 45 • Each edge 𝑗𝑘 ∈ 𝐹 is given a possibly noisy and corrupted relative rotation 𝑆 𝑗𝑘 ∗ = 𝑆 𝑗 ∗ 𝑆 𝑘 ∗−1 • The uncorrupted relative rotation for 𝑗𝑘 ∈ 𝐹 is 𝑆 𝑗𝑘 ∗ • Rotation Synchronization : Estimate 𝑆 𝑗 𝑗∈[𝑜] from 𝑆 𝑗𝑘 𝑗𝑘∈𝐹 ∗ 𝑆 𝑗∈[𝑜] for any rotation 𝑆 is also a solution • 𝑆 𝑗

Applications Camera orientation estimation in 3D reconstruction tasks: Structure from motion (SfM) Simultaneous localization and mapping (SLAM) Demonstration by Carl Olsson Demonstration by Raúl Mur-Artal

Adversarial Corruption Model ∗ : = 𝑆 𝑗 ∗ 𝑆 𝑘 ∗−1 , 𝑆 𝑗𝑘 𝑗𝑘 ∈ 𝐹 𝑕 (good edges) 𝑆 𝑗𝑘 = ቐ ෨ 𝑆 𝑗𝑘 , 𝑗𝑘 ∈ 𝐹 𝑐 (bad edges)

Adversarial Corruption Model ∗ : = 𝑆 𝑗 ∗ 𝑆 𝑘 ∗−1 , 𝑆 𝑗𝑘 𝑗𝑘 ∈ 𝐹 𝑕 (good edges) 𝑆 𝑗𝑘 = ቐ ෨ 𝑆 𝑗𝑘 , 𝑗𝑘 ∈ 𝐹 𝑐 (bad edges) ∗ ≔ 𝑒(𝑆 𝑗𝑘 , 𝑆 𝑗𝑘 ∗ ) Corruption Level 𝑡 𝑗𝑘 Commonly, 𝑒 is the geodesic distance on 𝑇𝑃(3)

Least Squares Solvers −1 , 𝑆 𝑗𝑘 ) 𝑒 2 (𝑆 𝑗 𝑆 𝑆 𝑗 𝑗∈[𝑜] ⊂𝑇𝑃(3) ෍ minimize 𝑘 𝑗𝑘∈𝐹 The most common approximate solution is the Lie algebraic averaging

Robust Solvers: 𝑚 𝑞 minimization ( 0 < 𝑞 ≤ 1 ) −1 , 𝑆 𝑗𝑘 ) 𝑒 𝑞 (𝑆 𝑗 𝑆 𝑆 𝑗 𝑗∈[𝑜] ⊂𝑇𝑃(3) ෍ minimize 𝑘 𝑗𝑘∈𝐹

−1 , 𝑆 𝑗𝑘 ) over 𝑆 𝑗 𝑗∈[𝑜] in SO(3)? How to minimize σ 𝑗𝑘∈𝐹 𝑒 𝑞 (𝑆 𝑗 𝑆 𝑘

−1 , 𝑆 𝑗𝑘 ) over 𝑆 𝑗 𝑗∈[𝑜] in SO(3)? How to minimize σ 𝑗𝑘∈𝐹 𝑒 𝑞 (𝑆 𝑗 𝑆 𝑘 Iteratively Reweighted Least Squares (IRLS): −1 , 𝑆 𝑗𝑘 ) σ 𝑗𝑘∈𝐹 𝑥 𝑗𝑘,𝑢 𝑒 2 (𝑆 𝑗 𝑆 𝑘 𝑆 𝑗,𝑢 𝑗∈[𝑜] = argmin 𝑆 𝑗 𝑗∈[𝑜] ⊂𝑇𝑃(3) −1 , 𝑆 𝑗𝑘 ) 𝑠 𝑗𝑘,𝑢 = 𝑒(𝑆 𝑗,𝑢 𝑆 𝑘,𝑢 𝑞−2 𝑥 𝑗𝑘,𝑢+1 = 𝑠 𝑗𝑘,𝑢

−1 , 𝑆 𝑗𝑘 ) over 𝑆 𝑗 𝑗∈[𝑜] in SO(3)? How to minimize σ 𝑗𝑘∈𝐹 𝑒 𝑞 (𝑆 𝑗 𝑆 𝑘 Iteratively Reweighted Least Squares (IRLS): −1 , 𝑆 𝑗𝑘 ) σ 𝑗𝑘∈𝐹 𝑥 𝑗𝑘,𝑢 𝑒 2 (𝑆 𝑗 𝑆 𝑘 𝑆 𝑗,𝑢 𝑗∈[𝑜] = argmin 𝑆 𝑗 𝑗∈[𝑜] ⊂𝑇𝑃(3) −1 , 𝑆 𝑗𝑘 ) 𝑠 𝑗𝑘,𝑢 = 𝑒(𝑆 𝑗,𝑢 𝑆 𝑘,𝑢 𝑞−2 𝑥 𝑗𝑘,𝑢+1 = 𝑠 𝑗𝑘,𝑢 2−𝑞 ∗ ≔ 𝑒(𝑆 𝑗𝑘 , 𝑆 𝑗𝑘 ∗ ) and 𝑥 𝑗𝑘,𝑢+1 ≈ 1 Ideally, 𝑠 𝑗𝑘,𝑢 ≈ 𝑡 𝑗𝑘 concentrates on 𝐹 𝑕 ∗ 𝑡 𝑗𝑘

Issue 1: Over-Aggressive Reweighting ∗ and thus 𝑠 𝑗𝑘,𝑢 ≉ 𝑡 𝑗𝑘 ∗ • Under severe corruption, 𝑆 𝑗,𝑢 ≉ 𝑆 𝑗 • In certain cases, 𝑠 𝑗𝑘,𝑢 ≈ 0 for 𝑗𝑘 ∈ 𝐹 𝑐 , and thus 2−𝑞 1 𝑥 𝑗𝑘,𝑢+1 = can be extremely high for 𝑗𝑘 ∈ 𝐹 𝑐 𝑠 𝑗𝑘,𝑢

Issue 2: Poor Least Squares Solution −1 , 𝑆 𝑗𝑘 ) 𝑥 𝑗𝑘,𝑢 𝑒 2 (𝑆 𝑗 𝑆 𝑘 𝑆 𝑗,𝑢 𝑗∈[𝑜] = argmin ෍ 𝑆 𝑗 𝑗∈[𝑜] ⊂𝑇𝑃(3) 𝑗𝑘∈𝐹

Issue 2: Poor Least Squares Solution −1 , 𝑆 𝑗𝑘 ) 𝑥 𝑗𝑘,𝑢 𝑒 2 (𝑆 𝑗 𝑆 𝑘 𝑆 𝑗,𝑢 𝑗∈[𝑜] = argmin ෍ 𝑆 𝑗 𝑗∈[𝑜] ⊂𝑇𝑃(3) 𝑗𝑘∈𝐹 Lie algebraic representation 𝜕 𝑗 𝑆 𝑗 Riemannian manifold of SO(3) 𝑆 𝑗,𝑢−1

Issue 2: Poor Least Squares Solution −1 , 𝑆 𝑗𝑘 ) 𝑥 𝑗𝑘,𝑢 𝑒 2 (𝑆 𝑗 𝑆 𝑘 𝑆 𝑗,𝑢 𝑗∈[𝑜] = argmin ෍ 𝑆 𝑗 𝑗∈[𝑜] ⊂𝑇𝑃(3) 𝑗𝑘∈𝐹 𝜕 𝑗 The Lie Algebraic Averaging uses the approximation −1 , 𝑆 𝑗𝑘 ≈ ԡ 𝜕 𝑗 − 𝑒 𝑆 𝑗 𝑆 𝜕 𝑘 − 𝜕 𝑗𝑘 ฮ 𝑆 𝑗 𝑘 2 −1 𝑆 𝑗 ) where 𝜕 𝑗 = log( 𝑆 𝑗,𝑢−1 𝑆 𝑗,𝑢−1 −1 𝑆 𝑗𝑘 𝑆 and 𝜕 𝑗𝑘 = log( 𝑆 𝑗,𝑢−1 𝑘,𝑢−1 ) ∗ and 𝑆 𝑗𝑘 ≈ 𝑆 𝑗 ∗ 𝑆 𝑘 ∗−1 The approximation is valid only when 𝑆 𝑗 ≈ 𝑆 𝑗

∗ without knowing 𝑆 𝑗 ∗ and 𝑆 𝑘 ∗ ? How to accurately estimate 𝑡 𝑗𝑘

Cycle-Edge Message Passing (CEMP) • Goal: Estimate corruption level ∗ ≔ 𝑒(𝑆 𝑗𝑘 , 𝑆 𝑗𝑘 ∗ ) 𝑡 𝑗𝑘 from 3-cycle inconsistency measure 𝑒 𝑗𝑘𝑙 ≔ 𝑒(𝑆 𝑗𝑘 𝑆 𝑘𝑙 𝑆 𝑙𝑗 , 𝐽)

Cycle-Edge Message Passing (CEMP) • Goal: Estimate corruption level ∗ ≔ 𝑒(𝑆 𝑗𝑘 , 𝑆 𝑗𝑘 ∗ ) 𝑡 𝑗𝑘 from 3-cycle inconsistency measure 𝑒 𝑗𝑘𝑙 ≔ 𝑒(𝑆 𝑗𝑘 𝑆 𝑘𝑙 𝑆 𝑙𝑗 , 𝐽) • For each 𝑗𝑘 ∈ 𝐹, sample 50 3-cycles 𝑗𝑘𝑙 and for each cycle compute 𝑒 𝑗𝑘𝑙

Cycle-Edge Message Passing (CEMP) • Goal: Estimate corruption level ∗ ≔ 𝑒(𝑆 𝑗𝑘 , 𝑆 𝑗𝑘 ∗ ) 𝑡 𝑗𝑘 from 3-cycle inconsistency measure 𝑒 𝑗𝑘𝑙 ≔ 𝑒(𝑆 𝑗𝑘 𝑆 𝑘𝑙 𝑆 𝑙𝑗 , 𝐽) • For each 𝑗𝑘 ∈ 𝐹, sample 50 3-cycles 𝑗𝑘𝑙 and for each cycle compute 𝑒 𝑗𝑘𝑙 • For fixed 𝑗𝑘 and good 3-cycles w.r.t. 𝑗𝑘 (That is, 𝑗𝑙, 𝑘𝑙 ∈ 𝐹 𝑕 ) ∗ 𝑆 𝑙𝑗 ∗ , 𝐽) = 𝑒(𝑆 𝑗𝑘 𝑆 𝑘𝑙 ∗ 𝑆 𝑙𝑗 ∗ 𝑆 𝑗𝑘 ∗ , 𝑆 𝑗𝑘 ∗ ) = 𝑒(𝑆 𝑗𝑘 , 𝑆 𝑗𝑘 ∗ ) = 𝑡 𝑗𝑘 ∗ ︸ 𝑒 𝑗𝑘𝑙 = 𝑒(𝑆 𝑗𝑘 𝑆 𝑘𝑙 = 𝐽 by cycle consistency

𝑒 𝑗𝑘𝑙 1 𝑒 𝑗𝑘𝑙 2 𝑒 𝑗𝑘𝑙 49 𝑒 𝑗𝑘𝑙 50

𝑒 𝑗𝑘𝑙 1 𝑢 𝑞 𝑗𝑘𝑙 1 ∗ 𝑒 𝑗𝑘𝑙 2 = 𝑡 𝑗𝑘 𝑢 𝑞 𝑗𝑘𝑙 2 𝑗𝑘 𝑢 𝑞 𝑗𝑘𝑙 49 ∗ 𝑒 𝑗𝑘𝑙 49 = 𝑡 𝑗𝑘 𝑢 𝑞 𝑗𝑘𝑙 50 𝑒 𝑗𝑘𝑙 50

𝑒 𝑗𝑘𝑙 1 𝑢 𝑞 𝑗𝑘𝑙 1 𝑒 𝑗𝑘𝑙 2 ∗ = 𝑡 𝑗𝑘 𝑢 𝑞 𝑗𝑘𝑙 2 ∗ ≈ 𝑡 𝑗𝑘,𝑢+1 : = 𝑢 𝑒 𝑗𝑘𝑙 1 𝑗𝑘 𝑢 σ 𝑙 𝑞 𝑗𝑘𝑙 𝑡 𝑗𝑘 𝑎 𝑗𝑘 𝑢 𝑞 𝑗𝑘𝑙 49 𝑒 𝑗𝑘𝑙 49 𝑢 𝑒 𝑗𝑘𝑙 𝑢 = σ 𝑙 𝑞 𝑗𝑘𝑙 𝑎 𝑗𝑘 ∗ 𝑢 = 𝑡 𝑗𝑘 𝑞 𝑗𝑘𝑙 50 𝑒 𝑗𝑘𝑙 50

𝑡 𝑗𝑙,𝑢 𝑡 𝑘𝑙,𝑢

Prob. that 𝑗𝑙 ∈ 𝐹 𝑕 given 𝑡 𝑗𝑙,𝑢 𝑓 −𝛾 𝑢 ∙ 𝑞 𝑗𝑙,𝑢 𝑡 𝑗𝑙,𝑢 𝑓 −𝛾 𝑢 ∙ 𝑞 𝑘𝑙,𝑢 𝑡 𝑘𝑙,𝑢 Prob. that j𝑙 ∈ 𝐹 𝑕 given 𝑡 𝑘𝑙,𝑢

Prob. that 𝑗𝑙 ∈ 𝐹 𝑕 given 𝑡 𝑗𝑙,𝑢 𝑓 −𝛾 𝑢 ∙ 𝑞 𝑗𝑙,𝑢 𝑡 𝑗𝑙,𝑢 Prob. that 𝑗𝑘𝑙 is good given {𝑡 𝑏𝑐,𝑢 : 𝑏𝑐 ∈ 𝐹} 𝑗𝑘𝑙 𝑢 = 𝑓 −𝛾 𝑢 (𝑡 𝑗𝑙,𝑢 +𝑡 𝑘𝑙,𝑢 ) 𝑞 𝑗𝑘𝑙 𝑓 −𝛾 𝑢 ∙ 𝑞 𝑘𝑙,𝑢 𝑡 𝑘𝑙,𝑢 Prob. that j𝑙 ∈ 𝐹 𝑕 given 𝑡 𝑘𝑙,𝑢

Cycle-Edge Message Passing (CEMP) ∗ ): The conditional probability that 𝑗𝑘𝑙 is good ( 𝑒 𝑗𝑘𝑙 = 𝑡 𝑗𝑘 = 𝑓 −𝛾 𝑢 (𝑡 𝑗𝑙,𝑢 +𝑡 𝑘𝑙,𝑢 ) = 𝑄𝑠(𝑒 𝑗𝑘𝑙 = 𝑡 𝑗𝑘 ∗ |{𝑡 𝑏𝑐,𝑢 : 𝑏𝑐 ∈ 𝐹}) 𝑢 𝑞 𝑗𝑘𝑙 The estimate of the corruption level: 𝑢 𝑒 𝑗𝑘𝑙 = 𝔽(𝑡 𝑗𝑘 ∗ |{𝑡 𝑏𝑐,𝑢 : 𝑏𝑐 ∈ 𝐹}) 1 𝑢 σ 𝑙 𝑞 𝑗𝑘𝑙 𝑡 𝑗𝑘,𝑢+1 : = 𝑎 𝑗𝑘

Theory If f 1 • the maximal ratio of corrupted cycles per edge < 5 • 𝛾 𝑢 increases exponentially with a sufficiently small rate, the hen • for all 𝑗𝑘 ∈ 𝐹, 𝑡 𝑗𝑘,𝑢 computed by CEMP linearly and uniformly ∗ . converges to 𝑡 𝑗𝑘

Message Passing Least Squares (MPLS) Init Initialization Run CEMP for 𝑈 iterations Build a weighted graph with edge weights 𝑡 𝑗𝑘,𝑈 𝑗𝑘∈𝐹 Find the minimal spanning tree using Prim’s Algorithm ize 𝑆 𝑗,0 by fixing 𝑆 1,0 = 𝐽 and 𝑆 𝑗,0 = 𝑆 𝑗𝑘 𝑆 𝑘,0 Initi nitializ ize weights 𝑥 𝑗𝑘,0 = 𝐺(𝑡 𝑗𝑘,𝑈 ) Initi nitializ For 𝑚 𝑞 minimization, 𝐺 𝑦 = 𝑦 𝑞−2