Using m -Best Solutions S. Hamid Rezatofighi Anton Milan Zhen - PowerPoint PPT Presentation

Joint Probabilistic Matching Using m -Best Solutions S. Hamid Rezatofighi Anton Milan Zhen Zhang Qinfeng Shi Antony Dick Ian Reid 1

Introduction  One-to-One Graph Matching in Computer Vision • Action Recognition • Feature Point Matching • Multi-Target Tracking ⋮ ⋮ • Person Re-Identification 2

Introduction  Most existing works focus on • Feature and/or metric learning [Zhao et al ., CVPR 2014 , Liu et al ., ECCV 2010] • Developing better solvers [Cho et al ., ECCV 2010 , Zhou & De la Torre, CVPR 2013]  The optimal solution does not necessarily yield the correct matching assignment  To improving the matching results, we propose • to consider more feasible solutions • a principle approach to combine the solutions 3

One-to-One Graph Matching  Formulating it as a constrained binary program ⋮ ⋮ 4

One-to-One Graph Matching  Formulating it as a constrained binary program 0 𝑦 1 1 𝑦 1 ⋮ ⋮ ⋮ 𝑂 𝑦 𝑁 5

One-to-One Graph Matching  Formulating it as a constrained binary program 0 𝑦 1 𝑘 = {0,1} 1 𝑦 𝑗 𝑦 1 𝑂 𝑈 ⊆ 𝔺 𝑁×(𝑂+1) 𝑘 , … , 𝑦 𝑁 0 , 𝑦 1 1 , … , 𝑦 𝑗 𝑌 = 𝑦 1 ⋮ ⋮ ⋮ 𝑂 𝑦 𝑁 6

One-to-One Graph Matching  Formulating it as a constrained binary program 0 𝑌 ∗ = argmin 𝑔 𝑌 𝑦 1 1 𝑌 ∈ 𝒴 𝑦 1 Or 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 ⋮ where 𝑘 = 0,1 , 𝑘 𝒴 = ቄ𝑌 = 𝑦 𝑗 ∀𝑗,𝑘 | 𝑦 𝑗 𝑘 ≤ 1, ∀𝑘: ∑ 𝑦 𝑗 𝑘 = 1 ∀𝑗: ∑ 𝑦 𝑗 ቅ ⋮ ⋮ 𝑂 𝑦 𝑁 7

One-to-One Graph Matching  Formulating it as a constrained binary program 0 𝑌 ∗ = argmin 𝑔 𝑌 𝑦 1 1 𝑌 ∈ 𝒴 𝑦 1 Or 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 ⋮ where 𝑘 = 0,1 , 𝑘 𝒴 = ቄ𝑌 = 𝑦 𝑗 ∀𝑗,𝑘 | 𝑦 𝑗 𝑘 ≤ 1, ∀𝑘: ∑ 𝑦 𝑗 𝑘 = 1 ∀𝑗: ∑ 𝑦 𝑗 ቅ ⋮ ⋮ 𝑂 𝑦 𝑁 8

One-to-One Graph Matching  Formulating it as a constrained binary program 0 𝑌 ∗ = argmin 𝑔 𝑌 𝑦 1 1 𝑌 ∈ 𝒴 𝑦 1 Or 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 ⋮ where 𝑘 = 0,1 , 𝑘 𝒴 = ቄ𝑌 = 𝑦 𝑗 ∀𝑗,𝑘 | 𝑦 𝑗 𝑘 ≤ 1, ∀𝑘: ∑ 𝑦 𝑗 𝑘 = 1 ∀𝑗: ∑ 𝑦 𝑗 ቅ ⋮ ⋮ 𝑂 𝑦 𝑁 9

One-to-One Graph Matching  Formulating it as a constrained binary program 0 𝑌 ∗ = argmin 𝑔 𝑌 𝑦 1 1 𝑌 ∈ 𝒴 𝑦 1 Or 𝐵𝑌 ≤ 𝐶 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 ⋮ where 𝑘 = 0,1 , 𝑘 𝒴 = ቄ𝑌 = 𝑦 𝑗 ∀𝑗,𝑘 | 𝑦 𝑗 𝑘 ≤ 1, ∀𝑘: ∑ 𝑦 𝑗 𝑘 = 1 ∀𝑗: ∑ 𝑦 𝑗 ቅ ⋮ ⋮ 𝑂 𝑦 𝑁 10

One-to-One Graph Matching  Formulating it as a constrained binary program 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∈ 𝒴 Or 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 where 𝑘 = 0,1 , 𝑘 𝒴 = ቄ𝑌 = 𝑦 𝑗 ∀𝑗,𝑘 | 𝑦 𝑗 𝑘 ≤ 1, ∀𝑘: ∑ 𝑦 𝑗 𝑘 = 1 ∀𝑗: ∑ 𝑦 𝑗 ቅ ⋮ ⋮ 11

One-to-One Graph Matching  Examples of joint matching distribution 𝑞 𝑌 and cost 𝑔 𝑌 in different applications • Multi-target tracking [Zheng et al. , CVPR 2008] and person re-identification [Das et al. , ECCV 2014 ] 𝑘 𝑘 𝑦 𝑗 𝑔 𝑌 = 𝐷 𝑈 𝑌 or equivalently 𝑞 𝑌 ∝ ς 𝑞 𝑦 𝑗 • Feature point matching [Leordeanu et al. , IJCV 2011] 𝑔 𝑌 = 𝑌 𝑈 𝑅 𝑌 • Stereo matching [Meltzer et al. , ICCV 2005] and iterative closest point [Zheng, IJCV 1994] higher-order constraints in addition to one-to-one constraints 12

Marginalization VS MAP Estimates  In general, globally optimal solution may or may not be easily achieved. 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 𝑌 ∈ 𝒴  Even the optimal solution does not necessarily yield the correct matching assignment 13

Marginalization VS MAP Estimates  In general, globally optimal solution may or may not be easily achieved. 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 𝑌 ∈ 𝒴  Even the optimal solution does not necessarily yield the correct matching assignment • Visual similarity • Other ambiguities in the matching space 14

Marginalization VS MAP Estimates  In general, globally optimal solution may or may not be easily achieved. 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 𝑌 ∈ 𝒴  Even the optimal solution does not necessarily yield the correct matching assignment • Visual similarity • Other ambiguities in the matching space 15

Marginalization VS MAP Estimates  In general, globally optimal solution may or may not be easily achieved. 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 𝑌 ∈ 𝒴  Even the optimal solution does not necessarily yield the correct matching assignment 16

Marginalization VS MAP Estimates  In general, globally optimal solution may or may not be easily achieved. 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 𝑌 ∈ 𝒴  Even the optimal solution does not necessarily yield the correct matching assignment 17

Marginalization VS MAP Estimates Motivation to use marginalization  Encoding the entire distribution to untangle potential ambiguities  MAP only considers one single value of that distribution  Improving matching ranking due to averaging / smoothing property Exact marginalization is NP-hard  Requiring all feasible permutations to built the joint distribution Solution  Approximation using m -Best solutions 18

Marginalization Using m -Best Solutions Marginalization by considering a fraction of the matching space  Using m -highest joint probabilities 𝑞 𝑌 / m -lowest values for 𝑔 𝑌 19

Marginalization Using m -Best Solutions Marginalization by considering a fraction of the matching space  Using m -highest joint probabilities 𝑞 𝑌 / m -lowest values for 𝑔 𝑌 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 ∗ is 𝑌 1 1-st optimal solution 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∈ 𝒴 20

Marginalization Using m -Best Solutions Marginalization by considering a fraction of the matching space  Using m -highest joint probabilities 𝑞 𝑌 / m -lowest values for 𝑔 𝑌 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 ∗ is 𝑌 2 2-nd optimal solution 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∈ 𝒴 21

Marginalization Using m -Best Solutions Marginalization by considering a fraction of the matching space  Using m -highest joint probabilities 𝑞 𝑌 / m -lowest values for 𝑔 𝑌 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 ∗ is 𝑌 3 3-rd optimal solution 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∈ 𝒴 22

Marginalization Using m -Best Solutions Marginalization by considering a fraction of the matching space  Using m -highest joint probabilities 𝑞 𝑌 / m -lowest values for 𝑔 𝑌 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 ∗ is 𝑌 𝑙 k- th optimal solution 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∈ 𝒴 23

Marginalization Using m -Best Solutions Marginalization by considering a fraction of the matching space  Using m -highest joint probabilities 𝑞 𝑌 / m -lowest values for 𝑔 𝑌 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 ∗ is 𝑌 𝑙 k -th optimal solution 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∈ 𝒴 24

Marginalization Using m -Best Solutions Marginalization by considering a fraction of the matching space  Using m -highest joint probabilities 𝑞 𝑌 / m -lowest values for 𝑔 𝑌 𝑌 ∗ = argmax 𝑞 𝑌 𝑌 ∈ 𝒴 ∗ is 𝑌 𝑙 k -th optimal solution 𝑌 ∗ = argmin 𝑔 𝑌 𝑌 ∈ 𝒴  Approximation error bound decreases exponentially by increasing number of solutions [Rezatofighi et al. , ICCV 2015] 25

Computing the m -Best Solutions Naïve exclusion strategy ∗ = argmin 𝑔 𝑌 𝑌 1 𝐵𝑌 ≤ 𝐶 26

Computing the m -Best Solutions Naïve exclusion strategy ∗ = argmin 𝑔 𝑌 𝑌 2 𝐵𝑌 ≤ 𝐶 ∗ ≤ ∗ 𝑌, 𝑌 1 𝑌 1 1 − 1 27

Computing the m -Best Solutions Naïve exclusion strategy ∗ = argmin 𝑔 𝑌 𝑌 3 𝐵𝑌 ≤ 𝐶 ∗ ≤ ∗ 𝑌, 𝑌 1 𝑌 1 1 − 1 ∗ ≤ ∗ 𝑌, 𝑌 2 𝑌 2 1 − 1 28

Computing the m -Best Solutions Naïve exclusion strategy ∗ = argmin 𝑔 𝑌 𝑌 𝑙 𝐵𝑌 ≤ 𝐶 ∗ ≤ ∗ 𝑌, 𝑌 1 𝑌 1 1 − 1 ∗ ≤ ∗ 𝑌, 𝑌 2 𝑌 2 1 − 1 ⋮ ∗ ∗ 𝑌, 𝑌 𝑙−1 ≤ 𝑌 𝑙−1 1 − 1 29

Computing the m -Best Solutions Naïve exclusion strategy ∗ = argmin 𝑔 𝑌  General approach 𝑌 𝑙 𝐵𝑌 ≤ 𝐶  Impractical for large values of m 𝐵𝑌 ≤ ሖ ሖ 𝐶 30

Computing the m -Best Solutions Naïve exclusion strategy ∗ = argmin 𝑔 𝑌  General approach 𝑌 𝑙 𝐵𝑌 ≤ 𝐶  Impractical for large values of m 𝐵𝑌 ≤ ሖ ሖ 𝐶 Binary Tree Partitioning Partitioning the space into a set of disjoint subspaces [Rezatofighi et al., ICCV 2015 ]  Efficient approach  Not a good strategy for weak solvers 31

Experimental Results Query images Gallery images 0 𝑑 1 Person Re-Identification None of them 1 𝑑 1 𝑂 𝑑 𝑁 32