Zhiwu Huang, Ruiping Wang, Shiguang Shan, Xianqiu Li, Xilin Chen - PowerPoint PPT Presentation

Zhiwu Huang, Ruiping Wang, Shiguang Shan, Xianqiu Li, Xilin Chen Institute of Computing Technology, Chinese Academy of Sciences Presented by Bo Xin July 9, 2015

Image Set Classification • Training/testing sample is a set of images involving a single subject + Rich information to describe subject – Complex appearance variations Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 2/24

Image Set Classification • Example – Video-based face recognition • Identify a subject with his/her video sequence – Treating video as image set Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 3/24

SPD Representation for Image Set • Represent image set with Gaussian model – From Gaussian model to SPD matrix • Information geometry theory [Amari & Nagaoka,2000; Lovri c ,2000] + 𝒏 𝑈 | − 1 𝒏 𝑒+1 𝑫 𝒏 ∼ 𝑻 = |𝑫 – 𝒪 𝑦 𝑛 , 𝐷 1 𝑈 𝒏 : covariance matrix of size 𝑒 × 𝑒 , 𝒏 : mean vector of size 𝑒 – 𝑫 Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 4/24

Riemannian Geometry of SPD Manifold • Account for Riemannian geometry – Riemannian metric 2 𝑻 1 , 𝑻 2 = 𝑼 𝟑 , 𝑼 𝟑 𝑻 1 = log 𝑻 1 𝑻 2 , log 𝑻 1 𝑻 2 𝑻 1 𝜀 𝑕 SPD matrix 𝑒 𝑈 𝑻 𝟐 𝕋 + 𝑼 𝟑 = log 𝑻 1 𝑻 2 𝑻 𝟐 𝑒 : SPD manifold 𝕋 + 𝑻 𝟑 𝑻 𝒋 : SPD matrix 𝛿(𝑢) 𝑒 : tangent space 𝑈 𝑻 𝟐 𝕋 + 𝑈 𝟑 : tangent vector 𝛿(𝑢) : geodesic 𝑒 𝕋 + Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 5/24

Riemannian Geometry of SPD Manifold • Account for Riemannian geometry – Affine-Invariant metric (AIM) −1/2 || ℱ 2 𝑻 1 , 𝑻 2 = log 𝑻 1 𝑻 2 , log 𝑻 1 𝑻 2 𝑻 1 = || log 𝑻 1 −1/2 𝑻 𝟑 𝑻 1 2 𝑒 𝑏 − 1 − 1 − 1 − 1 2 • 𝑼 𝟑 , 𝑼 𝟑 𝑻 1 = 𝑻 1 2 𝑼 𝟑 𝑻 1 2 , 𝑻 1 2 𝑼 𝟑 𝑻 1 𝑒 𝑈 𝑻 𝟐 𝕋 + 1 1 1 1 • 𝑼 𝟑 = log 𝑻 1 𝑻 2 = 𝑻 1 2 log(𝑻 1 2 𝑻 2 𝑻 1 2 )𝑻 1 2 • Computational cost is expensive 𝑻 𝟐 1 1 – Main cost: log(𝑻 1 2 𝑻 2 𝑻 1 2 ) 𝑻 𝟑 𝛿(𝑢) 𝑒 𝕋 + Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 6/24

Riemannian Geometry of SPD Manifold • Account for Riemannian geometry – Log-Euclidean metric (LEM) 2 𝑻 1 , 𝑻 2 = log 𝑻 1 𝑻 2 , log 𝑻 1 𝑻 2 𝑻 1 = || log 𝑻 1 − log 𝑻 𝟑 || ℱ 2 𝑒 𝑚 • 𝑼 𝟑 , 𝑼 𝟑 𝑻 1 = Dlog 𝑻 1 [𝑼 𝟑 ], Dlog 𝑻 1 [𝑼 𝟑 ] identity matrix • 𝑼 𝟑 = log 𝑻 1 𝑻 2 = D −1 log 𝑻 1 [log 𝑻 2 − log 𝑻 1 ] 𝑒 𝑈 𝑱 𝕋 + • Drastic reduction in computation time – Need Euclidean computation in 𝑱 the domain of matrix logarithms 𝑻 𝟑 𝛿(𝑢) 𝑒 𝕋 + Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 7/24

LEM-based Discriminant Learning Method 𝑻 1 𝑻 𝟒 𝑻 𝟑 𝑒 𝑈 𝑻 𝕋 + (c) ℋ 𝑱 𝑻 𝟐 𝛿(𝑢) 𝑒 (b) ℝ 𝑒 (a) 𝕋 + • Tangent space approximation ((a)-(b)) – e.g., Tosato et al. 2010, Carreira et al. 2012, Vemulapalli et al. 2015 • Hilbert space embedding ((a)-(c)-(b)) – e.g., Wang et al. 2012, Jayasumana et al. 2013, Minh et al. 2014 Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 8/24

LEM-based Discriminant Learning Method • Convert SPD matrix logarithm into vector-form in tangent space at identity matrix ((a)-(b1)/(b2)) – Ignore the symmetric property of SPD matrix logarithm – Work inefficiently on the SPD vector-form often of high dimensionality × 𝟑 × 𝟑 OR × 𝟑 (a) (c) (b1) (b2) Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 9/24

Our Approach 𝜊 𝑇 𝐸𝐺 𝑻 [𝜊 𝑇 ] 𝑙 𝑈 𝐺(𝑻) 𝕋 + 𝑒 𝑈 𝑻 𝕋 + 𝑬𝑮(𝑻) 𝐺(𝑻 ) 𝑻 𝐺(𝛿 𝑢 ) 𝛿(𝑢) 𝑙 𝕋 + 𝐺 𝑒 𝕋 + (d) (e) • Learn tangent-to-tangent map 𝑬𝑮(𝑻) – Work on the matrix-form of SPD matrix logarithm ((d)-(e)) • Keep the symmetric property of SPD matrix logarithm • Work efficiently on lower-dimensional matrix-form Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 10/24

Our Approach 𝜊 𝑇 𝐸𝐺 𝑻 [𝜊 𝑇 ] 𝑙 𝑈 𝐺(𝑻) 𝕋 + 𝑒 𝑈 𝑻 𝕋 + 𝑬𝑮(𝑻) 𝐺(𝑻 ) 𝑻 𝐺(𝛿 𝑢 ) 𝛿(𝑢) 𝑙 𝕋 + 𝐺 𝑒 𝕋 + • Learn tangent-to-tangent map 𝑒 to a more discriminant – From original tangent space 𝑈 𝑻 𝕋 + 𝑙 tangent space 𝑈 𝐺(𝑻) 𝕋 + 𝑒 → 𝑈 𝑙 • 𝐸𝐺 𝑻 : 𝑈 𝑻 𝕋 + 𝐺(𝑻) 𝕋 + 𝑒 → 𝕋 + 𝑙 • If 𝐸𝐺 𝑻 is an injection, the manifold-to-manifold map 𝐺: 𝕋 + is an immersion Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 11/24

Our Approach 𝜊 𝑇 𝐸𝐺 𝑻 [𝜊 𝑇 ] 𝑙 𝑈 𝐺(𝑻) 𝕋 + 𝑒 𝑈 𝑻 𝕋 + 𝑬𝑮(𝑻) 𝐺(𝑻 ) 𝑻 𝐺(𝛿 𝑢 ) 𝛿(𝑢) 𝑙 𝕋 + 𝐺 𝑒 𝕋 + • Learn tangent-to-tangent map – Specific form of tangent map • 𝐸𝐺(𝑻): 𝑔 log(𝑻) = 𝑿 𝑈 log(𝑻)𝑿 – log 𝑻 ∈ ℝ 𝑒×𝑒 , 𝑿 ∈ ℝ 𝑒×𝑙 , 𝑔 log(𝑻) ∈ ℝ 𝑙×𝑙 – if 𝑿 : column full rank, 𝑔 log(𝑻) yields a valid symmetric matrix Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 12/24

Our Approach 𝜊 𝑇 𝐸𝐺 𝑻 [𝜊 𝑇 ] 𝑙 𝑈 𝐺(𝑻) 𝕋 + 𝑒 𝑈 𝑻 𝕋 + 𝑬𝑮(𝑻) 𝐺(𝑻 ) 𝑻 𝐺(𝛿 𝑢 ) 𝛿(𝑢) 𝑙 𝕋 + 𝐺 𝑒 𝕋 + • Log-Euclidean metric on new SPD manifold 2 𝑔(𝑼 𝑗 ), 𝑔(𝑼 𝑘 ) = ||𝑿 𝑈 𝑼 𝑗 𝑿 − 𝑿 𝑈 𝑼 𝑘 𝑿|| 𝐺 2 * 𝑿𝑿 𝑈 (𝑼 𝑗 − 𝑼 𝑘 ) is required – 𝑒 𝑚 to be symmetric = 𝑢𝑠(𝑹(𝑼 𝑗 − 𝑼 𝑘 )(𝑼 𝑗 − 𝑼 𝑘 )) – 𝑼 𝑗 = log 𝑻 𝑗 , 𝑼 𝑘 = log 𝑻 𝑘 – 𝑹 = (𝑿𝑿 𝑈 ) 2 : PSD matrix Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 13/24

Our Approach • Objective function (matrix-form of the ITML method [Davis et al., 2007]) – arg min 𝐸 ℓ𝑒 𝑹, 𝑹 𝟏 + 𝜃𝐸 ℓ𝑒 𝝄, 𝝄 𝟏 𝑹,𝝄 𝑈 𝐁 𝑗𝑘 ≤ 𝝄 𝑑 𝑗,𝑘 , 𝑑 𝑗, 𝑘 ∈ 𝑻 s. t. , tr 𝑹𝑩 𝑗𝑘 𝑈 𝐁 𝑗𝑘 ≥ 𝝄 𝑑 𝑗,𝑘 , 𝑑(𝑗, 𝑘) ∈ 𝑬 tr 𝑹𝑩 𝑗𝑘 – 𝐸 ℓ𝑒 : LogDet divergence, 𝐁 𝑗𝑘 = log 𝑫 𝑗 − log (𝑫 𝑘 ), – 𝑻/𝑬 : constraint set involving sample pairs with the same /different label(s) Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 14/24

Our Approach • Optimization algorithm – Cyclic Bregman projection algorithm [Bregman,1967; Censor & Zenior, 1997] • Choose one constraint per iteration • Perform a projection so that the current solution satisfies the chosen constraint Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 15/24

Evaluated Methods Method Literature source abbr. Pennec et al ., IJCV’2006 AIM Sra et al ., NIPS’2012 SPD basic metric Stein Arsigny et al ., SIAM MAA’2007 LEM Harandi et al . , ECCV’2014 SPDML-AIM/Stein Harandi et al ., ECCV’2012 RSR-Stein SPD metric learning Wang et al ., CVPR’2012 CDL-LEM Vemulapalli et al., arXiv’2015 ITML-LEM • SPDML-AIM/Stein: SPD manifold learning (SPDML) with Affine-Invariant metric (AIM) or Stein divergence • RSR-Stein: Riemannian Sparse Representation (RSR) with Stein divergence • CDL-LEM: Covariance Discriminative Learning (CDL) with Log-Euclidean metric (LEM) • ITML-LEM: Information-Theoretic Metric Learning (ITML) on vector-form of SPD matrix logarithm with Log-Euclidean Metric (LEM) Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 16/24

Set-based Object Categorization • ETH-80 dataset (Leibe & Schiele, 2003) – 80 image sets of 8 object categories • Each category has 10 image sets – 20 × 20 resized intensity images | − 1 + 𝒏 𝑈 𝒏 𝑒+1 𝑫 𝒏 𝑻 = |𝑫 – 401 × 401 SPD feature 1 𝑈 𝒏 : covariance matrix, 𝒏 𝑫 : mean – Random selection for 10 tests • 50% for gallery, 50% for probe Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 17/24

Set-based Object Categorization: Results Method Accuracy 87.50 ± 5.77 AIM 88.00 ± 5.11 Stein 89.25 ± 4.72 LEM 90.75 ± 3.34 SPDML-AIM 90.50 ± 3.87 SPDML-Stein 93.25 ± 3.34 RSR-Stein 93.75 ± 3.43 CDL-LEM 93.75 ± 3.43 ITML-LEM 94.75 ± 2.49 LEML 96.00 ± 2.11 LEML-CDL Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 18/24

Video-based Face Identification • YouTube Celebrities dataset (Kim et al., 2008) – 1,910 video sequences of 47 subjects from YouTube • Highly compressed, low resolution – 20 × 20 resized intensity images | − 1 + 𝒏 𝑈 𝒏 𝑒+1 𝑫 𝒏 𝑻 = |𝑫 1 𝑈 𝒏 – 401 × 401 SPD feature : covariance matrix, 𝒏 𝑫 : mean – Random selection for 10 tests • 3 of 9 for gallery, 6 of 9 for probe Z. Huang, R. Wang, S. Shan, X. Li, X. Chen Log-Euclidean Metric Learning July 9, 2015 19/24