On Medians of (Randomized) Pairwise Means Pierre Laforgue 1 , Stephan - PowerPoint PPT Presentation

On Medians of (Randomized) Pairwise Means Pierre Laforgue 1 , Stephan Cl´ ¸on 1 , Patrice Bertail 2 emenc 1 LTCI, T´ el´ ecom Paris, Institut Polytechnique de Paris, France 2 Modal’X, UPL, Universit´ e Paris-Nanterre, France 1

The Median of Means . . . . . . x n − B +1 . . . x 1 x B x n mean mean . . . ˆ ˆ θ 1 θ K median ˆ θ MoM x 1 , . . . , x n i.i.d. realizations of r.v. X s.t. E [ X ] = θ , Var( X ) = σ 2 . ∀ δ ∈ [ e 1 − n 2 , 1[, set K := ⌈ log(1 /δ ) ⌉ , it holds [Devroye et al., 2016]:   �  � �  √ 1 + log(1 /δ ) � � � ˆ θ MoM − θ � > 2 2 e σ  ≤ δ. P  n 2

The Median of Randomized Means (1 st contribution) . . . x B +1 . . . x n − 1 x 1 x B x n mean mean ¯ . . . ¯ θ 1 θ K median ¯ θ MoRM With blocks formed by SWoR, ∀ τ ∈ ]0 , 1 / 2[ , ∀ δ ∈ [2 e − 8 τ 2 n 9 , 1[, set � log(2 /δ ) � � � 8 τ 2 n K := , and B := , it holds: 2(1 / 2 − τ ) 2 9 log(2 /δ )   � √  � �  � > 3 3 σ log(2 /δ ) � � � ¯ θ MoRM − θ  ≤ δ. P  2 τ 3 / 2 n 3

U -statistics & Pairwise Learning Estimate E [ h ( X 1 , X 2 )] from an i.i.d. sample x 1 , . . . , x n : � 2 U n ( h ) = h ( x i , x j ) . n ( n − 1) 1 ≤ i < j ≤ n Ex: the empirical variance when h ( x , x ′ ) = ( x − x ′ ) 2 . 2 Encountered e.g. in pairwise ranking or in metric learning : � 2 � R n ( r ) = ✶ { r ( x i , x j ) · ( y i − y j ) ≤ 0 } . n ( n − 1) 1 ≤ i < j ≤ n 4

The Median of (Randomized) U -statistics (2 nd contribution) Blocks are formed either by partitioning or by SWoR. Medians of the (randomized) U -statistics verify ∀ τ ∈ ]0 , 1 / 2[ w.p.a.l. 1 − δ : � � � � C 1 log 1 C 2 log 2 ( 1 � δ ) � � � � ˆ δ θ MoU − θ ( h ) � ≤ + � � , � n 2 n − 9 log 1 n δ � � � � C 2 ( τ ) log 2 ( 2 � C 1 ( τ ) log 2 δ ) � � � � ¯ δ θ MoRU − θ ( h ) � ≤ + � � , � n 8 n − 9 log 2 n δ with C 1 ( τ ) − − − → C 1 and C 2 ( τ ) − − − → C 2 . τ → 1 τ → 1 2 2 5

The Pairwise Tournament Procedure (3 rd contribution) Adapted from [Lugosi and Mendelson, 2016]. We want to find f ∗ ∈ argmin R ( f ) = E [ ℓ ( f , ( X , X ′ ))]. f ∈F � ℓ ( f , X , X ′ ). For any pair ( f , g ) ∈ F 2 : For f ∈ F , let H f := 1) Compute the MoU estimate of � H f − H g � L 1 � ˆ � U 1 | H f − H g | , . . . , ˆ Φ S ( f , g ) = median U K | H f − H g | . 2) If it is large enough , compute the match � ˆ � U 1 ( H 2 f − H 2 g ) , . . . , ˆ U K ′ ( H 2 f − H 2 Ψ S ′ ( f , g ) = median g ) . ˆ f winning all its matches verify w.p.a.l. 1 − exp( c 0 n min { 1 , r 2 } ) R (ˆ f ) − R ( f ∗ ) ≤ cr . 6

Conclusion • MoM exhibits good guarantees with few assumptions 7

Conclusion • MoM exhibits good guarantees with few assumptions • 1 st contrib. Guarantees preserved through randomization 7

Conclusion • MoM exhibits good guarantees with few assumptions • 1 st contrib. Guarantees preserved through randomization • 2 nd contrib. Extension to (randomized) U -statistics 7

Conclusion • MoM exhibits good guarantees with few assumptions • 1 st contrib. Guarantees preserved through randomization • 2 nd contrib. Extension to (randomized) U -statistics • 3 rd contrib. Pairwise tournament procedure 7

References Devroye, L., Lerasle, M., Lugosi, G., Oliveira, R. I., et al. (2016). Sub-gaussian mean estimators. The Annals of Statistics , 44(6):2695–2725. Lugosi, G. and Mendelson, S. (2016). Risk minimization by median-of-means tournaments. arXiv preprint arXiv:1608.00757 . Minsker, S. et al. (2015). Geometric Median and Robust Estimation in Banach Spaces. Bernoulli , 21(4):2308–2335. 8