Performance Measures: Stochastic Optimization & Statistical - PowerPoint PPT Presentation

Our goal: In n practice ice: (surrog ogate) te)

Our goal: In n practice ice: (surrog ogate) te) Part I was about solving this problem for non-decomposable measures with linear predictors

Our goal: In n practice ice: (surrog ogate) te) ?

does the given learning algorithm for a performance measure converge in in the limi imit of of inf nfinite te tr training data ta to the (Bayes) optimal mal pre redict ictor or for the measure?

Statistical Consistency Data Space Model l Space

Statistical Consistency Data Space Model l Space

Statistical Consistency Data Space Model l Space regr gret

Statistical Consistency Data Space Model l Space regret P → 0 ? regr gret

Statistical Consistency Underlying (unknown) distribution D over instances and labels

Statistical Consistency Underlying (unknown) distribution D over instances and labels

Statistical Consistency Underlying (unknown) distribution D over instances and labels

Statistical Consistency Underlying (unknown) distribution D over instances and labels

Statistical Consistency

Statistical Consistency • Decomposable measures – 0-1 classification error: Zhang, 04; Bartlett et al., 06 – Cost-weighted classification error: Scott, 12 – Balanced classification error: Narasimhan et al. , 13 – Logistic, squared, exponential losses (strictly proper losses): Reid & Williamson, 09, 10 • Pair-wise measures – AUC: Clemencon et al., 08; Agarwal et al., 14

Statistical Consistency • Decomposable measures – 0-1 classification error: Zhang, 04; Bartlett et al., 06 – Cost-weighted classification error: Scott, 12 – Balanced classification error: Narasimhan et al. , 13 – Logistic, squared, exponential losses (strictly proper losses): Reid & Williamson, 09, 10 • Pair-wise measures – AUC: Clemencon et al., 08; Agarwal et al., 14 • General non-decomposable measure?

Part I Stochastic Gradient Methods for Non-decomposable Performance Measures Part II Statistical Consistency of Plug-in Methods for Non-decomposable Performance Measures • Plug-in methods for classification measures • Main consistency result • Experimental results • Proof intuition

Plug-in Method Training Set

Plug-in Method Training Set Class Probability Estimate

Plug-in Method Training Set Class Probability Estimate Threshold Choice

Classification Measures -1 +1 +1 -1

Classification Measures -1 +1 tr true ue positive ive +1 (TPR) tr true ue nega gative ive -1 (TNR)

Classification Measures

Classification Measures AM-measure (1 - BER)

Classification Measures G-mean

Classification Measures F-measure where Prec = p proportion of p points ts with y =1 | h(x) = 1

Classification Measures non-dec ecomp mposab sable le

More formally, Underlying (unknown) distribution D with:

More formally, Underlying (unknown) distribution D with: proportion of positives