SVMpAUC-tight: A new algorithm for optimizing partial AUC based on a - PowerPoint PPT Presentation

SVMpAUC-tight: A new algorithm for optimizing partial AUC based on a tight convex upper bound Harikrishna Narasimhan and Shivani Agarwal Department of Computer Science and Automation Indian Institute of Science, Bangalore

Receiver Operating Characteristic Curve

Receiver Operating Characteristic Curve Binary Classification Vs. Spam Non-Spam Area Under the ROC Curve (AUC)

Receiver Operating Characteristic Curve Binary Classification Vs. Spam Non-Spam Area Under the ROC Curve (AUC) Bipartite Ranking Ranking of documents

Partial AUC? Full AUC

Partial AUC? Vs Full AUC Partial AUC

Ranking http://www.google.com/

Ranking http://www.google.com/

Medical Diagnosis http://en.wikipedia.org/

Medical Diagnosis KDD Cup 2008 http://en.wikipedia.org/

Bioinformatics ― Drug Discovery ― Gene Prioritization ― Protein Interaction Prediction ― …… http://en.wikipedia.org/wiki http://commons.wikimedia.org/ http://www.google.com/imghp

Bioinformatics ― Drug Discovery ― Gene Prioritization ― Protein Interaction Prediction ― …… http://en.wikipedia.org/wiki http://commons.wikimedia.org/ http://www.google.com/imghp

Partial Area Under the ROC Curve is critical to many applications

SVMpAUC (ICML 2013) Narasimhan, H. and Agarwal, S. “ A structural SVM based approach for optimizing partial AUC ”, ICML 2013. SVMpAUC

SVMpAUC (ICML 2013) Narasimhan, H. and Agarwal, S. “ A structural SVM based approach for optimizing partial AUC ”, ICML 2013. SVMpAUC SVM-AUC Joachims, 2005

Improved Version of SVMpAUC Tighter upper bound Improved accuracy Better runtime guarantee

Outline • Overview of SVMpAUC • Upper Bound Optimized by SVMpAUC • Improved Formulation: SVMpAUC-tight • Optimization Methods • Experiments

Receiver Operating Characteristic Curve …….. x 1 + x 2 + x 3 + x m + Positive Instances Training …….. Set x 1 - x 2 - x 3 - x n - Negative Instances GOAL? Learn a scoring function

Receiver Operating Characteristic Curve …….. x 1 + x 2 + x 3 + x m + Positive Instances Training …….. Set x 1 - x 2 - x 3 - x n - Negative Instances GOAL? Learn a scoring function Rank objects Build a classifier x 5 + x 5 + x 3 + x 3 + Threshold or x 1 - x 1 - x 6 + x 6 + …. …. x n - x n -

Receiver Operating Characteristic Curve …….. x 1 + x 2 + x 3 + x m + Positive Instances Training …….. Set x 1 - x 2 - x 3 - x n - Negative Instances GOAL? Learn a scoring function Rank objects Build a classifier Quality of scoring function? x 5 + x 5 + x 3 + x 3 + Threshold or x 1 - x 1 - Threshold Assignment x 6 + x 6 + …. …. x n - x n -

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 11 9 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 11 9 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 11 9 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 11 9 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 Area Under the True Positives ROC Curve 13 (AUC) 11 9 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 Area Under the True Positives ROC Curve 13 (AUC) 11 9 8 6 5 3 False Positives 2 0 Partial AUC

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 11 9 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 11 9 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 11 9 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 11 9 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 β = 0.5 11 9 Top 3 negatives! 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 β = 0.5 11 9 Top 3 negatives! 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 β = 0.5 11 9 Top 3 negatives! 8 6 5 3 False Positives 2 0

Receiver Operating Characteristic Curve ROC Curve 20 15 14 True Positives 13 β = 0.5 11 9 Top 3 negatives! 8 6 5 3 False Positives 2 0

(1 – pAUC) for f

Convex Upper Bound (1 – pAUC) for f

Convex Upper Bound (1 – pAUC) for f + Regularizer

SVMpAUC (ICML 2013) SVMpAUC: Structural SVM Approach Narasimhan and Agarwal, 2013

SVMpAUC (ICML 2013) Ordering of training examples: n 0 0 0 0 0 1 1 0 0 0 m 1 1 0 0 1 1 1 0 0 1 SVMpAUC: Structural SVM Approach Narasimhan and Agarwal, 2013

SVMpAUC (ICML 2013) Ordering of training examples: n 0 0 0 0 0 Scoring 1 1 0 0 0 m function f 1 1 0 0 1 1 1 0 0 1 SVMpAUC: Structural SVM Approach Narasimhan and Agarwal, 2013

SVMpAUC (ICML 2013) Ordering of training examples: n 0 0 0 0 0 Scoring 1 1 0 0 0 m function f 1 1 0 0 1 1 1 0 0 1 SVMpAUC: Structural SVM Approach Narasimhan and Agarwal, 2013

SVMpAUC (ICML 2013) Ordering of training examples: n 0 0 0 0 0 Scoring 1 1 0 0 0 m function f 1 1 0 0 1 1 1 0 0 1 SVMpAUC: Structural SVM Approach Narasimhan and Agarwal, 2013

Convex Upper Bound (1 – pAUC) for f + Regularizer ≤

Convex Upper Bound (1 – pAUC) for f + Regularizer ≤ How does this upper bound look?

Convex Upper Bound (1 – pAUC) for f + Regularizer ≤ Can we obtain a tighter upper bound?

Outline • Overview of SVMpAUC • Upper Bound Optimized by SVMpAUC • Improved Formulation: SVMpAUC-tight • Optimization Methods • Experiments

Upper bound we want? 1 - pAUC ∝

Upper bound we want? 1 - pAUC ∝

Upper bound we want? 1 - pAUC ∝

Upper bound we want? 1 - pAUC ∝ ≤ pair-wise hinge loss!

Upper optimized by SVMpAUC?

Upper optimized by SVMpAUC? = pair-wise hinge loss + extra term

Upper optimized by SVMpAUC? Subset of pairs of positive-negative examples = pair-wise hinge loss + extra term

Upper optimized by SVMpAUC? Subset of pairs of positive-negative examples = ? pair-wise hinge loss + extra term

Upper optimized by SVMpAUC?

Upper optimized by SVMpAUC? ≤ pair-wise hinge loss + extra term

Upper optimized by SVMpAUC? approx. pair-wise hinge loss + extra term ≤ ≤ pair-wise hinge loss + extra term

Upper optimized by SVMpAUC? ? approx. pair-wise hinge loss + extra term ≤ ≤ ? pair-wise hinge loss + extra term

Outline • Overview of SVMpAUC • Upper Bound Optimized by SVMpAUC • Improved Formulation: SVMpAUC-tight • Optimization Methods • Experiments

Rewriting the Partial AUC Loss α = 0, β = 0.5 20 15 14 3 + 2 + 2 = 7 True Positives 13 11 9 8 6 5 3 False Positives 2 0

Rewriting the Partial AUC Loss α = 0, β = 0.5 20 15 14 3 + 2 + 2 = 7 True Positives 13 2 + 2 + 1 = 5 11 9 8 6 5 3 False Positives 2 0

Rewriting the Partial AUC Loss α = 0, β = 0.5 20 15 14 3 + 2 + 2 = 7 True Positives 13 2 + 2 + 1 = 5 11 . 9 . 8 . 6 1 + 1 + 1 = 3 5 3 False Positives 2 0

Rewriting the Partial AUC Loss 1 - AUC restricted to α = 0, β = 0.5 top β fraction of negatives 20 15 Maximum! 14 3 + 2 + 2 = 7 True Positives 13 2 + 2 + 1 = 5 11 . 9 . 8 . 6 1 + 1 + 1 = 3 5 3 False Positives 2 0

Top j β negatives

Top j β negatives SVM-AUC

Negatives j α to j β

Negatives j α to j β Truncated SVMpAUC

SVMpAUC-tight: Improved Formulation SVMpAUC objective restricted to S

SVMpAUC-tight: Top j β Improved Formulation negatives

SVMpAUC-tight: Top j β Improved Formulation negatives Same pairs of positive-negative examples = pair-wise hinge loss + extra term

SVMpAUC-tight: Negatives Improved Formulation j α to j β

SVMpAUC-tight: Negatives Improved Formulation j α to j β approx. pair-wise hinge loss + extra term ≤ ≤ pair-wise hinge loss + extra term

Outline • Overview of SVMpAUC • Upper Bound Optimized by SVMpAUC • Improved Formulation: SVMpAUC-tight • Optimization Methods • Experiments

SVMpAUC-tight: Optimization Problem + Regularizer exponential in size

SVMpAUC-tight: Optimization Problem + Regularizer exponential in size Quadratic program with an exponential number of constraints