making generative classifiers robust to selection bias
play

Making Generative Classifiers Robust to Selection Bias Andrew Smith - PowerPoint PPT Presentation

May 12, 2023 •131 likes •736 views

Making Generative Classifiers Robust to Selection Bias Andrew Smith Charles Elkan November 30th, 2007 Outline What is selection bias? Types of selection bias. Overcoming learnable bias with weighting. Overcoming bias with

  1. Making Generative Classifiers Robust to Selection Bias Andrew Smith Charles Elkan November 30th, 2007

  2. Outline ◮ What is selection bias? ◮ Types of selection bias. ◮ Overcoming learnable bias with weighting. ◮ Overcoming bias with maximum likelihood (ML). ◮ Experiment 1: ADULT dataset. ◮ Experiment 2: CA-housing dataset. ◮ Future work & conclusions.

  3. What is selection bias? Traditional semi-supervised learning assumes: ◮ Some samples are labeled, some are not. ◮ Labeled and unlabeled examples are identically distributed.

  4. What is selection bias? Traditional semi-supervised learning assumes: ◮ Some samples are labeled, some are not. ◮ Labeled and unlabeled examples are identically distributed. Semi-supervised learning under selection bias: ◮ Labeled examples are selected from the general population not at random. ◮ Labeled and unlabeled examples may be differently distributed .

  5. Examples ◮ Loan application approval ◮ Goal is to model repay/default behavior of all applicants . ◮ But the training set only includes labels for people who were approved for a loan.

  6. Examples ◮ Loan application approval ◮ Goal is to model repay/default behavior of all applicants . ◮ But the training set only includes labels for people who were approved for a loan.

  7. Examples ◮ Loan application approval ◮ Goal is to model repay/default behavior of all applicants . ◮ But the training set only includes labels for people who were approved for a loan. ◮ Spam filtering ◮ Goal is an up-to-date spam filter. ◮ But, while up-to-date unlabeled emails are available, hand-labeled data sets are expensive and may be rarely updated .

  8. Framework Types of selection bias are distinguised by conditional independence assumptions between: ◮ x is the feature vector. ◮ y is the class label. If y is binary, y ∈ { 1 , 0 } . ◮ s is the binary selection variable. If y i is observable then s i = 1, otherwise s i = 0.

  9. Types of selection bias – No bias x y s s ⊥ x , s ⊥ y ◮ The standard semi-supervised learning scenario.

  10. Types of selection bias – No bias x y s s ⊥ x , s ⊥ y ◮ The standard semi-supervised learning scenario. ◮ Labeled examples are selected completely at random from the general population.

  11. Types of selection bias – No bias x y s s ⊥ x , s ⊥ y ◮ The standard semi-supervised learning scenario. ◮ Labeled examples are selected completely at random from the general population. ◮ The missing labels are said to be “missing completely at random” (MCAR) in the literature.

  12. Types of selection bias – Learnable bias x y s s ⊥ y | x ◮ Labeled examples are selected from the general population only depending on features x .

  13. Types of selection bias – Learnable bias x y s s ⊥ y | x ◮ Labeled examples are selected from the general population only depending on features x . ◮ A model p ( s | x ) is learnable.

  14. Types of selection bias – Learnable bias x y s s ⊥ y | x ◮ Labeled examples are selected from the general population only depending on features x . ◮ A model p ( s | x ) is learnable. ◮ The missing labels are said to be “missing at random” (MAR), or ”ignorable bias” in the literature. ◮ p ( y | x , s = 1) = p ( y | x ).

  15. Model mis-specification under learnable bias p ( y | x , s = 1) = p ( y | x ) implies decision boundaries are the same in the labeled and general populations. But suppose the model is misspecified?

  16. Model mis-specification under learnable bias p ( y | x , s = 1) = p ( y | x ) implies decision boundaries are the same in the labeled and general populations. But suppose the model is misspecified? Then a sub-optimal decision boundary may be learned under MAR bias. Ignoring samples Viewing hidden labels without labels + + + + + + + + + + + + + + + + + + + + + + + + + + + + + − + + + + + + + + + + − + − + + + + + + + + + + + + + + + + + + + + + + + + + + − + + + + + + + + + + + + + + + + + + + + − + + + + + − + + + + + + − + + + + + + + + + + + − + + + + + + + + + + + + + + − + − + + + + + + + + + + + + + − − + + + − + + + + − + + + + − + + − − + + + + + + + + + + + + + − − + + − − − + + + + + + + + + + + + + + − + + + + + − − + + + + − + − + + + + + + + − + + − − − + − + + − − − + + − − + + + + − − − − + − − − − + + + + − + − + − + − + + + + + − + − − − − + + + − + + + − − − + + + + − + + + − − − − − + + + − − + − − + + + − + + − − − + + + + + − − − − + − − − − + − − + − − + − − − − + + + + + − − − + + − − − − − − − − − + + − − − − − − − − − − + − − − + + + − − − − − − − − − − − − − − − − − − + + − − + − − + + − + + − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − best mis−specified bondary estim. mis−specified boundary − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − true boundary − − estim. well−specified boundary − − − − − − − − − − − − − − −

  17. Types of selection bias – Arbitrary bias x y s ◮ Labeled examples are selected from the general population possibly depending on the label itself.

  18. Types of selection bias – Arbitrary bias x y s ◮ Labeled examples are selected from the general population possibly depending on the label itself. ◮ No independence assumptions can be made.

  19. Types of selection bias – Arbitrary bias x y s ◮ Labeled examples are selected from the general population possibly depending on the label itself. ◮ No independence assumptions can be made. ◮ The missing labels are said to be “missing not at random” (MNAR) in the literature.

  20. Overcoming bias – Two alternate goals The training data consist of { ( x i , y i ) | s i = 1 } and { ( x i ) | s i = 0 } . Two goals are possible:

  21. Overcoming bias – Two alternate goals The training data consist of { ( x i , y i ) | s i = 1 } and { ( x i ) | s i = 0 } . Two goals are possible: ◮ General population modeling: Learn p ( y | x ), e.g. loan application approval.

  22. Overcoming bias – Two alternate goals The training data consist of { ( x i , y i ) | s i = 1 } and { ( x i ) | s i = 0 } . Two goals are possible: ◮ General population modeling: Learn p ( y | x ), e.g. loan application approval. ◮ Unlabeled population modeling: Learn p ( y | x , s = 0), e.g. spam filtering.

  23. Overcoming learnable bias – General population modeling Lemma 1 Under MAR bias in the labeling, p ( s = 1) p ( x , y ) = p ( s = 1 | x ) p ( x , y | s = 1) if all probabilities are non-zero. The distribution of samples in the general population is a weighted version of the distribution of labeled samples. Since p ( s | x ) is learnable, we can estimate weights.

Recommend

Making Default Address Selection More Robust FoolProof

Making Default Address Selection More Robust FoolProof

Making Default Address Selection More Robust FoolProof draft-linkova-6man-default-addr-selection-update-00 Jen Linkova IETF99, Prague, July 2017 When Does a Host Stop Using an Address? Preferred lifetime expired An RA received

540 views • 14 slides
Cognitive Modeling Unseen Examples 2 Bayes Classifiers Lecture 14: Naive Bayes Classifiers

Cognitive Modeling Unseen Examples 2 Bayes Classifiers Lecture 14: Naive Bayes Classifiers

Decision Making as Classification Decision Making as Classification Bayes Classifiers Bayes Classifiers Naive Bayes Classifiers Naive Bayes Classifiers 1 Decision Making as Classification Decision Making Frequencies and Probabilities

269 views • 5 slides
Logistic Regression, Generative and Discriminative Classifiers Recommended reading: Ng and

Logistic Regression, Generative and Discriminative Classifiers Recommended reading: Ng and

Logistic Regression, Generative and Discriminative Classifiers Recommended reading: Ng and Jordan paper On Discriminative vs. Generative classifiers: A comparison of logistic regression and nave Bayes, A. Ng and M. Jordan, NIPS

392 views • 28 slides
Fast and Robust Classifiers Adjusted for Skewness Mia Hubert and Stephan Van der Veeken

Fast and Robust Classifiers Adjusted for Skewness Mia Hubert and Stephan Van der Veeken

Fast and Robust Classifiers Adjusted for Skewness Mia Hubert and Stephan Van der Veeken Katholieke Universiteit Leuven, Department of Mathematics Mia.Hubert@wis.kuleuven.be COMPSTAT 2010 Mia Hubert, August 24, 2010 Robust classifiers. - p.

729 views • 39 slides
Decision-making Bias in Instance Matching Model Selection Mayank Kejriwal, Daniel P . Miranker

Decision-making Bias in Instance Matching Model Selection Mayank Kejriwal, Daniel P . Miranker

Decision-making Bias in Instance Matching Model Selection Mayank Kejriwal, Daniel P . Miranker Acknowledgements: US National Science Foundation, Microsoft Research Instance Matching 50+ year old Artificial Intelligence problem When do

231 views • 19 slides
Robust Estimation and Generative Adversarial Networks Weizhi ZHU Hong Kong University of Science

Robust Estimation and Generative Adversarial Networks Weizhi ZHU Hong Kong University of Science

Robust Estimation and Generative Adversarial Networks Weizhi ZHU Hong Kong University of Science and Technology wzhuai@ust.hk April 3, 2019 Robust Estimation and Generative Adversarial Nets [GLYZ18] Generative Adversarial Nets for Robust

252 views • 24 slides
Review Selection bias, overfitting Bias v. variance v. residual Bias-variance tradeoff

Review Selection bias, overfitting Bias v. variance v. residual Bias-variance tradeoff

Review Selection bias, overfitting Bias v. variance v. residual Bias-variance tradeoff 1 n=1 Cramr-Rao bound n=4 0.8 n=30 0.6 CDF of max of n samples of 0.4 N( =2, 2 =1) 0.2 [representing error estimates for n

517 views • 28 slides
Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection

Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection

Variable Selection Variable Selection Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection Variable selection Variable Selection Bias in Classification bias bias rpart Trees and Ensemble Methods

183 views • 6 slides
Bias and Generalization in Deep Generative Models Shengjia Zhao*, Hongyu Ren*, Arianna Yuan,

Bias and Generalization in Deep Generative Models Shengjia Zhao*, Hongyu Ren*, Arianna Yuan,

Bias and Generalization in Deep Generative Models Shengjia Zhao*, Hongyu Ren*, Arianna Yuan, Jiaming Song, Noah Goodman and Stefano Ermon *equal contribution Success in Generative Modeling of Images Brock A, et al. "Large scale gan

350 views • 19 slides
Probabilistic Classifiers -- Generative Naive Bayes Announcements Math for Visual

Probabilistic Classifiers -- Generative Naive Bayes Announcements Math for Visual

Probabilistic Classifiers -- Generative Naive Bayes Announcements Math for Visual Computing Course Agenda: First classifier using our knowledge of probabilities Quick overview of Linear Algebra Least Square

384 views • 19 slides
Automatic Sample-by- sample Model Selection Between Two Off-the-shelf Classifiers Steve P.

Automatic Sample-by- sample Model Selection Between Two Off-the-shelf Classifiers Steve P.

Automatic Sample-by- sample Model Selection Between Two Off-the-shelf Classifiers Steve P. Chadwick University of Texas at Dallas Model Selection by Predicting the Better Classifier Idea: Two classifiers, "primary" and

269 views • 6 slides
Generative vs. discriminative Generative Discriminative Belief network A is more More

Generative vs. discriminative Generative Discriminative Belief network A is more More

Generative vs. discriminative Generative Discriminative Belief network A is more More robust modular Dont need precise model Class-conditional densities are specification, so long as it is likely to be local, characteristic

1.09k views • 15 slides
Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin

Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin

Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin Layton & Mark Gales 9 December 2005 Cambridge University Engineering Department NIPS 2005 Augmented Statistical Models: Exploiting Generative

726 views • 25 slides
Invertible Generative Models for Inverse Problems Mitigating Representation Error and Dataset Bias

Invertible Generative Models for Inverse Problems Mitigating Representation Error and Dataset Bias

Invertible Generative Models for Inverse Problems Mitigating Representation Error and Dataset Bias M. Asim, M. Daniels, O. Leong, P . Hand, A. Ahmed Inverse Problems with Generative Models as Image Priors Inverse Problems with Generative Models

369 views • 22 slides
Robust Statistics and Generative Adversarial Networks Yuan YAO HKUST 1 Chao GAO Jiyi LIU

Robust Statistics and Generative Adversarial Networks Yuan YAO HKUST 1 Chao GAO Jiyi LIU

Robust Statistics and Generative Adversarial Networks Yuan YAO HKUST 1 Chao GAO Jiyi LIU Weizhi ZHU U. Chicago Yale U. HKUST 2 Deep Learning is Notoriously Not Robust! Imperceivable adversarial examples are ubiquitous to fail

833 views • 72 slides
Selection Regions Assume we have a set of classifiers D = { D 1, D 2, ..., D L } Let R n be

Selection Regions Assume we have a set of classifiers D = { D 1, D 2, ..., D L } Let R n be

Selection Regions Assume we have a set of classifiers D = { D 1, D 2, ..., D L } Let R n be divided into K selection regions (also called regions of competence ) called {R 1, R 2, ..., R k } Let E map each input x to its corresponding Region R j E

466 views • 3 slides
Steganalysis by Ensemble Classifiers with Boosting by Regression, and Post-Selection of Features

Steganalysis by Ensemble Classifiers with Boosting by Regression, and Post-Selection of Features

Steganalysis by Ensemble Classifiers - Marc Chaumont - ICIP2012 Steganalysis by Ensemble Classifiers with Boosting by Regression, and Post-Selection of Features Marc Chaumont, Sarra Kouider LIRMM, Montpellier, France October 2, 2012 IEEE

376 views • 26 slides
Robust Statistics and Generative Adversarial Networks Yuan YAO HKUST Chao Gao (Chicago) Jiyu

Robust Statistics and Generative Adversarial Networks Yuan YAO HKUST Chao Gao (Chicago) Jiyu

Robust Statistics and Generative Adversarial Networks Yuan YAO HKUST Chao Gao (Chicago) Jiyu Liu (Yale) Weizhi Zhu (HKUST) Deep Learning is Notoriously Not Robust! Imperceivable adversarial examples are ubiquitous to fail neural networks

1.8k views • 133 slides
CSE 571 A simple algorithm for learning robust Probabilistic Robotics classifiers Freund

CSE 571 A simple algorithm for learning robust Probabilistic Robotics classifiers Freund

11/13/2007 Why boosting? CSE 571 A simple algorithm for learning robust Probabilistic Robotics classifiers Freund & Shapire, 1995 Friedman, Hastie, Tibshhirani, 1998 Boosting Provides efficient algorithm for sparse feature

739 views • 8 slides
A Study on Generative Adversarial Networks Exacerbating Social Data Bias Thesis by Niharika Jain

A Study on Generative Adversarial Networks Exacerbating Social Data Bias Thesis by Niharika Jain

A Study on Generative Adversarial Networks Exacerbating Social Data Bias Thesis by Niharika Jain Chair: Dr. Subbarao Kambhampati Committee Members: Dr. Huan Liu and Dr. Lydia Manikonda https://www.forbes.com/sites/bernardmarr/2018/11/05/does-

287 views • 17 slides
How Non Ignorable is the Selection Bias in Non Probability Samples? An Illustration of New

How Non Ignorable is the Selection Bias in Non Probability Samples? An Illustration of New

11/8/2018 How Non Ignorable is the Selection Bias in Non Probability Samples? An Illustration of New Measures using a Large Genetic Study on Facebook Brady T. West 1 Phil Boonstra 2 Roderick J.A. Little 1,2 Jingwei Hu 1 Fernanda Alvarado

382 views • 14 slides
Nonlinear Classifiers II 2 Nonlinear Classifiers: Introduction Classifiers Supervised

Nonlinear Classifiers II 2 Nonlinear Classifiers: Introduction Classifiers Supervised

1 Nonlinear Classifiers II 2 Nonlinear Classifiers: Introduction Classifiers Supervised Classifiers XOR problem Linear Classifiers Perceptron Least Squares Methods Linear Support Vector Machine Nonlinear

355 views • 20 slides
Robust Models in Information Retrieval Nedim Lipka Benno Stein Bauhaus-Universitt Weimar

Robust Models in Information Retrieval Nedim Lipka Benno Stein Bauhaus-Universitt Weimar

Robust Models in Information Retrieval Nedim Lipka Benno Stein Bauhaus-Universitt Weimar [www.webis.de] Robust Models in Information Retrieval Outline Introduction Bias and Variance Robust Models in IR Summary Excursus: Bias

821 views • 51 slides
So you think you have all the data? Causes and consequences of selection bias David J. Hand

So you think you have all the data? Causes and consequences of selection bias David J. Hand

So you think you have all the data? Causes and consequences of selection bias David J. Hand Imperial College London and Winton Capital Management 16 th September 2016 1 BACKGROUND The promise of big data: Out with every theory of human behavior,

402 views • 38 slides

More recommend