fairness ethics and machine learning
play

Fairness, Ethics, and Machine Learning Prof. Mike Hughes Many - PowerPoint PPT Presentation

Apr 12, 2023 •184 likes •536 views

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2019s/ Fairness, Ethics, and Machine Learning Prof. Mike Hughes Many ideas/slides attributable to: Alexandra Chouldechova Moritz Hardt 3 Fairness: Unit

  1. Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2019s/ Fairness, Ethics, and Machine Learning Prof. Mike Hughes Many ideas/slides attributable to: Alexandra Chouldechova Moritz Hardt 3

  2. Fairness: Unit Objectives • How to think systematically about end-to-end ML • Where does data come from? • What features am I measuring? What protected information can leak in unintentionally? • Who will be impacted? • How to define and measure notions fairness • Use concepts: accuracy, TPR, FPR, PPV, NPV • What is achievable? What is impossible? Mike Hughes - Tufts COMP 135 - Spring 2019 4

  3. Example Concerns about Fairness Mike Hughes - Tufts COMP 135 - Spring 2019 5

  4. Unfair image search Mike Hughes - Tufts COMP 135 - Spring 2019 6

  5. Unfair Word Embeddings Mike Hughes - Tufts COMP 135 - Spring 2019 7

  6. Unfair Hiring?

  7. Job Ad Classifier: Is this fair?

  8. Unfair Recidivism Prediction Mike Hughes - Tufts COMP 135 - Spring 2019 10

  9. Focus: Binary Classifier • Let’s say we have two groups, A and B • Could be any protected group (race / gender / age) • We’re trying to build a binary classifier that will predict individuals as HIGH or LOW risk • Likelihood of recidivism • Ability to pay back a loan Mike Hughes - Tufts COMP 135 - Spring 2019 11

  10. Group Discussion • When should protected information (gender, race, age, etc) be provided as input to a predictor? • Can you build a “race-blind” classifier? • How could we measure if the predictions are fair? • Is it enough to ensure accuracy parity? • ACC( group A) = ACC( group B ) Mike Hughes - Tufts COMP 135 - Spring 2019 12

  11. Notation for Binary Classifier Mike Hughes - Tufts COMP 135 - Spring 2019 13

  12. Example of Accuracy Parity Group A Group B true outcomes Y 0 0 1 1 0 0 1 1 1 = would fail to appear in court classifier prediction C 0 0 0 0 1 1 1 1 1 = too risky for bail Is this fair? Mike Hughes - Tufts COMP 135 - Spring 2019 14

  13. Case Study: The COMPAS future crime prediction algorithm Mike Hughes - Tufts COMP 135 - Spring 2019 15

  14. COMPAS classifier HIGH RISK of future crime hold in jail before trial LOW RISK of future crime release before trial other features (e.g. demographics, questionnaire answers, family history) Mike Hughes - Tufts COMP 135 - Spring 2019 16

  15. 2 Mike Hughes - Tufts COMP 135 - Spring 2019

  16. Mike Hughes - Tufts COMP 135 - Spring 2019 18

  17. The COMPAS tool assigns defendants scores from 1 to 10 that indicate how likely they are to reoffend based on more than 100 factors, including age, sex and criminal history. Notably, race is not used. These scores profoundly affect defendants’ lives: defendants who are defined as medium or high risk, with scores of 5-10, are more likely to be detained while awaiting trial than are low-risk defendants, with scores of 1-4. Mike Hughes - Tufts COMP 135 - Spring 2019 19

  18. Mike Hughes - Tufts COMP 135 - Spring 2019 20

  19. Mike Hughes - Tufts COMP 135 - Spring 2019 21

  20. Full Document: https://www.documentcloud.org/documents/2702103-Sample-Risk- Mike Hughes - Tufts COMP 135 - Spring 2019 Assessment-COMPAS-CORE.html 22

  21. ProPublica says: “Groups have different False Pos. Rates ” Mike Hughes - Tufts COMP 135 - Spring 2019 23

  22. Compas Team Says: “Groups have same predictive value ” Mike Hughes - Tufts COMP 135 - Spring 2019 24

  23. False Positive Rate = • When true outcome is 0, how often does classifier say “1”.

  24. True Positive Rate = • When true outcome is 1, how often does classifier say “1”.

  25. Positive Predictive Value = When classifier says “1”, how often is true label 1.

  26. Negative Predictive Value = When classifier says “0”, how often is true label 0.

  27. ProPublica says: “Groups have different False Pos. Rates ” Mike Hughes - Tufts COMP 135 - Spring 2019 29

  28. Compas Team Says: “Groups have same predictive value ” Mike Hughes - Tufts COMP 135 - Spring 2019 30

  29. Worksheet Mike Hughes - Tufts COMP 135 - Spring 2019 31

  30. Equation of the Day 1 − PPV p FPR = TPR 1 − p PPV where prevalence p = Pr(Y = 1) If two groups have different p values, can we simultaneously have TPR parity AND FPR parity AND PPV parity AND NPV parity? Mike Hughes - Tufts COMP 135 - Spring 2019 32

  31. https://www.propublica.org/article/bias-in-criminal-risk-scores- is-mathematically-inevitable-researchers-say Mike Hughes - Tufts COMP 135 - Spring 2019 33

  32. Unless classifier is perfect, must chose one: Disparate Treatment (PPV or NPV not equal) or Disparate Impact (FPR or TPR not equal) Mike Hughes - Tufts COMP 135 - Spring 2019 34

  33. Try demo of making decisions from risk scores: goo.gl/P8rmA3 Mike Hughes - Tufts COMP 135 - Spring 2019 35

  34. Fairness: Unit Objectives • How to think systematically about end-to-end ML • Where does data come from? • What features am I measuring? What protected information can leak in unintentionally? • Who will be impacted? • How to define and measure notions fairness • Use concepts: accuracy, TPR, FPR, PPV, NPV • What is achievable? What is impossible? Mike Hughes - Tufts COMP 135 - Spring 2019 36

Recommend

INTRO TO ETHICS AND INTRO TO ETHICS AND FAIRNESS FAIRNESS Eunsuk Kang Required reading: R.

INTRO TO ETHICS AND INTRO TO ETHICS AND FAIRNESS FAIRNESS Eunsuk Kang Required reading: R.

INTRO TO ETHICS AND INTRO TO ETHICS AND FAIRNESS FAIRNESS Eunsuk Kang Required reading: R. Caplan, J. Donovan, L. Hanson, J. Matthews. "Algorithmic Accountability: A Primer", Data & Society (2018). 1 LEARNING GOALS LEARNING

732 views • 48 slides
ETHICS & FAIRNESS IN AI- ETHICS & FAIRNESS IN AI- ENABLED SYSTEMS ENABLED SYSTEMS

ETHICS & FAIRNESS IN AI- ETHICS & FAIRNESS IN AI- ENABLED SYSTEMS ENABLED SYSTEMS

ETHICS & FAIRNESS IN AI- ETHICS & FAIRNESS IN AI- ENABLED SYSTEMS ENABLED SYSTEMS Christian Kaestner (with slides from Eunsuk Kang) Required reading: R. Caplan, J. Donovan, L. Hanson, J. Matthews. " Algorithmic Accountability:

1.41k views • 75 slides
Fairness in Machine Learning Fairness in Supervised Learning Make decisions by machine learning:

Fairness in Machine Learning Fairness in Supervised Learning Make decisions by machine learning:

Fairness in Machine Learning Fairness in Supervised Learning Make decisions by machine learning: Software can make decision free of human biases Fairness in Supervised Learning Make decisions by machine learning: Software is

501 views • 46 slides
Fairness in Machine Learning: Part I Privacy & Fairness in Data Science CS848 Fall 2019

Fairness in Machine Learning: Part I Privacy & Fairness in Data Science CS848 Fall 2019

Fairness in Machine Learning: Part I Privacy & Fairness in Data Science CS848 Fall 2019 Outline High Level View Recap Supervised Learning: Binary Classification Survey of Approaches to Fairness in Supervised Learning Warmup:

593 views • 35 slides
Machine Learning for NLP Ethics and Machine Learning Aurlie Herbelot 2019 Centre for

Machine Learning for NLP Ethics and Machine Learning Aurlie Herbelot 2019 Centre for

Machine Learning for NLP Ethics and Machine Learning Aurlie Herbelot 2019 Centre for Mind/Brain Sciences University of Trento 1 Today 1. Predicting or not predicting? That is the question. 2. Data and people: personalisation, bubbling,

1.07k views • 66 slides
Bias and Fairness in Machine Learning Irene Y. Chen @irenetrampoline

Bias and Fairness in Machine Learning Irene Y. Chen @irenetrampoline

Bias and Fairness in Machine Learning Irene Y. Chen @irenetrampoline http://gendershades.org/overview.html https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing COMPAS Correctional Offender Management

918 views • 48 slides
Fairness in Machine Learning: Practicum Privacy & Fairness in Data Science CS848 Fall 2019

Fairness in Machine Learning: Practicum Privacy & Fairness in Data Science CS848 Fall 2019

Fairness in Machine Learning: Practicum Privacy & Fairness in Data Science CS848 Fall 2019 Human Decision Making Suppose we want to recommend a movie. Data Decision Maker Decision Jane likes Bollywood musicals. Bob: You should

3.14k views • 26 slides
Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics Debjani Saha , Candice

Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics Debjani Saha , Candice

Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics Debjani Saha , Candice Schumann, Duncan C. McElfresh, John P. Dickerson, Michelle L. Mazurek, Michael Carl Tschantz 37th International Conference on Machine Learning (ICML)

980 views • 69 slides
Bias, Fairness, Accountability, and Transparency in Machine Learning CS 115 Computing for the

Bias, Fairness, Accountability, and Transparency in Machine Learning CS 115 Computing for the

Bias, Fairness, Accountability, and Transparency in Machine Learning CS 115 Computing for the Socio-Techno Web Instructor: Brian Brubach Announcements Adjustment to deadline schedule Assignment 5 due Tuesday Project milestone 4 due

845 views • 22 slides
15: Ethics in Machine Learning, plus Artificial General Intelligence and some old Science Fiction

15: Ethics in Machine Learning, plus Artificial General Intelligence and some old Science Fiction

15: Ethics in Machine Learning, plus Artificial General Intelligence and some old Science Fiction Machine Learning and Real-world Data Ann Copestake and Simone Teufel Computer Laboratory University of Cambridge Lent 2017 Some ethical issues

403 views • 21 slides
Fairness and bias in Machine Learning A quick review on tools to detect biases in machine

Fairness and bias in Machine Learning A quick review on tools to detect biases in machine

QCon 2019 Fairness and bias in Machine Learning A quick review on tools to detect biases in machine learning model Thierry Silbermann, Tech Lead Data Science at Nubank thierry.silbermann@nubank.com.br Data collection Todays

870 views • 45 slides
Fairness and Discrimination in Mechanism Design and Machine Learning Jessie Finocchiaro 1 ,

Fairness and Discrimination in Mechanism Design and Machine Learning Jessie Finocchiaro 1 ,

Fairness and Discrimination in Mechanism Design and Machine Learning Jessie Finocchiaro 1 , Roland Maio 2 , Faidra Monachou 3 Gourab K Patro 4 , Manish Raghavan 5 , Ana-Andreea Stoica 2 , Stratis Tsirtis 6 1 Colorado 2 Columbia 3 Stanford 4 IIT

521 views • 14 slides
Recent Results on Algorithmic Fairness and Meta-Learning Massimiliano Pontil Computational

Recent Results on Algorithmic Fairness and Meta-Learning Massimiliano Pontil Computational

Recent Results on Algorithmic Fairness and Meta-Learning Massimiliano Pontil Computational Statistics and Machine Learning Istituto Italiano di Tecnologia and Department of Computer Science University College London 4th Annual Machine

439 views • 26 slides
Machine Learning 2 DS 4420 - Spring 2020 Bias and fairness Byron C. Wallace Material in this

Machine Learning 2 DS 4420 - Spring 2020 Bias and fairness Byron C. Wallace Material in this

Machine Learning 2 DS 4420 - Spring 2020 Bias and fairness Byron C. Wallace Material in this lecture modified from materials created by Jay Alammar (http://jalammar.github.io/ illustrated-transformer/) and Sasha Rush

1.6k views • 79 slides
FairSquare: Probabilistic Verification of Program Fairness Aws Albarghouthi Loris DAntoni

FairSquare: Probabilistic Verification of Program Fairness Aws Albarghouthi Loris DAntoni

FairSquare: Probabilistic Verification of Program Fairness Aws Albarghouthi Loris DAntoni Samuel Drews University of Wisconsin-Madison Aditya V. Nori Microsoft Research Machine Data Model Learning Machine Data Model Learning small

654 views • 39 slides
arXiv:1710.06921v1 [cs.CY] 18 Oct 2017 ABSTRACT and fairness-aware ML methods [6, 7, 8, 9, 10,

arXiv:1710.06921v1 [cs.CY] 18 Oct 2017 ABSTRACT and fairness-aware ML methods [6, 7, 8, 9, 10,

Themis-ml: A Fairness-aware Machine Learning Interface for End-to-end Discrimination Discovery and Mitigation Niels Bantilan Arena.io New York, NY niels.bantilan@gmail.com arXiv:1710.06921v1 [cs.CY] 18 Oct 2017 ABSTRACT and fairness-aware

311 views • 8 slides
On the Ethics of Algorithmic Decision-Making in Healthcare Dr. Thomas Grote Ethics and

On the Ethics of Algorithmic Decision-Making in Healthcare Dr. Thomas Grote Ethics and

On the Ethics of Algorithmic Decision-Making in Healthcare Dr. Thomas Grote Ethics and Philosophy Lab, Cluster of Excellence Machine Learning: New Perspectives for Science ; thomas.grote@uni-tuebingen.de 07.05.2020 My aims in this talk

637 views • 28 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
P ARADOXES IN F AIR M ACHINE L EARNING Paul Glz, Anson Kahng, and Ariel Procaccia NeurIPS 2019

P ARADOXES IN F AIR M ACHINE L EARNING Paul Glz, Anson Kahng, and Ariel Procaccia NeurIPS 2019

P ARADOXES IN F AIR M ACHINE L EARNING Paul Glz, Anson Kahng, and Ariel Procaccia NeurIPS 2019 R ESEARCH Q UESTION Fairness in machine learning vs. fairness in fair division What is the relationship between statistical notions of fairness

380 views • 15 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

559 views • 17 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

401 views • 26 slides
Reflecting opportunities and risks AI ethics and its broad range of issues Dr. Thilo

Reflecting opportunities and risks AI ethics and its broad range of issues Dr. Thilo

Reflecting opportunities and risks AI ethics and its broad range of issues Dr. Thilo Hagendorff University of Tuebingen Cluster of Excellence Machine Learning Dimensions of AI ethics events distance t irritation orientation

640 views • 36 slides
CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

1/22/19 CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine Learning? A Simple Task: Recognize Obama How do you program a computer to Recognize faces? Recommend movies? Decide which web

582 views • 5 slides

More recommend