Promoting Fairness in Learned Models by Learning to Active Learn - PowerPoint PPT Presentation

Promoting Fairness in Learned Models by Learning to Active Learn under Parity Constraints Amr Sharaf Hal Daumé III University of Maryland University of Maryland Microsoft Research amr@cs.umd.edu me@hal3.name 1

Can we learn to active learn under fairness parity constraints? 2

P ANDA Test Time Behavior U Pre-existing data D = (U,) 3

P ANDA Test Time Behavior Transformer Selection Policy π Feed Forward Decoder U Pre-existing data D = (U,) 4

P ANDA Test Time Behavior x B sampled items Transformer Selection Policy π Distribution Q Over U, Y Feed Forward Decoder + U Q= π (h 0 ,D) Gumbel(0) Pre-existing data D = (U,) 5

P ANDA Test Time Behavior x B sampled items Transformer Selection Policy π Distribution Q Over U, Y Feed Forward Decoder + U Train Classifier Q= π (h 0 ,D) Gumbel(0) h B = A(D B ) on B Samples Pre-existing data D = (U,) 6

P ANDA Test Time Behavior Evaluate Meta-Loss on held-out data V x B sampled items Transformer on accuracy / parity: Selection Policy π 𝔽 V ℓ (h B ) / Δ v (h B ) Distribution Q Over U, Y Feed Forward Decoder + U Train Classifier Q= π (h 0 ,D) Gumbel(0) h B = A(D B ) on B Samples Pre-existing data D = (U,) 7

Goal: can we manage an e ffi cacy vs annotation cost trade-o ff under a target parity constraint? 8

P ANDA Train Time Behavior Evaluate Meta-Loss on held-out data V x B sampled items Transformer on accuracy / parity: Selection Policy π 𝔽 V ℓ (h B ) / Δ v (h B ) Distribution Q Over U, Y Feed Forward Decoder + U Train Classifier Q= π (h 0 ,D) Gumbel(0) h B = A(D B ) on B Samples 9

P ANDA Train Time Behavior Evaluate Meta-Loss on held-out data V x B sampled items Transformer on accuracy / parity: Selection Policy π 𝔽 V ℓ (h B ) / Δ v (h B ) Distribution Q Over U, Y Feed Forward Decoder + Y U Train Classifier Q= π (h 0 ,D) Gumbel(0) h B = A(D B ) on B Samples Pre-existing data D = (U, Y) 10

P ANDA Train Time Behavior Evaluate Meta-Loss on held-out data V x B sampled items Transformer on accuracy / parity: Selection Policy π 𝔽 V ℓ (h B ) / Δ v (h B ) Distribution Q Over U, Y Feed Forward Decoder + Y U Train Classifier Q= π (h 0 ,D) Gumbel(0) h B = A(D B ) on B Samples Pre-existing data D = (U, Y) update π to minimize Compute Gradients w.r.t performance/parity loss parameters of π 11

Experimental Results Random Sampling Fairlearn PANDA Fair Active Learning Entropy Sampling Group Aware Random Sampling 12

Experimental Results Demographic Disparity vs Budget for F-Score vs Budget for di ff erent di ff erent Active Learning Algorithms Active Learning Algorithms 0.14 Demographic Disparity 0.7 0.105 0.6 F-score 0.07 0.5 0.035 0.4 0.3 0 0 100 200 300 400 0 100 200 300 400 Budget Budget Random Sampling Fairlearn PANDA Fair Active Learning Entropy Sampling Group Aware Random Sampling 13

Experimental Results Demographic Disparity vs F-Score Error Rate Balance vs F-Score 0.6 0.6 0.563 0.563 F-Score F-Score 0.525 0.525 0.488 0.488 0.45 0.45 0.025 0.044 0.062 0.081 0.1 0.025 0.119 0.213 0.306 0.4 Demographic Disparity Error Rate Balance Random Sampling Fairlearn PANDA Fair Active Learning Entropy Sampling Group Aware Random Sampling Fairlearn PANDA Fair Active Learning Entropy Sampling Group Aware 14

Conclusion - Q: Can we learn to active learn under fairness parity constraints? - A: Yes, using meta-learning + Forward Backward Splitting; - We compare to alternative active learning strategies; - P ANDA outperforms alternative strategies in most setting. 15

Questions? amr@cs.umd.edu 16