fair resource allocation in federated learning
play

Fair Resource Allocation in Federated Learning Tian Li (CMU) , - PowerPoint PPT Presentation

Jan 27, 2023 •619 likes •1.19k views

Fair Resource Allocation in Federated Learning Tian Li (CMU) , Maziar Sanjabi (Facebook AI), Ahmad Beirami (Facebook AI), Virginia Smith (CMU) tianli@cmu.edu 1 Federated Learning 2 2 Federated Learning Privacy-preserving training in

  1. Fair Resource Allocation in Federated Learning Tian Li (CMU) , Maziar Sanjabi (Facebook AI), Ahmad Beirami (Facebook AI), Virginia Smith (CMU) tianli@cmu.edu 1

  2. Federated Learning 2 2

  3. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2 2

  4. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2 2

  5. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2 2

  6. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2

  7. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2

  8. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2

  9. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2

  10. Challenges 3

  11. Challenges ) … ( objective: + p 2 min p 1 + p N + F 1 F 2 F N w 3

  12. Challenges ) … ( objective: + p 2 min p 1 + p N + F 1 F 2 F N w no accuracy guarantee for individual devices 3

  13. Challenges ) … ( objective: + p 2 min p 1 + p N + F 1 F 2 F N w no accuracy guarantee for individual devices model performance can vary widely 3

  14. Challenges ) … ( objective: + p 2 min p 1 + p N + F 1 F 2 F N w no accuracy guarantee for individual devices Can we devise an efficient federated optimization method to encourage a more fair (i.e., more uniform ) distribution of the model performance across devices? model performance can vary widely 3

  15. Fair Resource Allocation Objective

  16. Fair Resource Allocation Objective ) … ( + p 2 min p 1 + p N + F 1 F 2 F N w

  17. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w

  18. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w Inspired by 𝛽 -fairness for fair resource allocation in wireless networks

  19. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w Inspired by 𝛽 -fairness for fair resource allocation in wireless networks A tunable framework ( : previous objective; : minimax fairness) q = 0 q = ∞

  20. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w Inspired by 𝛽 -fairness for fair resource allocation in wireless networks A tunable framework ( : previous objective; : minimax fairness) q = 0 q = ∞ Theory

  21. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w Inspired by 𝛽 -fairness for fair resource allocation in wireless networks A tunable framework ( : previous objective; : minimax fairness) q = 0 q = ∞ Theory

  22. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w Inspired by 𝛽 -fairness for fair resource allocation in wireless networks A tunable framework ( : previous objective; : minimax fairness) q = 0 q = ∞ Theory Generalization guarantees Increasing results in more ‘uniform’ accuracy distributions (in q terms of various uniformity measures such as variance)

  23. Fair Resource Allocation Objective ) … q + 1 q + 1 q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N w

  24. Fair Resource Allocation Objective ) … q + 1 q + 1 q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N w # test accuracy 0.4 0.2 0.6 0.8

  25. Fair Resource Allocation Objective ) … q + 1 q + 1 q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N w Baseline # test accuracy 0.4 0.2 0.6 0.8

  26. Fair Resource Allocation Objective ) … q + 1 q + 1 q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N w Baseline q -FFL # test accuracy 0.4 0.2 0.6 0.8

  27. Fair Resource Allocation Objective ) … q + 1 q + 1 q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N w Baseline q -FFL # test accuracy 0.4 0.2 0.6 0.8

  28. Efficient Solver

  29. Efficient Solver Challenges

  30. Efficient Solver Challenges Different fairness/accuracy tradeoffs: different q’s

  31. Efficient Solver Challenges Different fairness/accuracy tradeoffs: different q’s

  32. Efficient Solver Challenges Different fairness/accuracy tradeoffs: different q’s Heterogeneous networks, expensive communication

  33. Efficient Solver Challenges Different fairness/accuracy tradeoffs: different q’s Heterogeneous networks, expensive communication

  34. Efficient Solver Challenges Different fairness/accuracy tradeoffs: different q’s Heterogeneous networks, expensive communication

  35. Efficient Solver Challenges High level ideas Different fairness/accuracy tradeoffs: different q’s Heterogeneous networks, expensive communication

  36. Efficient Solver Challenges High level ideas Different fairness/accuracy Dynamically estimate the step tradeoffs: different q’s sizes associated with different ’s q Heterogeneous networks, expensive communication

  37. Efficient Solver Challenges High level ideas Different fairness/accuracy Dynamically estimate the step tradeoffs: different q’s sizes associated with different ’s q Heterogeneous networks, expensive communication

  38. Efficient Solver Challenges High level ideas Different fairness/accuracy Dynamically estimate the step tradeoffs: different q’s sizes associated with different ’s q Heterogeneous networks, Allow for low device expensive communication participation, local updating

  39. Empirical Results Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespeare q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  40. Empirical Results Benchmark: LEAF (leaf.cmu.edu) Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespeare q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  41. Empirical Results Benchmark: LEAF (leaf.cmu.edu) Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  42. Empirical Results Benchmark: similar average LEAF (leaf.cmu.edu) accuracy Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  43. Empirical Results Benchmark: LEAF (leaf.cmu.edu) Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  44. Empirical Results Benchmark: decrease variance LEAF (leaf.cmu.edu) significantly Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  45. Empirical Results Benchmark: LEAF (leaf.cmu.edu) Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  46. Empirical Results increase the Benchmark: accuracy of the LEAF (leaf.cmu.edu) worst 10% devices Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  47. Empirical Results slightly decrease Benchmark: the accuracy of the LEAF (leaf.cmu.edu) best devices Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

Recommend

Dominant Resource Fairness: Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei

Dominant Resource Fairness: Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei

Dominant Resource Fairness: Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker, Ion Stoica UC Berkeley Introduction: Resource Allocation There is a need to design fair,

604 views • 22 slides
D Dominant Resource Fairness (DRF) i t R F i (DRF) Fair Allocation of Multiple Resource Types

D Dominant Resource Fairness (DRF) i t R F i (DRF) Fair Allocation of Multiple Resource Types

D Dominant Resource Fairness (DRF) i t R F i (DRF) Fair Allocation of Multiple Resource Types Ali Ghodsi , Matei Zaharia Benjamin Hindman, Andy Konwinski, B j i Hi d A d K i ki Scott Shenker, Ion Stoica University of California, Berkeley

781 views • 51 slides
Dominant Resource Fairness : Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei

Dominant Resource Fairness : Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei

Dominant Resource Fairness : Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker, Ion Stoica University of California, Berkeley Talk by: Yael Amsterdamer Introduction Why

800 views • 22 slides
Recent Advances in Fair Resource Allocation Rupert Freeman and Nisarg Shah Microsoft Research

Recent Advances in Fair Resource Allocation Rupert Freeman and Nisarg Shah Microsoft Research

Recent Advances in Fair Resource Allocation Rupert Freeman and Nisarg Shah Microsoft Research New York University of Toronto EC'19, AAAI'20 and AAMAS'20 Tutorial on Recent Advances in Fair Resource Allocation Rupert Freeman and Nisarg

2.07k views • 160 slides
Max-Min Fair Resource Allocation for Multiuser Amplify-and-Forward Relay Networks Alireza

Max-Min Fair Resource Allocation for Multiuser Amplify-and-Forward Relay Networks Alireza

Max-Min Fair Resource Allocation for Multiuser Amplify-and-Forward Relay Networks Alireza Sharifian*, Petar Djukic*, Halim Yanikomeroglu*, and Jietao Zhang *Department of Computer and System engineering Carleton University Huawei Wireless

466 views • 21 slides
Fair and Sustainable Resource Allocation in Self-Organising Multi-Agent Systems Jeremy Pitt

Fair and Sustainable Resource Allocation in Self-Organising Multi-Agent Systems Jeremy Pitt

Fair and Sustainable Resource Allocation in Self-Organising Multi-Agent Systems Jeremy Pitt Department of Electrical and Electronic Engineering Journ ees Francophones sur les Syst` emes Multi-Agents (JFSMA) Plate-forme Intelligence

619 views • 22 slides
Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q

Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q

Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q Federated learning: FedSGD and FedAvg q Related research in federated learning q Open problems Outline q Preliminary: deep learning and SGD q Federated

616 views • 50 slides
FAT in Resource Allocation: Equilibrium to the Rescue Noam Nisan Hebrew U. Computer Science

FAT in Resource Allocation: Equilibrium to the Rescue Noam Nisan Hebrew U. Computer Science

FAT in Resource Allocation: Equilibrium to the Rescue Noam Nisan Hebrew U. Computer Science Moshe Babaioff Microsoft Research Inbal Talgam-Cohen Technion Computer Science (+ LL.B.) (Based on our paper Fair Allocation through

664 views • 30 slides
Section 6.1: Resource Allocation Issues Chapter 6: Congestion Control and Resource Allocation

Section 6.1: Resource Allocation Issues Chapter 6: Congestion Control and Resource Allocation

Section 6.1: Resource Allocation Issues Chapter 6: Congestion Control and Resource Allocation CS/ECPE 5516: Comm. Network Prof. Abrams Spring 2000 CS/ECPE5516 1 2 CS/ECPE5516 Whats wrong with doing How to prevent traffic jams

311 views • 6 slides
Machine Learning as Enabler for Cross-Layer Resource Allocation: Opportunities and Challenges

Machine Learning as Enabler for Cross-Layer Resource Allocation: Opportunities and Challenges

| 1 Machine Learning as Enabler for Cross-Layer Resource Allocation: Opportunities and Challenges with Deep Reinforcement Learning Fatemeh Shah-Mohammadi and Andres Kwasinski Rochester Institute of Technology | 2 Outline Benefits for

524 views • 18 slides
Analyzing Federated Learning through an Adversarial Lens Arjun Nitin Bhagoji 1 , Supriyo

Analyzing Federated Learning through an Adversarial Lens Arjun Nitin Bhagoji 1 , Supriyo

Analyzing Federated Learning through an Adversarial Lens Arjun Nitin Bhagoji 1 , Supriyo Chakraborty 2 , Prateek Mittal 1 and Seraphin Calo 2 1 Princeton University 2 IBM Research ICML 2019 Federated learning (with a malicious agent) Federated

1.46k views • 59 slides
On Self-adaptive Resource Allocation through Reinforcement Learning Jacopo Panerati , Filippo

On Self-adaptive Resource Allocation through Reinforcement Learning Jacopo Panerati , Filippo

On Self-adaptive Resource Allocation through Reinforcement Learning Jacopo Panerati , Filippo Sironi , Matteo Carminati , Martina Maggio , Giovanni Beltrame , Piotr J. Gmytrasiewicz , Donatella Sciuto and Marco D.

1.33k views • 109 slides
Federated Machine Learning via Over-the-Air Computation Yuanming Shi ShanghaiTech University 1

Federated Machine Learning via Over-the-Air Computation Yuanming Shi ShanghaiTech University 1

Federated Machine Learning via Over-the-Air Computation Yuanming Shi ShanghaiTech University 1 Outline Motivations Big data, IoT,AI Three vignettes: Federated machine learning Federated model aggregation Over-the-air

1.02k views • 56 slides
Resource Allocation Task Force Resource Allocation Task Force Gigi Karmous Edwards Gigi

Resource Allocation Task Force Resource Allocation Task Force Gigi Karmous Edwards Gigi

Resource Allocation Task Force Resource Allocation Task Force Gigi Karmous Edwards Gigi ke@ncsu.edu Gigi_ke@ncsu.edu GLIF Winter Meeting, Hong Kong 2011 Purpose of R Resource Allocation Task Force All i T k F What does the user want

342 views • 7 slides
Resource Allocation and Deadlock Resource Allocation and Deadlock Handling Conditions for

Resource Allocation and Deadlock Resource Allocation and Deadlock Handling Conditions for

Resource Allocation and Deadlock Resource Allocation and Deadlock Handling Conditions for Deadlock [Coffman-etal 1971] 4 conditions must hold simultaneously for a deadlock to occur: deadlock to occur: Mutual exclusion: only one process at

520 views • 31 slides
Fair Allocation of Indivisible Goods Ulle Endriss Institute for Logic, Language and Computation

Fair Allocation of Indivisible Goods Ulle Endriss Institute for Logic, Language and Computation

Fair Allocation of Indivisible Goods T ubingen, 7 April 2016 Fair Allocation of Indivisible Goods Ulle Endriss Institute for Logic, Language and Computation University of Amsterdam Ulle Endriss 1 Fair Allocation of Indivisible Goods T

568 views • 20 slides
Adversarial Risk Analysis Models for Urban Security Resource Allocation Urban Security Resource

Adversarial Risk Analysis Models for Urban Security Resource Allocation Urban Security Resource

Adversarial Risk Analysis Models for Urban Security Resource Allocation Urban Security Resource Allocation David Ros Insua, Royal Academy Cesar Gil, U. Rey Juan Carlos Jess Ros, IBM Research YH COST Smart Cities Wshop Paris, September

507 views • 22 slides
CS473 Federated Text Search Luo Si Department of Computer Science Purdue University Abstract

CS473 Federated Text Search Luo Si Department of Computer Science Purdue University Abstract

CS473 Federated Text Search Luo Si Department of Computer Science Purdue University Abstract Outline Introduction to federated search Main research problems Resource Representation Resource Selection Results Merging A

599 views • 47 slides
CS490W Federated Text Search Luo Si Department of Computer Science Purdue University Abstract

CS490W Federated Text Search Luo Si Department of Computer Science Purdue University Abstract

CS490W Federated Text Search Luo Si Department of Computer Science Purdue University Abstract Outline Introduction to federated search Main research problems Resource Representation Resource Selection Results Merging A

650 views • 47 slides
CS490W Federated Text Search Luo Si Department of Computer Science Purdue University Abstract

CS490W Federated Text Search Luo Si Department of Computer Science Purdue University Abstract

CS490W Federated Text Search Luo Si Department of Computer Science Purdue University Abstract Outline Introduction to federated search Main research problems Resource Representation Resource Selection Results Merging A

360 views • 24 slides
CS54701 Federated Text Search Luo Si Department of Computer Science Purdue University Abstract

CS54701 Federated Text Search Luo Si Department of Computer Science Purdue University Abstract

CS54701 Federated Text Search Luo Si Department of Computer Science Purdue University Abstract Outline Introduction to federated search Main research problems Resource Representation Resource Selection Results Merging

832 views • 29 slides
A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation (2006) Paper: Gerald

A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation (2006) Paper: Gerald

A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation (2006) Paper: Gerald Tesauro, Nicholas K. Jongt, Rajarshi Das and Mohamed N. Bennanit Presentation: Nat McAleese (nm583) What is autonomic computing? computing systems

288 views • 11 slides
Differentially-Private Federated Linear Bandits Introduction Federated Learning Contextual

Differentially-Private Federated Linear Bandits Introduction Federated Learning Contextual

Differentially- Private Federated Linear Bandits Dubey and Pentland June 2020 Differentially-Private Federated Linear Bandits Introduction Federated Learning Contextual Bandits Summary Background Abhimanyu Dubey and Alex Pentland

437 views • 20 slides
CSC304 Lecture 20 Fair Division 3: Leximin Allocation (computational resources, matching with

CSC304 Lecture 20 Fair Division 3: Leximin Allocation (computational resources, matching with

CSC304 Lecture 20 Fair Division 3: Leximin Allocation (computational resources, matching with dichotomous prefs, classroom allocation) Utilitarian Allocation (rent division) CSC304 - Nisarg Shah 1 Computational Resources Setting: We

682 views • 20 slides

More recommend