applied research group
play

Applied research group Systems+database people building prototypes, - PowerPoint PPT Presentation

Aug 03, 2023 •29 likes •879 views

Applied research group Systems+database people building prototypes, publishing papers Applied research group Systems+database people building prototypes, publishing papers Collaborating with Big Data product group at MS Shipping our code to

  3. Applied research group Systems+database people building prototypes, publishing papers

  4. Applied research group Systems+database people building prototypes, publishing papers Collaborating with Big Data product group at MS Shipping our code to production

  5. Applied research group Systems+database people building prototypes, publishing papers Collaborating with Big Data product group at MS Shipping our code to production Open-sourcing our code Apache Hadoop, REEF, Heron

  6. Resource Distributed Query management tiered storage optimization Stream Log analytics processing

  7. Resource Distributed Query management tiered storage optimization Stream Log analytics processing

  8. Node Node Node Manager Manager Manager

  9. • Node Node Node Manager Manager Manager

  10. • • Node Node Node Manager Manager Manager

  11. • • Node Node Node Manager Manager Manager

  12. • • 1. Request Node Node Node Manager Manager Manager

  13. • • 1. Request 2. Allocation Node Node Node Manager Manager Manager

  14. • • 1. Request 2. Allocation 3. Start task Node Node Node Manager Manager Manager

  15. • • 1. Request 2. Allocation • 3. Start task Node Node Node Manager Manager Manager

  16. • • 1. Request 2. Allocation • • 3. Start task Node Node Node Manager Manager Manager

  17. • • 1. Request Do we really need a Resource Manager? 2. Allocation • • 3. Start task Node Node Node Manager Manager Manager

  18. Hadoop 1 World Hadoop 2 World • monolithic Users Application Frameworks Hive / Pig Hive / Pig Ad-hoc Ad-hoc app Ad-hoc Apps Ad-hoc Ad-hoc Ad-hoc Scope app Programming app app MR v1 app Model(s) on MR ... YARN Tez Giraph Storm Spark Dryad v2 Heron REEF Hadoop 1.x (MapReduce) Cluster OS (Resource YARN Management) File System HDFS 2 HDFS 1 Hardware

  19. Hadoop 1 World Hadoop 2 World • monolithic Users Application Frameworks • Reuse of RM Hive / Pig Hive / Pig Ad-hoc Ad-hoc app Ad-hoc Apps Ad-hoc component Ad-hoc Ad-hoc Scope app Programming app app MR v1 app Model(s) on MR ... YARN Tez Giraph Storm Spark Dryad v2 Heron REEF Hadoop 1.x (MapReduce) Cluster OS (Resource YARN Management) File System HDFS 2 HDFS 1 Hardware

  20. Hadoop 1 World Hadoop 2 World • monolithic Users Application Frameworks • Reuse of RM Hive / Pig Hive / Pig Ad-hoc Ad-hoc app Ad-hoc Apps Ad-hoc component Ad-hoc Ad-hoc Scope app Programming app app MR v1 app Model(s) on MR ... YARN Tez Giraph Storm Spark Dryad v2 Heron REEF Hadoop 1.x (MapReduce) YARN Cluster OS • (Resource YARN Management) layering abstractions File System HDFS 2 HDFS 1 Hardware

  22. But is all this good enough for the Microsoft clusters?

  23. High resource Scalability utilization Production jobs Workload and heterogeneity predictability

  24. 100% Utilization

  26. 0

  27. • Wide variety

  28. • Wide variety

  29. • Wide variety • •

  30. deadlines recurring >60% • Predictability over-provisioned

  31. 4 Hadoop committers in CISL 404 patches as of last night • Rayon/Morpheus: • Mercury/Yaq: • YARN Federation: • Medea:

  32. 4 Hadoop committers in CISL 404 patches as of last night • Rayon/Morpheus: • Mercury/Yaq: • YARN Federation: • Medea:

  33. [Hadoop 3.0; ATC 2015, EuroSys 2016]

  34. RM N1 N2

  35. j1 RM N1 N2

  36. j1 RM N1 N2

  37. j2 RM N1 N2

  38. j2 RM N1 N2

  39. j2 RM N1 N2

  40. j2 RM N1 N2

  41. j2 RM N1 N2

  42. j2 RM • Feedback delays idle between allocations N1 N2

  43. j2 RM • Feedback delays idle between allocations N1 N2 5 sec 10 sec 50 sec Mixed-5-50 Cosmos-gm 60.59% 78.35% 92.38% 78.54% 83.38%

  44. j2 RM • Feedback delays idle between allocations N1 N2 5 sec 10 sec 50 sec Mixed-5-50 Cosmos-gm 60.59% 78.35% 92.38% 78.54% 83.38% • Actual

  45. • Introduce task queuing at nodes • Mask feedback delays • Improve cluster utilization • Improve task throughput (by up to 40%) • Container types • GUARANTEED and OPPORTUNISTIC • Keep guarantees for important jobs • Use opportunistic execution to improve utilization

  46. RM N1 N2

  47. RM N1 N2

  48. j1 RM N1 N2

  49. j1 RM N1 N2

  50. j2 RM N1 N2

  51. j2 RM N1 N2

  52. j2 RM N1 N2

  53. j2 RM N1 N2

  54. • j2 RM N1 N2

  55. • j2 RM • N1 N2

  56. • j2 RM • N1 N2

  58. •

  59. • •

  60. • So all we need to do is use long queues? •

  62. can be detrimental for job completion times • Despite the utilization gains

  63. can be detrimental for job completion times • Despite the utilization gains Proper queue management techniques are required

  64. N1 N2 N3

  65. N1 N2 N3

  66. N1 N2 N3

  67. N1 N2 N3

  68. Prioritize task Place tasks to execution node queues (queue reordering) Bound queue lengths

  69. Prioritize task Place tasks to execution node queues (queue reordering) Bound queue lengths Yaq improves median job completion time by 1.7x over YARN

  70. RM N1 N2 N3

  71. queue length RM N1 N2 N3

  72. queue length RM N1 N2 N3

  73. queue length RM N1 N2 N3

  74. queue length RM queue wait time N1 N2 N3

  75. queue length RM queue wait time N1 N2 N3

  76. • Shortest Remaining Job First (SRJF) • Least Remaining Tasks First (LRTF)

  77. RM j2: 5 tasks j3: 9 tasks j1: 21 tasks • Shortest Remaining Job First (SRJF) • Least Remaining Tasks First (LRTF) N1 N2 N3

  78. RM j2: 5 tasks j3: 9 tasks j1: 21 tasks • Shortest Remaining Job First (SRJF) • Least Remaining Tasks First (LRTF) N1 N2 N3

  79. RM j2: 5 tasks j3: 9 tasks j1: 21 tasks • Shortest Remaining Job First (SRJF) • Least Remaining Tasks First (LRTF) N1 N2 N3

  80. RM j2: 5 tasks j3: 9 tasks j1: 21 tasks • Shortest Remaining Job First (SRJF) • Least Remaining Tasks First (LRTF) N1 N2 N3 job-aware

  81. lower throughput longer job completion times

  82. • 1.7x improvement in median JCT over YARN

  83. • Container types distributed scheduling any distributed scheduler over-commitment multi-tenancy • Pricing

  84. cluster utilization queue management techniques job completion time

Recommend

The Applied Research Institute Overview The University of Virginia Applied Research Institute

The Applied Research Institute Overview The University of Virginia Applied Research Institute

The Applied Research Institute Overview The University of Virginia Applied Research Institute (ARI) was created to enlarge the scope of projects and opportunities available to faculty and students. ARI leverages the Universitys human and

405 views • 11 slides
work? Dr Richard Fitton (ABERG Salford) Applied Buildings and Energy Research Group Why are we

work? Dr Richard Fitton (ABERG Salford) Applied Buildings and Energy Research Group Why are we

Energy Savings Measures, do they work? Dr Richard Fitton (ABERG Salford) Applied Buildings and Energy Research Group Why are we researching retrofit (UK)? Solid wall properties such as those represented by the Energy house currently number

710 views • 53 slides
Agenda Introduction to ARP ARP functionality Proxy ARP RARP Applied

Agenda Introduction to ARP ARP functionality Proxy ARP RARP Applied

1/13 ARP: Address Resolution Protocol Surasak Sanguanpong nguan@ku.ac.th http://www.cpe.ku.ac.th/~nguan Last updated: July 3, 2002 Applied Network Research Group Department of Computer Engineering, Kasetsart

510 views • 7 slides
ASURE an applied-research organization affiliated with the nation s fastest growing research

ASURE an applied-research organization affiliated with the nation s fastest growing research

The Aerospace & Defense Forum Arizona Chapter May 12, 2015 ASU RESEARCH ENTERPRISE Introductory Briefing Security | Innovation | Service We conceive | We develop | We implement | We deliver ASURE ASURE an applied-research

488 views • 16 slides
Big Data for Health Policy in Early Childhood: The TARGet Kids! Network The Applied Research

Big Data for Health Policy in Early Childhood: The TARGet Kids! Network The Applied Research

Big Data for Health Policy in Early Childhood: The TARGet Kids! Network The Applied Research Group for Kids! Cory Borkhoff PhD Research Associate, Department of Pediatric Medicine, The Hospital for Sick Children Assistant Professor, iHPME,

622 views • 9 slides
Archiving Computational Research in Virtual Machines Sorin Mitran Applied Mathematics

Archiving Computational Research in Virtual Machines Sorin Mitran Applied Mathematics

UNCvm Prologue Teaching Research Epilogue Archiving Computational Research in Virtual Machines Sorin Mitran Applied Mathematics University of North Carolina Chapel Hill Applied Mathematics Perspectives Reproducible Research Workshop

812 views • 55 slides
The Evolution of Research on Doctoral Student Success at UM Higher Education Applied Research

The Evolution of Research on Doctoral Student Success at UM Higher Education Applied Research

The Evolution of Research on Doctoral Student Success at UM Higher Education Applied Research (HEAR) seminar November 28, 2017 John A. Gonzalez, Ph.D. Director of Institutional Research Merle Feldbaum Research Area Specialist Senior Allyson

329 views • 16 slides
Riskcenter is a research group that specializes in the field of risk management , insurance and

Riskcenter is a research group that specializes in the field of risk management , insurance and

Riskcenter is a research group that specializes in the field of risk management , insurance and finance based at the University of Barcelona . The research group on Risk in Insurance and Finance is attached to the Institute of Applied Economics

358 views • 10 slides
applied in Egypt? P. Michael Link 1,2,3 Jrgen Scheffran 1,2 1 University of Hamburg, Research

applied in Egypt? P. Michael Link 1,2,3 Jrgen Scheffran 1,2 1 University of Hamburg, Research

Adaptation to sea level rise in the European Wadden Sea: can promising approaches be applied in Egypt? P. Michael Link 1,2,3 Jrgen Scheffran 1,2 1 University of Hamburg, Research Group Climate Change and Security 2 University of Hamburg,

462 views • 29 slides
Development of a Common Research Model for Applied CFD Validation Studies John Vassberg &

Development of a Common Research Model for Applied CFD Validation Studies John Vassberg &

BOEING PROPRIETARY Development of a Common Research Model for Applied CFD Validation Studies John Vassberg & Mark DeHaan The Boeing Company Phantom Works Melissa Rivers & Rich Wahls NASA Langley Research Center AIAA Applied

690 views • 20 slides
Applied Geology APPLIED GEOLOGY SUBJECT AREA GROUP Tuning Africa Phase II Third General Meeting

Applied Geology APPLIED GEOLOGY SUBJECT AREA GROUP Tuning Africa Phase II Third General Meeting

Applied Geology APPLIED GEOLOGY SUBJECT AREA GROUP Tuning Africa Phase II Third General Meeting Accra, Ghana, 17th to 19th October, 2016 By: Dr. Digne Edmond Rwabuhungu Rwatangabo Group Coordinator Coordinator: Digne Edmond RWABUHUNGU R.

198 views • 16 slides
Applied Physics Day 2017 Welcome! Department of Applied Physics Present AlbaNova Electrum

Applied Physics Day 2017 Welcome! Department of Applied Physics Present AlbaNova Electrum

Applied Physics Day 2017 Welcome! Department of Applied Physics Present AlbaNova Electrum SciLifeLab Applied Physics: 185 people 88 % Research 12 % Basic edu 220 MSEK turnover KTH Kista KTH Main campus KTH Flemingsberg KTH

344 views • 12 slides
Access to EU Research Funding for Universities of Applied Sciences 19-20 November, Spanish

Access to EU Research Funding for Universities of Applied Sciences 19-20 November, Spanish

Welcome to the international AESIS course on Access to EU Research Funding for Universities of Applied Sciences 19-20 November, Spanish National Research Council, Brussels, Belgium Access to EU Research Funding for Universities of Applied

1.04k views • 88 slides
Center for Applied Energy Research at University of Kentucky www.caer.uky.edu Catalysis Research

Center for Applied Energy Research at University of Kentucky www.caer.uky.edu Catalysis Research

Center for Applied Energy Research at University of Kentucky www.caer.uky.edu Catalysis Research and Testing Center Research and Testing Center 2540 Research Park Dr. Lexington, KY 45011 Anderson-Schulz-Flory Product Distribution Can it

576 views • 39 slides
Cornell University Program on Applied Demographics Applied Demographics CaRDI Spring Research

Cornell University Program on Applied Demographics Applied Demographics CaRDI Spring Research

Cornell University Program on Applied Demographics Applied Demographics CaRDI Spring Research Roundtable April 6, 2009 April 6 2009 Program on Applied Demographics Web: http://pad.human.cornell.edu Email: PADInfo@Cornell.edu Mission

336 views • 18 slides
The$Canadian$Centre$for$Applied$ Research$in$Cancer$Control$(ARCC)$

The$Canadian$Centre$for$Applied$ Research$in$Cancer$Control$(ARCC)$

The$Canadian$Centre$for$Applied$ Research$in$Cancer$Control$(ARCC)$ Dr.$Stuart$Peacock,$$Dr.$Jeffrey$Hoch,$Kimberly$van$der$Hoek$and$Sarah$Benn ! ! ARCC!Research! 5$themaGc$research$program$areas$to$address$cancer$control$ I

194 views • 3 slides
Process Dr. Michael Yukish Simon Miller Research Associate Doctoral Candidate Applied Research

Process Dr. Michael Yukish Simon Miller Research Associate Doctoral Candidate Applied Research

ARL Design as a Sequential Decision Process Dr. Michael Yukish Simon Miller Research Associate Doctoral Candidate Applied Research Laboratory at Applied Research Laboratory at Penn State University Penn State University may106@arl.psu.edu

498 views • 8 slides
What is a group and why should I care? Daniel Platt October 10, 2019 What is a Group? Very

What is a group and why should I care? Daniel Platt October 10, 2019 What is a Group? Very

What is a group and why should I care? Daniel Platt October 10, 2019 What is a Group? Very general mathematical concept, can be applied to: Rubiks Cube What is a Group? Very general mathematical concept, can be applied to: Symmetry Group

840 views • 70 slides
Applied research and teaching opportunities between the EU and China Dr Ana Mateus, RVC LinkTADs

Applied research and teaching opportunities between the EU and China Dr Ana Mateus, RVC LinkTADs

Applied research and teaching opportunities between the EU and China Dr Ana Mateus, RVC LinkTADs session at EPIZONE: Coordinating veterinary research between EU and China 29th September, 2016, Madrid, Spain This project has received funding

703 views • 20 slides
and DAP: Do they all play together? Kyle Snow, Ph.D. Director, Center for Applied Research

and DAP: Do they all play together? Kyle Snow, Ph.D. Director, Center for Applied Research

Quality Ratings, Common Core, and DAP: Do they all play together? Kyle Snow, Ph.D. Director, Center for Applied Research RESEARCH POLICY PRACTICE 1 Overview POLL 2 Overview 1. Early Childhood Education Now is the time 2. Making sense

685 views • 32 slides
Session: Innovation and research needs Topic: Future topics of applied FOWT research Kolja Mller

Session: Innovation and research needs Topic: Future topics of applied FOWT research Kolja Mller

LIFES50+ Final Event Bilbao, 03 April 2019 Session: Innovation and research needs Topic: Future topics of applied FOWT research Kolja Mller 1 , Kimon Argyriadis 2 1 University of Stuttgart, 2 DNV GL Qualification of innovative floating

396 views • 14 slides
Budapest University of Technology and Economics Department of Automation and Applied Informatics

Budapest University of Technology and Economics Department of Automation and Applied Informatics

Budapest University of Technology and Economics Department of Automation and Applied Informatics Visual Modeling Languages Research Group Institute for Software-Integrated Systems, Vanderbilt University Introduction SE on a higher

859 views • 49 slides
HHS Research: Advancing Basic and Applied Science for the Greater Good Overview What is

HHS Research: Advancing Basic and Applied Science for the Greater Good Overview What is

HHS Research: Advancing Basic and Applied Science for the Greater Good Overview What is health and human sciences? What kinds of research? Some concrete examples Whose greater good? HHS research facts Strategies to support

365 views • 23 slides
Non-Market Strategies and Asset Specific Investments Conference on Applied Infrastructure Research

Non-Market Strategies and Asset Specific Investments Conference on Applied Infrastructure Research

Non-Market Strategies and Asset Specific Investments Conference on Applied Infrastructure Research INFRADAY 2008 Fabian Kirsch With Johannes Fuhr Technische Universitt Berlin Conference on Applied Infrastructure Research INFRADAY 2008, Oct.

186 views • 15 slides

More recommend