how i learned to
play

How I Learned to Rohit Zambre,* Stop Aparna - PowerPoint PPT Presentation

Jan 09, 2024 •624 likes •1.16k views

How I Learned to Rohit Zambre,* Stop Aparna Chandramowlishwaran,* Worrying Pavan Balaji About *University of California, Irvine User-Visible Argonne National Laboratory Endpoints and Love MPI 2 HOW I LEARNED TO STOP WORRYING

  1. How I Learned to Rohit Zambre,* Stop Aparna Chandramowlishwaran,* Worrying Pavan Balaji ⌃ About *University of California, Irvine User-Visible ⌃ Argonne National Laboratory Endpoints and Love MPI

  2. 2 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI everywhere Node Core Process

  3. 3 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI everywhere ▸ Model artifact : high memory requirements that worsen with increase domain- dimensionality and number of ranks. ▸ Hardware usage : resource wastage with static split of limited resources on processor Node Core Process Increasing number of cores Decreasing memory per core

  4. 4 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI+threads ▸ Model artifact : reduces duplicated data by a factor of number of threads. ▸ Hardware usage : able to use the many cores while sharing all of processor’s resources. Node Core Process Thread Increasing number of cores Decreasing memory per core

  5. 5 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI 7 Category Computation 6 Corresponding Allgatherv AlltoAllv MPI everywhere 5 Time (seconds) MPI+threads 4 OOM! 3 runs 2 1 0 1x440x110 1x220x220 1x110x440 6x180x45 6x90x90 6x45x180 Processor Grid (threads x processor rows x processor columns) Buluc et al. Distributed BFS (https://arxiv.org/abs/1705.04590)

  6. 6 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI 7 Category Computation 6 Corresponding Allgatherv AlltoAllv MPI everywhere 5 Time (seconds) MPI+threads 4 OOM! 3 runs 2 1 0 1x440x110 1x220x220 1x110x440 6x180x45 6x90x90 6x45x180 Processor Grid (threads x processor rows x processor columns) Buluc et al. Distributed BFS (https://arxiv.org/abs/1705.04590) MPI_Isend (8 B) 60 Communication Million Messages/s performance of 40 Model MPI everywhere MPI+threads is MPI+threads (MPI_THREAD_FUNNELED) MPI+threads (MPI_THREAD_MULTIPLE) 20 dismal 0 1 2 4 8 16 Number of cores

  7. 7 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI Node Outdated view : Network is a single device Modern reality : Network features parallelism Network Interface Card Network Interface Card Network hardware context

  8. 8 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI everywhere MPI+threads P0 P1 P2 P3 P0 Application MPI library Network Interface Card Network Interface Card Software communication channel Network hardware context

  9. 9 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI everywhere MPI+threads P0 P1 P2 P3 P0 Application MPI library Network Interface Card Network Interface Card Global critical section + 1 communication channel per process Software communication channel Network hardware context

  10. 10 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI No logical MPI everywhere MPI+threads parallelism expressed P0 P1 P2 P3 P0 Application MPI library Network Interface Card Network Interface Card Global critical section + 1 communication channel per process Software communication channel Network hardware context

  11. 11 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI_Comm_create_endpoints(…,num_ep,…,comm_eps[]); MPI_Isend/Irecv(…,comm_eps[tid],ep_rank,…); MPI process EP0 EP2 EP3 EP4 MPI library Network Interface Card MPI Communicator MPI Endpoint Software communication channel Network hardware context

  12. 12 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI_Comm_create_endpoints(…,num_ep,…,comm_eps[]); MPI_Isend/Irecv(…,comm_eps[tid],ep_rank,…); MPI process Pros ▸ Explicit control over network contexts EP0 EP2 EP3 EP4 Cons MPI library ▸ Intrusive extension of the MPI standard ▸ Onus of managing network contexts on user Network Interface Card MPI Communicator MPI Endpoint Software communication channel Network hardware context

  13. 13 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI Logical parallelism MPI everywhere MPI+threads expressed P0 P0 P1 P2 P3 Application C0 C1 C2 C3 MPI library Network Interface Card Network Interface Card Fine-grained critical sections + Multiple communication channel per process MPI Communicator Software communication channel Network hardware context

  14. 14 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI Do we need user-visible endpoints? Logical parallelism MPI everywhere MPI+threads expressed P0 P0 P1 P2 P3 Application C0 C1 C2 C3 MPI library Network Interface Card Network Interface Card Fine-grained critical sections + Multiple communication channel per process MPI Communicator Software communication channel Network hardware context

  15. 15 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI CONTRIBUTIONS AS DEVIL’S ADVOCATE ▸ In-depth comparison between MPI-3.1 and user-visible endpoints ▸ A fast MPI+threads library that adheres to MPI-3.1’s constraints ▸ Optimized parallel communication streams applicable to all MPI libraries ▸ Recommendations for the MPI user to express logical parallelism with MPI-3.1 MPI library Interconnects Evaluation ▸ Based on ▸ Intel Omni-Path (OPA) with OFI:PSM2 platforms MPICH:CH4 ▸ Mellanox InfiniBand (IB) with UCX:Verbs

  16. 16 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI OUTLINE ▸ Introduction ▸ For MPI users: Parallelism in the MPI standard ▸ For MPI developers: Fast MPI+threads ▸ Fine-grained critical sections for thread safety ▸ Virtual Communication Interfaces (VCIs) for parallel communication streams ▸ Microbenchmark and Application analysis

  17. 17 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI POINT-TO-POINT COMMUNICATION ▸ <comm,rank,tag> decides matching ▸ Non-overtaking order ▸ Receive wildcards Two or more operations on a Can be issued on parallel process with communication streams? Comm Rank Tag Send Recv

  18. 18 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI POINT-TO-POINT COMMUNICATION ▸ <comm,rank,tag> decides matching Rank 0 (sender) ▸ Non-overtaking order ▸ Receive wildcards Two or more operations on a Can be issued on parallel <CA,R1,T1> <CB,R1,T1> process with communication streams? Comm Rank Tag Send Recv <CA,R0,T1> <CB,R0,T1> Different or Different or Different Yes Yes Same Same Rank 1 (receiver)

  19. 19 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI POINT-TO-POINT COMMUNICATION ▸ <comm,rank,tag> decides matching Rank 0 (sender) ▸ Non-overtaking order ▸ Receive wildcards Two or more operations on a Can be issued on parallel <CA,R1,T1> <CA,R2,T1> process with communication streams? Comm Rank Tag Send Recv <CA,R0,T1> <CA,ANY,T1> Different or Different or Different Yes Yes Same Same Different or Same Different Yes No Rank 1 (receiver) Same Wildcards

  20. 20 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI POINT-TO-POINT COMMUNICATION Rank 0 (sender) ▸ <comm,rank,tag> decides matching ▸ Non-overtaking order ▸ Receive wildcards <CA,R1,T1> Two or more operations on a Can be issued on parallel process with communication streams? <CA,R1,T2> Comm Rank Tag Send Recv Different or Different or Different Yes Yes <CA,R0,T3> <CA,R0,ANY> Same Same Different or Same Different Yes No Same Wildcards Different or Rank 1 (receiver) Same Same No No Same Non-overtaking order

  21. 21 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI RMA COMMUNICATION Two or more operations Can be issued on parallel communication streams? on a process with Window Rank Put Get Accumulate Different or Different Yes Yes Yes Same Same Different Yes Yes Yes Same Same Yes Yes No

  22. 22 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI RMA COMMUNICATION Two or more operations Can be issued on parallel communication streams? on a process with Explicitly expressing Window Rank Put Get Accumulate parallelism Different or Different Yes Yes Yes Same Same Different Yes Yes Yes Implicit parallelism Same Same Yes Yes No No order between multiple Gets and Puts

  23. 23 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI RMA COMMUNICATION Two or more operations Can be issued on parallel communication streams? on a process with Explicitly expressing Window Rank Put Get Accumulate parallelism Different or Different Yes Yes Yes Same Same Different Yes Yes Yes Implicit parallelism Same Same Yes Yes No Ordering of accumulate No order between operations to the same multiple Gets and Puts memory location

  24. 24 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI OUTLINE ▸ Introduction ▸ For MPI users: Parallelism in the MPI standard ▸ For MPI developers: Fast MPI+threads ▸ Fine-grained critical sections for thread safety ▸ Virtual Communication Interfaces (VCIs) for parallel communication streams ▸ Microbenchmark and Application analysis

Recommend

Lessons Learned Lessons Learned From From Lessons Learned Lessons Learned From From

Lessons Learned Lessons Learned From From Lessons Learned Lessons Learned From From

Protg 2006, July 26th, Stanford I R I S A C 1 R E C Saint-Cyr Lessons Learned Lessons Learned From From Lessons Learned Lessons Learned From From Ontology Ontology Design Design Ontology Ontology Design Design Jean Andr B

371 views • 12 slides
Wh t h What have we learned from What have we learned from Wh t h l l d f d f building the

Wh t h What have we learned from What have we learned from Wh t h l l d f d f building the

Wh t h What have we learned from What have we learned from Wh t h l l d f d f building the LHC (CMS) DAQ building the LHC (CMS) DAQ systems. systems. S. Cittolin PH-CMD. CERN Openlab meeting. 3-4 March 2009 DAQ at LHC overview CMS

352 views • 33 slides
4 5 6 CSE 142 vs CSE 143 CSE 142 / AP CS A CSE 143 You learned how to write Return of

4 5 6 CSE 142 vs CSE 143 CSE 142 / AP CS A CSE 143 You learned how to write Return of

4 5 6 CSE 142 vs CSE 143 CSE 142 / AP CS A CSE 143 You learned how to write Return of the objects programs and You learned to solve decompose large more complex tasks problems with: efficiently Print statements Data

705 views • 22 slides
WHAT HAVE WHAT HAVE I LEARNED I LEARNED AT THE AT THE DISC DISC SUMMER SCHOOL? SUMMER

WHAT HAVE WHAT HAVE I LEARNED I LEARNED AT THE AT THE DISC DISC SUMMER SCHOOL? SUMMER

WHAT HAVE WHAT HAVE I LEARNED I LEARNED AT THE AT THE DISC DISC SUMMER SCHOOL? SUMMER SCHOOL? Christos G. Cassandras CODES Lab. - Boston University WHY HYBRID SYSTEMS ? COMPLEXITY ( PHYSICAL , BUT MOSTLY OPERATIONAL/ORGANIZATIONAL )

691 views • 11 slides
What We Have Learned from the Pandemic NURSING SIMULATION: What We Have Learned from the

What We Have Learned from the Pandemic NURSING SIMULATION: What We Have Learned from the

July 17, 2020 Nursing Simulation: What We Have Learned from the Pandemic NURSING SIMULATION: What We Have Learned from the Pandemic PAMELA R. JEFFRIES CRYSTEL L. FARINA PAUL LUND PAMELA SLAVEN-LEE George Washington University George

790 views • 21 slides
Object-Oriented Design What Do We Mean by OO Design? Remember how we learned about functions?

Object-Oriented Design What Do We Mean by OO Design? Remember how we learned about functions?

Module 21 Object-Oriented Design What Do We Mean by OO Design? Remember how we learned about functions? First learned to call functions made for us Then learned to define our functions Finally learned to properly design functions

471 views • 30 slides
has learned Col Rakesh Jetly OMM,CD,MD,FRCPC. Lots of Ground to Cover!! Why am I here?

has learned Col Rakesh Jetly OMM,CD,MD,FRCPC. Lots of Ground to Cover!! Why am I here?

Mental Health In the Workplace: What CAF has learned Col Rakesh Jetly OMM,CD,MD,FRCPC. Lots of Ground to Cover!! Why am I here? Lessons learned from Military medicine CAF in Kandahar MH Lessons Learned MH in The

1.18k views • 80 slides
Electrical Team Marina Beshai, Leslie Kim, and Sara Sacks Overview of the Semester: Learned

Electrical Team Marina Beshai, Leslie Kim, and Sara Sacks Overview of the Semester: Learned

Electrical Team Marina Beshai, Leslie Kim, and Sara Sacks Overview of the Semester: Learned how to test the point system. Learned how to use the oscilloscope. Started the portable power generator. Solder: Learned how to

252 views • 14 slides
What have we learned after 10 years and 120,000 cryoablations? Richard Schilling conflicts -

What have we learned after 10 years and 120,000 cryoablations? Richard Schilling conflicts -

What have we learned after 10 years and 120,000 cryoablations? Richard Schilling conflicts - speaker fees and research grants Medtronic, Biosense Webster Talk outline Brief procedure description What have I learned anecdote, personal

575 views • 23 slides
Institutionalizing Lessons Learned October 25, 2006 Loren Plisco Region II Background

Institutionalizing Lessons Learned October 25, 2006 Loren Plisco Region II Background

Institutionalizing Lessons Learned October 25, 2006 Loren Plisco Region II Background SEP 2002 Davis-Besse Lessons Learned Task Force AUG 2004 - Effectiveness Review Lessons Learned Task Force JAN 2005 - EDO Charters

269 views • 22 slides
OSHA Lessons Learned Adam Fries OSHA Compliance Officer February 13, 2018 OSHA Lessons Learned

OSHA Lessons Learned Adam Fries OSHA Compliance Officer February 13, 2018 OSHA Lessons Learned

OSHA Lessons Learned Adam Fries OSHA Compliance Officer February 13, 2018 OSHA Lessons Learned Jobsite Safety and Health Inspections/Audits better known as; PAPER WORK You hate it, Your safety director requires it, OSHA loves it, Does it

926 views • 35 slides
3/8/2019 Epidemiology, Risk Factors, and Outcomes of Pediatric PVD: LESSONS learned from the

3/8/2019 Epidemiology, Risk Factors, and Outcomes of Pediatric PVD: LESSONS learned from the

3/8/2019 Epidemiology, Risk Factors, and Outcomes of Pediatric PVD: LESSONS learned from the Spanish Registry: Lessons Learned from the Registries o Is It possible (and worthy) to do a national registry? Lessons learned from the o

1.16k views • 43 slides
DEBUGGING LESSONS LEARNED WHILE DEBUGGING LESSONS LEARNED WHILE FIXING NETBSD FIXING NETBSD

DEBUGGING LESSONS LEARNED WHILE DEBUGGING LESSONS LEARNED WHILE FIXING NETBSD FIXING NETBSD

DEBUGGING LESSONS LEARNED WHILE DEBUGGING LESSONS LEARNED WHILE FIXING NETBSD FIXING NETBSD ABOUT ME ABOUT ME maya@NetBSD.org coypu@sdf.org NetBSD/pkgsrc for the last 3 years THIS TALK THIS TALK Mix of a bunch of bugs Not solo work

695 views • 31 slides
Lessons Learned A Value Added Product of the Project Life Cycle R Gilman April 19, 2006 Agenda

Lessons Learned A Value Added Product of the Project Life Cycle R Gilman April 19, 2006 Agenda

Lessons Learned A Value Added Product of the Project Life Cycle R Gilman April 19, 2006 Agenda Introduction What are Lessons Learned? Value to the Project Process and Organization Keanes Approach to Lessons Learned Keys to

261 views • 24 slides
LEARNED SO FAR C A R M E N &amp; N I C O L E &amp; A N I TA D D C I N T E R N S 2 0 1 7

LEARNED SO FAR C A R M E N &amp; N I C O L E &amp; A N I TA D D C I N T E R N S 2 0 1 7

WHAT WEVE LEARNED SO FAR C A R M E N &amp; N I C O L E &amp; A N I TA D D C I N T E R N S 2 0 1 7 THE HISTORY OF DISABILITY Di Disability sability Hi Hist story ory st start rted ed in n 40 400 0 B.C. Disability

439 views • 15 slides
{ We are at the end of our first cycle of Learning Outcomes Assessment What has COA learned:

{ We are at the end of our first cycle of Learning Outcomes Assessment What has COA learned:

{ We are at the end of our first cycle of Learning Outcomes Assessment What has COA learned: Weve learned that Learning Assessment is not going away. Weve learned that Learning Assessment can provide valuable information.

313 views • 15 slides
The Learned Intermediary Doctrine: A Retrospective Review to the Present Jennifer A. Guidea,

The Learned Intermediary Doctrine: A Retrospective Review to the Present Jennifer A. Guidea,

The Learned Intermediary Doctrine: A Retrospective Review to the Present Jennifer A. Guidea, Esq. Shana E. Russo, Esq. Reed Smith Life Sciences Industry Group 1 Background and General Application of the Learned Intermediary Doctrine 2

519 views • 50 slides
Four Major Events in Our History and the Lessons Learned Coleen Reiswig, Isobel McDonald, Ted

Four Major Events in Our History and the Lessons Learned Coleen Reiswig, Isobel McDonald, Ted

Four Major Events in Our History and the Lessons Learned Coleen Reiswig, Isobel McDonald, Ted Pincock, Carolyn Bouchard Joanne Archer Outline: 1) Common Themes 2) The Event and lessons learned: Spanish Influenza 1918 SARS

417 views • 38 slides
Lessons Learned from Japans NPL Experience Ladies and gentlemen, today I would like to discuss

Lessons Learned from Japans NPL Experience Ladies and gentlemen, today I would like to discuss

Beijing International Finance Forum Chairman Junichi Ujiie 2004/05/19 Lessons Learned from Japans NPL Experience Ladies and gentlemen, today I would like to discuss the lessons to be learned from Japans NPL problems: first, the disposal

344 views • 3 slides
Some lessons learned from Team Science Some lessons learned from Team Science Lewis Cantley Weill

Some lessons learned from Team Science Some lessons learned from Team Science Lewis Cantley Weill

Some lessons learned from Team Science Some lessons learned from Team Science Lewis Cantley Weill Cornell Medical College, New York Presbyterian Hospital Past participation in Team Science Period Period Type of grant Type of grant Role in Grant

380 views • 20 slides
MDG-SDG TRANSITION IN IVORY COAST: LESSONS LEARNED AND IMPLICATIONS ON CURRENT PUBLIC POLICIES

MDG-SDG TRANSITION IN IVORY COAST: LESSONS LEARNED AND IMPLICATIONS ON CURRENT PUBLIC POLICIES

MDG-SDG TRANSITION IN IVORY COAST: LESSONS LEARNED AND IMPLICATIONS ON CURRENT PUBLIC POLICIES SAKO G. . Oumar CONTENTS I. Context II. Assessment and lessons learned from implemention of MDGs III. Domestication/contextaulisation of SDGs

151 views • 12 slides
VALUE: Lessons Learned Kate McConnell &amp; Erin Horan Association of American Colleges and

VALUE: Lessons Learned Kate McConnell &amp; Erin Horan Association of American Colleges and

VALUE: Lessons Learned Kate McConnell &amp; Erin Horan Association of American Colleges and Universities Todays session Overview of the MSC Scoring and reporting Lessons learned as related to validity Unintended consequences?

661 views • 40 slides
Final Exam Prep Lecture ME EN 412 Andrew Ning aning@byu.edu Outline What Have We Learned

Final Exam Prep Lecture ME EN 412 Andrew Ning aning@byu.edu Outline What Have We Learned

Final Exam Prep Lecture ME EN 412 Andrew Ning aning@byu.edu Outline What Have We Learned Topics and Problems Examples What Have We Learned Topics: Dimensional Analysis, Balance Laws Potential Flow, Boundary Layers, Lift and Drag

483 views • 13 slides
FACILITATED BY: Robin Booth, CPA 2 Training Topics Lessons Learned Best Practices

FACILITATED BY: Robin Booth, CPA 2 Training Topics Lessons Learned Best Practices

Office of Housing Counseling Best 9/20/2016 Practices/Lessons Learned - LHCAs U.S. Department of Housing and Urban Development Office of Housing Counseling Best Practices/Lessons Learned LHCAs September 20, 2016 12:00 PM Eastern Standard

569 views • 9 slides

More recommend