to understand parallel program behaviors
play

to Understand Parallel Program Behaviors LAI WEI , JOHN - PowerPoint PPT Presentation

Apr 03, 2023 •203 likes •1.15k views

Automated Analysis of Time Series Data to Understand Parallel Program Behaviors LAI WEI , JOHN MELLOR-CRUMMEY RICE UNIVERSITY HOUSTON, TX, USA Background Parallel computers of increasing scale Support scientific simulations of increasing

  1. Automated Analysis of Time Series Data to Understand Parallel Program Behaviors LAI WEI , JOHN MELLOR-CRUMMEY RICE UNIVERSITY HOUSTON, TX, USA

  2. Background Parallel computers of increasing scale ◦ Support scientific simulations of increasing ambition Performance of many applications fail to scale accordingly ◦ Load imbalance, serialization, network congestion, etc. Performance tools to understand application behaviors ◦ Measure and present performance data ◦ Used by experts to manually identify performance inefficiencies 2

  3. Profile Breaks down application run time into sources of costs Calling context P0 P1 P2 P3 main() 9s 9s 9s 9s init() 1s 1s 1s 1s solve() 8s 8s 8s 8s compute() 4s 4.1s 3.9s 4s sync() 4s 3.9s 4.1s 4s 3

  4. Profile Breaks down application run time into sources of costs Calling context P0 P1 P2 P3 main() 9s 9s 9s 9s init() 1s 1s 1s 1s solve() 8s 8s 8s 8s compute() 4s 4.1s 3.9s 4s sync() 4s 3.9s 4.1s 4s 4

  5. Profile Breaks down application run time into sources of costs Calling context P0 P1 P2 P3 main() 9s 9s 9s 9s init() 1s 1s 1s 1s solve() 8s 8s 8s 8s compute() 4s 4.1s 3.9s 4s sync() 4s 3.9s 4.1s 4s Performance loss, why? 5

  6. Time series Presents application behavior over time init() solve() Calling context P0 main() init() P1 solve() P2 compute() sync() P3 3s 5s 7s 9s Depth = 1 1s 6

  7. Time series Presents application behavior over time init() compute() sync() Calling context P0 main() init() P1 solve() P2 compute() sync() P3 3s 5s 7s 9s Depth = 2 1s 7

  8. Time series Presents application behavior over time init() compute() sync() Calling context P0 main() init() P1 solve() P2 compute() sync() P3 3s 5s 7s 9s Depth = 2 1s 8

  9. Time series Presents application behavior over time init() compute() sync() Calling context P0 main() init() P1 solve() P2 compute() sync() P3 3s 5s 7s 9s Depth = 2 1s Load imbalance 9

  10. Time series Presents application behavior over time init() compute() sync() Calling context P0 main() init() P1 solve() P2 compute() sync() P3 3s 5s 7s 9s Depth = 2 1s Load imbalance 10

  11. Time series Presents application behavior over time init() compute() sync() Calling context P0 main() init() P1 solve() P2 compute() sync() P3 3s 5s 7s 9s Depth = 2 1s Load imbalance 11

  12. Motivation Experts manually examine time series ◦ Understand how and why performance inefficiencies arise Time series of large scale parallel executions ◦ Vast in three dimensions ◦ Process ◦ Time ◦ Call path depth ◦ Manual analysis is difficult if not impractical 12

  13. Related work -- automated analysis Analysis of profiles [Huck, SC’05] [Tallent, SC’10 ] ◦ Often insufficient for diagnosing how and why parallel inefficiencies arise Analysis of execution traces ◦ Collecting instrumentation-based traces are costly in time and space ◦ Fine-grained traces explode at large scale ◦ Analysis at coarse granularity [Gonzalez , IPDPS’09] [Llort , IPDPS’10] ◦ Still needs lots of manual effort ◦ Analysis at fine granularity for short intervals [Geimer , CCPE’10 ] [Böhme, TOPC’16] ◦ Requires prior knowledge for selective tracing 13

  14. Our contribution Automated analysis of sample-based time-series data ◦ Feasible for large-scale programs ◦ Data volume is manageable ◦ Derive compact top-down summaries ◦ Uncover patterns and variance ◦ Direct attention to potential performance losses ◦ Attribute losses to code regions where they originate 14

  15. Approach 1. Collect and prepare sample-based time-series for further analysis ◦ Collect a time series of call paths with HPCToolkit ◦ Organize each time series as a tree of program calling contexts ◦ Identify iterative behaviors in the time series 2. Build clusters across threads and loop iterations 3. Quantify performance losses and attribute them to call paths 15

  16. Collect call path samples over time 1 Depth = 0 2 Depth = 1 3 4 Depth = 2 5 Depth = 3 6 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 16

  17. Collect call path samples over time 1 Depth = 0 2 Depth = 1 3 4 Depth = 2 5 Depth = 3 6 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 17

  18. Collect call path samples over time 1 Depth = 0 2 Depth = 1 3 4 Depth = 2 5 Depth = 3 6 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 18

  19. Collect call path samples over time 1 Depth = 0 2 Depth = 1 3 4 Depth = 2 5 Depth = 3 6 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 19

  20. Collect call path samples over time 1 Depth = 0 2 Depth = 1 3 4 Depth = 2 5 Depth = 3 6 T1 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 20

  21. Collect call path samples over time 1 main() Depth = 0 2 foo@13 Depth = 1 3 C@5 4 Depth = 2 5 Depth = 3 6 T1 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 21

  22. Collect call path samples over time 1 main() Depth = 0 2 foo@13 Depth = 1 3 C@5 4 Depth = 2 5 Depth = 3 6 T1 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 22

  23. Collect call path samples over time 1 main() Depth = 0 2 foo@13 Depth = 1 3 C@5 4 Depth = 2 5 Depth = 3 6 T1 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 23

  24. Collect call path samples over time 1 main() Depth = 0 2 foo@13 Depth = 1 3 C@5 4 Depth = 2 5 Depth = 3 6 T1 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 24

  25. Collect call path samples over time 1 main() Depth = 0 2 foo@13 Depth = 1 3 C@5 4 Depth = 2 5 Depth = 3 6 T1 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 25

  26. Collect call path samples over time 1 main() Depth = 0 2 foo@13 Depth = 1 3 C@5 4 Depth = 2 5 Depth = 3 6 T1 T2 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 26

  27. Collect call path samples over time 1 main() main() Depth = 0 2 foo@13 foo@13 Depth = 1 3 C@5 4 Depth = 2 A@7 5 Depth = 3 6 T1 T2 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 27

  28. Collect call path samples over time 1 main() main() Depth = 0 2 foo@13 foo@13 Depth = 1 3 C@5 4 Depth = 2 A@7 5 Depth = 3 6 T1 T2 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 28

  29. Collect call path samples over time 1 main() main() Depth = 0 2 foo@13 foo@13 Depth = 1 3 C@5 loop@6 4 Depth = 2 A@7 5 Depth = 3 6 T1 T2 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 29

  30. Collect call path samples over time 1 main() main() Depth = 0 2 foo@13 foo@13 Depth = 1 3 C@5 loop@6 4 Depth = 2 A@7 5 Depth = 3 6 T1 T2 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 30

  31. Collect call path samples over time 1 main() main() Depth = 0 2 foo@13 foo@13 Depth = 1 3 C@5 loop@6 4 Depth = 2 A@7 5 Depth = 3 6 T1 T2 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 31

  32. Collect call path samples over time 1 main() main() Depth = 0 2 foo@13 foo@13 Depth = 1 3 C@5 loop@6 4 Depth = 2 A@7 5 Depth = 3 6 T1 T2 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 32

  33. Collect call path samples over time 1 main() main() Depth = 0 2 foo@13 foo@13 Depth = 1 3 C@5 loop@6 4 Depth = 2 A@7 5 Depth = 3 6 T1 T2 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 33

  34. Collect call path samples over time 1 main() main() main() Depth = 0 2 foo@13 foo@13 foo@13 Depth = 1 3 C@5 loop@6 loop@6 4 Depth = 2 A@7 B@8 5 Depth = 3 6 T1 T2 T3 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 34

  35. Collect call path samples over time 1 main() main() main() Depth = 0 2 foo@13 foo@13 foo@13 Depth = 1 3 C@5 loop@6 loop@6 4 Depth = 2 A@7 B@8 5 Depth = 3 6 T1 T2 T3 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 35

  36. Collect call path samples over time 1 main() main() main() Depth = 0 2 foo@13 foo@13 foo@13 Depth = 1 3 C@5 loop@6 loop@6 4 Depth = 2 A@7 B@8 5 Depth = 3 6 T1 T2 T3 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 36

  37. Collect call path samples over time 1 main() main() main() main() Depth = 0 2 foo@13 foo@13 foo@13 foo@13 Depth = 1 3 C@5 loop@6 loop@6 loop@6 4 Depth = 2 A@7 B@8 A@7 5 Depth = 3 6 T1 T2 T3 T4 7 8 Depth = 0 9 Depth = 1 10 Depth = 2 11 12 Depth = 3 13 14 15 37

  38. Collect call path samples over time 1 main() main() main() main() Depth = 0 2 foo@13 foo@13 foo@13 foo@13 Depth = 1 3 C@5 loop@6 loop@6 loop@6 4 Depth = 2 A@7 B@8 A@7 5 Depth = 3 6 T1 T2 T3 T4 7 main() 8 Depth = 0 9 foo@13 Depth = 1 10 loop@6 Depth = 2 11 A@7 12 Depth = 3 13 T5 14 15 38

Recommend

Using Sampling to Understand Parallel Program Performance Nathan Tallent John Mellor-Crummey M.

Using Sampling to Understand Parallel Program Performance Nathan Tallent John Mellor-Crummey M.

Using Sampling to Understand Parallel Program Performance Nathan Tallent John Mellor-Crummey M. Krentel, L. Adhianto, M. Fagan, X. Liu Dept. of Computer Science, Rice University Parallel Tools Workshop TU Dresden Sept 26-27, 2011

409 views • 39 slides
iRobot Create Behaviors and GUI CSSE 120Rose Hulman Institute of Technology Thought for the

iRobot Create Behaviors and GUI CSSE 120Rose Hulman Institute of Technology Thought for the

iRobot Create Behaviors and GUI CSSE 120Rose Hulman Institute of Technology Thought for the week Any fool can write code that a computer can understand. Good programmers write code that humans can understand. R. Fowler, Refactoring:

389 views • 12 slides
20-03-06 7. Learning Sequences/Behaviors How to use sequences/behaviors? Sequences and more

20-03-06 7. Learning Sequences/Behaviors How to use sequences/behaviors? Sequences and more

20-03-06 7. Learning Sequences/Behaviors How to use sequences/behaviors? Sequences and more generally behaviors are about Sequences are used integrating the concept of time into what is learned. In } to analyze time dependent data general,

298 views • 6 slides
Use of Task Graph Model for Parallel Program Design Detailed steps for parallel program design

Use of Task Graph Model for Parallel Program Design Detailed steps for parallel program design

CS140, 2014 I-1 Use of Task Graph Model for Parallel Program Design Detailed steps for parallel program design and implementation 1. Preparing Parallelism Computational task partitioning. Aggregate tasks when needed.

374 views • 16 slides
Performance analysis Goals are to be able to understand better why your program has the

Performance analysis Goals are to be able to understand better why your program has the

Performance analysis Goals are to be able to understand better why your program has the performance it has, and what could be preventing its performance from being better. Speedup Parallel time T P (p) is the time it takes the

838 views • 81 slides
High Impact Leadership Behaviors Self-Assessment Tool This self-assessment tool is intended to

High Impact Leadership Behaviors Self-Assessment Tool This self-assessment tool is intended to

High Impact Leadership Behaviors Self-Assessment Tool This self-assessment tool is intended to help leaders understand and improve their effectiveness by practicing the high impact behaviors. The questions and statements under each behavior are

176 views • 3 slides
Specification-based Testing Software Engineering Gordon Fraser Saarland University Program

Specification-based Testing Software Engineering Gordon Fraser Saarland University Program

Specification-based Testing Software Engineering Gordon Fraser Saarland University Program behaviors Specified Implemented Structural Testing Program behaviors Specified Implemented Functional Testing Program behaviors Specified

163 views • 15 slides
Image Sharpening Example Running a simple parallel program Aims (i) To familiarise yourself

Image Sharpening Example Running a simple parallel program Aims (i) To familiarise yourself

Image Sharpening Example Running a simple parallel program Aims (i) To familiarise yourself with running parallel programs To run a real parallel code (that does file I/O) on different numbers of cores measure the time

700 views • 11 slides
ENGAGING STAKEHOLDERS IN SUSTAINABLE BEHAVIORS Kimberly A. Horndeski Water Resource Program

ENGAGING STAKEHOLDERS IN SUSTAINABLE BEHAVIORS Kimberly A. Horndeski Water Resource Program

ENGAGING STAKEHOLDERS IN SUSTAINABLE BEHAVIORS Kimberly A. Horndeski Water Resource Program Coordinator Texas Parks and Wildlife Department COMMUNITY BASED SOCIAL MARKETING Combines the knowledge from psychology and social Identify

527 views • 24 slides
PARALLEL Joachim Nitschke PROGRAMMING Project Seminar Parallel Programming, Summer

PARALLEL Joachim Nitschke PROGRAMMING Project Seminar Parallel Programming, Summer

PARALLEL Joachim Nitschke PROGRAMMING Project Seminar Parallel Programming, Summer Semester 2011 CONTENT Introduction Parallel program design Patterns for parallel programming A: Algorithm structure B: Supporting

831 views • 40 slides
Transformation based parallel programming Program parallelization techniques. 1. Program Mapping

Transformation based parallel programming Program parallelization techniques. 1. Program Mapping

CS140, 2014 III-1 Transformation based parallel programming Program parallelization techniques. 1. Program Mapping Program partitioning (with task aggregation). Dependence analysis. Scheduling & load balancing. Code

717 views • 38 slides
Performance Management Program Procedures Ob Objectives ectives To understand the

Performance Management Program Procedures Ob Objectives ectives To understand the

Performance Management Program Procedures Ob Objectives ectives To understand the Performance Management Program Process To understand how to complete the Work Plan and Appraisal Form Ov Overview erview Proc rocesses esses

551 views • 25 slides
Scheduling MIMD parallel program A number of tasks executing serially or in parallel

Scheduling MIMD parallel program A number of tasks executing serially or in parallel

Scheduling MIMD parallel program A number of tasks executing serially or in parallel The scheduling problem NP-complete problem (in general) Distribute tasks on processors so that minimal execution time Lecture 5: Load Balancing

348 views • 6 slides
Parallel and Distributed Programming Introduction Kenjiro Taura 1 / 21 Contents 1 Why Parallel

Parallel and Distributed Programming Introduction Kenjiro Taura 1 / 21 Contents 1 Why Parallel

Parallel and Distributed Programming Introduction Kenjiro Taura 1 / 21 Contents 1 Why Parallel Programming? 2 What Parallel Machines Look Like, and Where Performance Come From? 3 How to Program Parallel Machines? 4 How to Program Parallel

337 views • 33 slides
Applying Machine Learning to Understand Write Performance of Large-scale Parallel Filesystems

Applying Machine Learning to Understand Write Performance of Large-scale Parallel Filesystems

Applying Machine Learning to Understand Write Performance of Large-scale Parallel Filesystems Presented by Bing Xie Bing Xie, Zilong Tan, Philip Carns, Jeff Chase, Kevin Harms, Jay Lofstead, Sarp Oral, Sudharshan S. Vazhkudai, Feiyi Wang ORNL

222 views • 18 slides
Elmer Parallel Computing ElmerTeam CSC IT Center for Science Ltd. CSC, April 2013 Parallel

Elmer Parallel Computing ElmerTeam CSC IT Center for Science Ltd. CSC, April 2013 Parallel

Elmer Parallel Computing ElmerTeam CSC IT Center for Science Ltd. CSC, April 2013 Parallel computing concepts Parallel computation means executing tasks concurrently A task encapsulates a sequential program and local data, and its

500 views • 31 slides
Introduction to ARCHER and Cray MPI Running a Simple Parallel Program Aims To familiarise

Introduction to ARCHER and Cray MPI Running a Simple Parallel Program Aims To familiarise

Introduction to ARCHER and Cray MPI Running a Simple Parallel Program Aims To familiarise yourself with running parallel programs how to compile on ARCHER how to submit jobs to the compute nodes using PBS To run a real parallel

356 views • 10 slides
Ideal strategy follows program execution behaviors Multiple-area collection Problem: CPU

Ideal strategy follows program execution behaviors Multiple-area collection Problem: CPU

Ideal strategy follows program execution behaviors Multiple-area collection Problem: CPU cost of scavenging depends in part on size of objects Copying small objects no more expensive than marking with bitmap Cost of copying large

415 views • 16 slides
Ideal strategy follows program execution behaviors Multiple-area collection Problem: CPU

Ideal strategy follows program execution behaviors Multiple-area collection Problem: CPU

Ideal strategy follows program execution behaviors Multiple-area collection Problem: CPU cost of scavenging depends in part on size of objects Copying small objects no more expensive than marking with bitmap Cost of copying large

482 views • 18 slides
Learning objectives Understand how automated program analysis complements testing and

Learning objectives Understand how automated program analysis complements testing and

Learning objectives Understand how automated program analysis complements testing and manual inspection Most useful for properties that are difficult to test Program Analysis Understand fundamental approaches of a few

298 views • 8 slides
Dementia Related Behaviors: What Do We Know? What Can We Do? Lisa P. Gwyther, MSW, LCSW

Dementia Related Behaviors: What Do We Know? What Can We Do? Lisa P. Gwyther, MSW, LCSW

8/28/2012 Dementia Related Behaviors: What Do We Know? What Can We Do? Lisa P. Gwyther, MSW, LCSW Associate Professor, Department of Psychiatry and Behavioral Sciences Director, Duke Family Support Program Education Director, Bryan Alzheimers

529 views • 23 slides
Learning Objectives At the end of this session, attendees will: Understand what issues

Learning Objectives At the end of this session, attendees will: Understand what issues

New Jersey Vaccine Preventable Disease Program Vaccine for Children Program Christopher Ryan, MICP, CPM Learning Objectives At the end of this session, attendees will: Understand what issues constitute fraud and abuse Understand best

356 views • 23 slides
High Performance Fortran (HPF) Source: Chapter 7 of "Designing and building parallel

High Performance Fortran (HPF) Source: Chapter 7 of "Designing and building parallel

High Performance Fortran (HPF) Source: Chapter 7 of "Designing and building parallel programs (Ian Foster, 1995) Question Can't we just have a clever compiler generate a parallel program from a sequential program? Fine-grained

362 views • 24 slides
5/2/2017 1 Influence of teachers personal health behaviors on operationalizing obesity

5/2/2017 1 Influence of teachers personal health behaviors on operationalizing obesity

5/2/2017 1 Influence of teachers personal health behaviors on operationalizing obesity prevention policy in Head Start preschools: A project of the Childrens Healthy Living Program (CHL). Esquivel MK, Nigg C, Fialkowski JK, Braun K, Li F

414 views • 29 slides

More recommend