Automating Topology Aware Mapping for Supercomputers Abhinav - PowerPoint PPT Presentation

Automating Topology Aware Mapping for Supercomputers Abhinav Bhatele, Gagan Gupta Laxmikant V. Kale 1 1

Application Topologies Patch Compute Proxy � � �� 2 2

Interconnect Topologies • Three dimensional meshes • 3D Torus: Blue Gene/L, Blue Gene/P , Cray XT4/5 • Trees • Fat-trees (Infiniband) and CLOS networks (Federation) • Dense Graphs • Kautz Graph (SiCortex), Hypercubes • Future Topologies? • Blue Waters, Blue Gene/Q 3 3

The Mapping Problem • Applications have a communication topology and processors have an interconnect topology • Definition: Given a set of communicating parallel “entities”, map them on to physical processors to optimize communication • Goals: • Balance computational load • Minimize communication traffic and hence contention 4 4

Scope of this work • Currently we are focused on 3D mesh/torus machines • For certain classes of applications Computation Communication bound bound Latency tolerant Latency sensitive 5 5

Application specific mapping OpenAtom Default Time per step (s) 0.3 Topology 0.225 0.15 0.075 0 512 1024 2048 4096 8192 Number of cores A. Bhatele, E. Bohm, and L. V. Kale. A Case Study of Communication A. Bhatele, L. V. Kale and S. Kumar, Dynamic Topology Aware Load Optimizations on 3D Mesh Interconnects. In Euro-Par, LNCS 5704, pages Balancing Algorithms for Molecular Dynamics Applications, In 23rd ACM 1015–1028, 2009. Distinguished Paper Award. International Conference on Supercomputing (ICS), 2009. 6 6

Application specific mapping OpenAtom NAMD Outer Brick Patch 2 Inner Brick Patch 1 Time per step (ms) Default Time per step (s) 0.3 15 Topology Oblivious Topology TopoAware Patches 0.225 11.25 TopoAware LDBs 0.15 7.5 0.075 3.75 0 0 512 1024 2048 4096 8192 512 1024 2048 4096 8192 16384 Number of cores Number of cores A. Bhatele, E. Bohm, and L. V. Kale. A Case Study of Communication A. Bhatele, L. V. Kale and S. Kumar, Dynamic Topology Aware Load Optimizations on 3D Mesh Interconnects. In Euro-Par, LNCS 5704, pages Balancing Algorithms for Molecular Dynamics Applications, In 23rd ACM 1015–1028, 2009. Distinguished Paper Award. International Conference on Supercomputing (ICS), 2009. 6 6

Automatic Mapping • Obtaining the processor topology and the application communication graph • Pattern matching to identify regular patterns • 2D/3D near-neighbor communication • A suite of heuristics: the right strategy invoked depending on the communication scenario: • Regular communication • Irregular communication 7 7

Topology Discovery • Topology Manager API: for 3D interconnects (Blue Gene, XT) • Information required for mapping: • Physical dimensions of the allocated job partition • Mapping of ranks to physical coordinates and vice versa • On Blue Gene machines such information is available and the API is a wrapper • On Cray XT machines, jump several hoops to get this information and make it available through the same API http://charm.cs.uiuc.edu/~bhatele/phd/TopoMgrAPI.tar.gz 8 8

Application communication graph • Several ways to obtain the graph • MPI applications: • Graph obtained from a run can only be used in a subsequent run • Profiling tools (IBM’s HPCT tools) • Charm++ applications: • Instrumentation at runtime • Enables dynamic mapping for changing communication graphs 9 9

Pattern Matching • We want to identify simple communication patterns 0 Processors Pattern matching to identify simple communication patterns such as 2D/3D near-neighbor graphs 31 10 10

Communication Graphs • Regular communication: • POP (Parallel Ocean Program): 2D Stencil like computation • WRF (Weather Research and Forecasting model): 2D Stencil • MILC (MIMD Lattice Computation): 4D near-neighbor • Irregular communication: • Unstructured mesh computations: FLASH, CPSD code • Many other classes of applications 11 11

Mapping Regular Graphs • Maximum Overlap (MXOVLP) Object Graph: 7 x 4 Processor Graph: 4 x 7 • Expand from Corner (EXCO) • Affine Mapping (AFFN) 12 12

Example Mapping Object Graph: 6 x 11 Processor Graph: 11 x 6 Aleliunas, R. and Rosenberg, A. L. On Embedding Rectangular Grids in Square Grids. IEEE Trans. Comput., 31(9):907–913, 1982 13 13

Different mapping solutions Object graph of 14 x 6 to processor graph of 7 x 12 Algorithms in order: MXOVLP , MXOV+AL, EXCO, COCE, AFFN, STEP 14 14

Evaluation Metric: Hop-bytes • Weighted sum of message sizes where the weights are the number of links traversed by each message d i = distance b i = bytes n = no. of messages • Indicator of the communication traffic and hence contention on the network • Previously used metric: maximum dilation 15 15

Evaluation 30 MXOVLP MXOV+AL Hops per processor EXCO COCE 22.5 AFFN STEP Lower Bound 15 7.5 0 14X6 to 7X12 16X16 to 8X32 27X35 to 45X21 Different mapping configurations 16 16

Results: WRF • Performance Average hops per byte per core Default improvement Topology 4 Lower Bound negligible on 256 and 512 cores 3 • On 1024 cores: 2 • Hops reduce by: 64% 1 • Time for communication reduces by 45% 0 • Performance improves 256 512 1024 2048 by 17% Number of nodes 17 17

Automating Topology Aware Mapping for Supercomputers Abhinav - PowerPoint PPT Presentation

Automating Topology Aware Mapping for Supercomputers Abhinav Bhatele, Gagan Gupta Laxmikant V. Kale 1 1 Application Topologies Patch Compute Proxy

IS TOPOLOGY IMPORTANT AGAIN? Effects of contention on message latencies in large supercomputers

Topology-Aware Data Aggregation for Intensive I/O on Large-Scale Supercomputers Franois Tessier

Adaptive Metric-Aware Job Scheduling for Production Supercomputers Wei Tang, Dongxu Ren,

Implementing new Topology Mapping Primitives Guillermo Baltra Prior Work Primitives for

CAM: CONSTRAINT AWARE APPLICATION MAPPING FOR APPLICATION MAPPING FOR EMBEDDED SYSTEMS Luis

iToM: An Internet Topology Mapping Project Kamil Sarac (ksarac@utdallas.edu) Department of

Volumetric instance-aware semantic mapping and 3D object discovery Margarita Grinvald, Fadri

Topology-aware OpenMP Process Scheduling Peter Thoman, Hans Moritsch, and Thomas Fahringer

Supercomputers and Supercomputers and Clusters and Clusters and Grid, Grid, Oh My! Oh My!

Network Topology-aware Traffic Scheduling Emin Gabrielyan cole Polytechnique Fdrale de

Communication and Topology-aware Load Balancing in Charm++ with TreeMatch Joint lab 10th workshop

Operationalizing Yarrp: High-Speed Active Network Topology Mapping from AWS

A Topology-Aware Performance Monitoring Tool for Shared Resource Management in Multicore Systems

Multicurve Cohomology of Mapping Class Groups Rasmus Villemoes Center for the Topology and

Topology-Aware Cooperative Data Protection in Blockchain-Based Decentralized Storage Networks

TARA: Topology-Aware Resource Adaptation for Congestion Avoidance in Wireless Sensor Networks

Network Topology-aware Traffic Scheduling Emin Gabrielyan, Roger D. Hersch cole Polytechnique

Enhancing Compact Routing in CCN with Prefix Embedding and Topology-Aware Hashing Stefanie Roos 1

Protecting Insecure Communications with Topology-aware Network Tunnels Georgios Kontaxis Angelos

Towards system-scale optimisation of HPC applications TADaaM : Topology-Aware System-Scale Data

ARCHITECTURE-AWARE MAPPING AND SCHEDULING OF IMA PARTITIONS ON MULTI-CORE PLATFORMS

Primitives for Active Internet Topology Mapping: Toward High-Frequency Characterization Robert

Strategies for Bulk RNA-seq Analysis Genome Transcriptome Assembly Mapping Mapping Reads

Network Topology-aware Traffic Scheduling Emin Gabrielyan, Roger D. Hersch cole Polytechnique