GPU Clusters for HPC Bill Kramer Director of Blue Waters National - PowerPoint PPT Presentation

GPU Clusters for HPC Bill Kramer Director of Blue Waters National Center for Supercomputing Applications University of Illinois at Urbana- Champaign National Center for Supercomputing Applications University of Illinois at Urbana-Champaign

National Center for Supercomputing Applications: 30 years of leadership • NCSA • R&D unit of the University of Illinois at Urbana-Champaign • One of original five NSF-funded supercomputing centers • Mission : Provide state-of-the-art computing capabilities (hardware, software, hpc expertise) to nation’ s scientists and engineers • The Numbers • Approximately 200 staff (160+ technical/professional staff) • Approximately 15 graduate students (+ new SPIN program), 15 undergrad students • Two major facilities (NCSA Building, NPCF) • Operating NSF’s most powerful computing system: Blue Waters • Managing NSF’s national cyberinfrastructure: XSEDE Source: Thom Dunning

Petas cale Computing Facility: Home to Blue Waters • Blue Waters • 13PF, 1500TB, 300PB • >1PF On real apps • NAMD, MILC, WRF, PPM, NWChem, etc • Energy Efficiency • Modern Data Center • LEED certified Gold • 90,000+ ft 2 total • Power Utilization Efficiency • 30,000 ft 2 raised floor = 1.1 – 1.2 20,000 ft 2 machine room gallery Source: Thom Dunning

Data Intensive Computing Personalized Medicine w/ Mayo LSST, DES Source: Thom Dunning

NCSA’s Industrial Partners Source: Thom Dunning

NCSA, NVIDIA and GPUs • NCSA and NVIDIA have been partners for over a decade, building the expertise, experience and technology. • The efforts were at first exploratory and small scale, but have now blossomed into providing the largest GPU production resource in the US academic cyber- infrastructure • Today, we are focusing on helping world class science and engineering teams decrease their time to insight for some of the world’s most important and challenging computational and data analytical problems Imaginations unbound

Original Blue Waters Goals • Deploy a computing system capable of sustaining more than one petaflops or more for a broad range of applications • Cray system achieves this goal using a well defined metrics • Enable the Science Teams to take full advantage of the sustained petascale computing system • Blue Waters Team has established strong partnership with Science Teams, helping them to improve the performance and scalability of their applications • Enhance the operation and use of the sustained petascale system • Blue Waters Team is developing tools, libraries and other system software to aid in operation of the system and to help scientists and engineers make effective use of the system • Provide a world-class computing environment for the petascale computing system • The NPCF is a modern, energy-efficient data center with a rich WAN environment (100-400 Gbps) and data archive (>300 PB) • Exploit advances in innovative computing technology • Proposal anticipated the rise of heterogeneous computing and planned to help the computational community transition to new modes for computational and data-driven science and engineering Imaginations unbound

Blue Waters Computing S ys tem Aggregate Memory – 1.6 PB >1 TB/sec IB Switch 10/40/100 Gb External Servers Ethernet Switch 100 GB/sec 120+ Gb/sec Spectra Logic: 300 usable PB Sonexion: 26 usable PB 100-300 Gbps WAN Imaginations unbound

Details of Blue Waters Imaginations unbound

Computation by Discipline on Blue Waters Actual Usage by Discipline STEM Education Physics Social Sciences 0.01% 2.5% 0.3% Astronomy and Astrophysics Particle Physics 17.8% 25.9% Atmospheric and Climate Sciences 10.4% Nuclear Physics 0.7% Mechanical and Dynamic Systems 0.03% Biology and Biophysics 23.6% Materials Fluid Humanities Science Systems 0.0002% 3.3% 5.1% Geophysics 1.3% Chemistry Engineering 6.5% 0.05% Earth Sciences 2.0% Computer Science 0.5% Imaginations unbound

XK7 Usage by NSF PRAC teams – A Behavior Experiment – First year • An observed experiment – teams self select what type of node is most useful • First year of usage MD- NAMD/VMD MD - Amber MD - Gromacs QCD – MILC and Chroma Increasing allocation size Imaginations unbound

Production Computational Science with XK nodes • The Computational Microscope • PI – Klaus Schulten • Simulated flexibility of ribosome trigger factor complex at full length and obtained better starting configuration of trigger factor model (simulated to 80ns) • 100ns simulation of cylindrical HIV 'capsule’ of CA proteins revealed it is stabilized by hydrophobic interactions between CA hexamers; maturation involves detailed remodeling rather than disassembly/re-assembly of CA lattice, as had been proposed. • 200ns simulation of CA pentamer surrounded by CA hexamers suggested interfaces in hexamer-hexamer and hexamer-pentamer pairings involve different patterns of interactions • Simulated photosynthetic membrane of a chromatophore in bacterium Rps. photometricum for 20 ns -- simulation of a few hundred nanoseconds will be needed Images from Klaus Schulten and John Stone, University of Illinois at Urbana-Champaign Imaginations unbound

Production Computational Science with XK nodes • Lattice QCD on Blue Waters • PI - Robert Sugar, University of California, Santa Barbara • The USQCD Collaboration, which consists of nearly all of the high-energy and nuclear physicists in the United States working on the numerical study of quantum chromodynamics (QCD), will use Blue Waters to study the theory of the strong interactions of sub-atomic physics, including simulations at the physical masses of the up and down quarks, the two lightest of the six quarks that are the fundamental constituents of strongly interacting matter Imaginations unbound

Production Computational Science with XK nodes • Hierarchical molecular dynamics sampling for assessing pathways and free energies of RNA catalysis, ligand binding, and conformational change • PI - Thomas Cheatham, University of Utah • Attempting to decipher the full landscape of RNA structure and function. • Challenging because • RNA require modeling the flexibility and subtle balance between charge, stacking and other molecular interactions • structure of RNA is highly sensitive to its surroundings, and RNA can adopt multiple functionally relevant conformations. • Goal - Fully map out the conformational, energetic and chemical landscape of RNA. • "Essentially we are able to push enhanced sampling methodologies for molecular dynamics simulation, specifically replica-exchange, to complete convergence for conformational ensembles (which hasn't really been investigated previously) and perform work that normally would take 6 months to years in weeks . This is critically important for validating and assessing the force fields for nucleic acids ,” - Cheatham. Images courtesy – T Cheatham Imaginations unbound

Imaginations unbound nodes Most Recent Computational Use of XK Node*Hours 1,000,000.0 2,000,000.0 3,000,000.0 4,000,000.0 5,000,000.0 6,000,000.0 7,000,000.0 8,000,000.0 9,000,000.0 - Teams with both XE and XK usage - July 1, 2014 to Sept 30, Karimabadi-3D Kinetic Sims. of… Sugar-Lattice QCD Yeung-Complex Turbulent Flows… Schulten-The Computational… Cheatham-MD Pathways and… Aksimentiev-Pioneering… Shapiro-Signatures of Compact… Mori-Plasma Physics Sims. using… Ott -CCSNe, Hypermassive… Voth -Multiscale Sims. of… Tajkhorshid -Complex Biology in… Glotzer-Many-GPU Sims. of Soft… Woosley-Type Ia Supernovae Jordan -Earthquake System… 2014 Aluru-QMC of H2O-Graphene,… Tomko-Redesigning Comm. and… Bernholc -Quantum Sims.… Pande -Simulating Vesicle Fusion Kasson-Influenza Fusion… Chemla-Chemla Lusk-Sys. Software for Scalable… Thomas -QC during Steel… Fields-Benchmark Human Variant… Makri-QCPI Proton & Electron… Hirata -Predictive Comp. of… Elghobashi-DNS of Vaporizing… Jongeneel-Accurate Gene… Lazebnik -Large-Scale… Beltran -Spot Scanning Proton… Woodward -Turbulent Stellar… XE Node Hrs XK Node Hrs Total Node*hrs

Most Resent Computational Use of XK nodes Teams with both XE and XK usage - July 1, 2014 to Sept 30, 2014 9,000,000.0 8,000,000.0 7,000,000.0 Node*Hours 6,000,000.0 5,000,000.0 4,000,000.0 3,000,000.0 2,000,000.0 1,000,000.0 Total Node*hrs XK Node Hrs - XE Node Hrs Imaginations unbound

Evolving XK7 Use on BW - Major Advance in Understanding of Collisionless Plasmas Enabled through Petascale Kinetic Simulations • PI: Homayoun Karimabadi, University of California, San Diego • Major results to date: • Global fully kinetic simulations of magnetic reconnection • First large-scale 3D simulations of decaying collisionless plasma turbulence • 3D global hybrid simulations addressing coupling between Fully kinetic simulation shock physics & (all species kinetic; Large scale hybrid kinetic simulation: code: VPIC) magnetosheath turbulence (kinetic ions + fluid electrons; ~up to 10 10 cells codes: H3D, HYPERES) ~up to 4x10 12 particles ~up to 1.7x10 10 cells ~120 TB of memory ~up to 2x10 12 particles ~10 7 CPU-HRS ~130 TB of memory Slide courtesy of H Karimardi ~up to 500,000 cores Imaginations unbound