Mitglied der Helmholtz-Gemeinschaft
Thomas Lippert
Institute for Advanced Simulation Jülich Supercomputing Centre
Far more than Petaflops:
The Jülich Supercomputing Centre
ScicomP 15 & SP-XXL
Barcelona Supercomputing Centre May 20, 2009
Far more than Petaflops: The Jlich Supercomputing Centre ScicomP 15 - - PowerPoint PPT Presentation
Mitglied der Helmholtz-Gemeinschaft Far more than Petaflops: The Jlich Supercomputing Centre ScicomP 15 & SP-XXL Thomas Lippert Barcelona Supercomputing Centre Institute for Advanced Simulation May 20, 2009 Jlich Supercomputing
Mitglied der Helmholtz-Gemeinschaft
Thomas Lippert
Institute for Advanced Simulation Jülich Supercomputing Centre
ScicomP 15 & SP-XXL
Barcelona Supercomputing Centre May 20, 2009
Geophysics Solid State Physics Chemistry Particle Physics Structure of Matter Plasma Physics Nuclear Physics Astrophysik Kosmologie Astrophysics Cosmology New Physics
Environment Weather/Climatology Pollution / Ozone Hole Ageing Society Medicine Biology Energy Plasma Physics Fuel Cells Materials Spintronics Nano-Science
Supercomputing Drives Engineering and Business Competitiveness
Reducing design costs by virtual prototyping:
market
Allowing investigations where economics or ethics preclude experimentation
supercomputing
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
5
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
6
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
7
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
8
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
9
Soft Matter Biophysics Hadron Physics Nano/Material Science
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
10
1961 Zentralinstitut für Angewandte Mathematik (ZAM) 1987 Höchstleistungsrechenzentrum HLRZ 1998 HLRZJohn von Neumann Institut für Computing (NIC) 2007 ZAM Jülich Supercomputing Centre (JSC) Member of Gauss Centre for Supercomputing 2008 Institute for Advanced Simulation Coordinator of the PRACE Project 2010 European Supercomputing Centre
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
12
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
13
MATSE Education
Large Scale Facility Grid & Infrastructures
Distributed Systems & Grid Computing
Computational Science Mathematical Methods
Computational Science SimLab Biology SimLab Plasma Phys. SimLab
Complex Systems
NIC Research Group
HPC Systems HPC Operations HPC Data Management HPC System Development
European HPC Infrastructure
UNICORE Development Grid Research Performance Analysis
Helmholtz Young Investigators Group
Mathematics & Education Modeling & Methods
Numerical Algorithms Mathematical Software
Education HPC Application Support Applied Visualization Program Optimization Programming Environments SL Operation
Research Group Quant.-Inf.
Distributed Systems Communication Systems HPC Networking JuNet &
Security Network Technologies Director Secretaries, Administration Technology Technology Development
File- & Archive Systems
D-Grid Operation Technical Infrastructure NIC Coordination Public Relations User/Project Management
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
15
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
16
1956 First Computer in Jülich 1989 Cray YMP 0.003 Teraflop/s 1996 Cray T3E 0.8 Teraflop/s 2003 IBM p690 9 Teraflop/s 2006 BGL: JUBL 46 Teraflop/s 2008 BGP: JUGENE 223 Teraflop/s 2009 JuRoPA 200 Teraflop/s HPC-FF 100 Teraflop/s BGP: JUGENE 1000 Teraflop/s
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
17
IBM Blue Gene/L JUBL, 45 TFlop/s
2004 2005/6 2007/8
IBM Blue Gene/P JUGENE, 223 TFlop/s
2009
File Server GPFS File Server GPFS, Lustre
IBM Power 4+ JUMP, 9 TFlop/s IBM Power 6 JUMP, 9 TFlop/s IBM Blue Gene/P JUGENE, 1 PFlop/s
Intel Nehalem Clusters HPC-FF 100 TFlop/s JUROPA 200 TFlop/s General-Purpose Highly-Scalable
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
18
IBM Blue Gene/P JUGENE
core 850 MHz, 4-way SMP
5 PByte online disk capacity and up to 25 PByte offline tape capacity
First Petaflop system in Europe
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
20
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
21
8640 cores, 101 TF peak
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
22
23 x 4 QNEM modules, 24 ports each 6 x M9 switches, 648 ports max. each, 468/276 links used Mellanox MTS3600 switches (Shark), 36 ports, for service nodes 4 Compute Sets (CS) with 15 Compute Cells (CC) each CC with 18 Compute Nodes (CN) and 1 Mellanox MTS3600 (Shark) switch each Virtual 648-port switches constructed from 54x/44x Mellanox MTS3600
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
23
GPFS Storage Cluster for all our Supercomputers Supercomputers are Remote Clusters for GPFS 1 PB Capacity today, expansion to 6 PB in Q4 2009 20 GB/sec Bandwidth, expansion to 66 GB/sec in Q4 Tivoli Storage Manager (TSM) for backup, archive and HSM 2 SUN tape libraries used with TSM
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
25
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
26
Accelerators promise exciting performances at low power
CUDA, Brook, Cell-SDK, CellSS, RapidMind, OpenCL, ...
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
27
QPACE / eQPACE
Main design goal: energy- and cost-efficiency Developed by SFB/TR “Hadron Physics” 3-D torus network based on FPGA – SPE-to-SPE comm. Ultra-dense packaging: 25,6 TFlop/s per rack
Enhanced communication: Beyond nearest neighbor & MEM-to- MEM Support of standard communication layers (MPI)
JUICEnext QS22 cluster
Cell based computational platform and test facility
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
28
Cluster Management
Operating System
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
29
Internet Client Firewall JUGGLE
Globus LCG/gLite
UNICORE 5 Gateway
to the other UNICORE sites
UNICORE5
DMZ
UNICORE6
UNICORE 6 Registry SoftComp
UNICORE5 UNICORE6
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
31
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
32
design and operation
10 Gbit/s network
planning and operation
peering
R&D in on-demand
DEISA Phosphorus LOFAR
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
33
external connections
1,5 Tbit/s
>8 Tbit/s
FhG-Birlinghoven
DEISA, LOFAR, Phosphorus
Thomas Eickermann, PRACE Project Coordination@FZ-Jülich
20.5. 2009 SP-XXL, Barcelona Thomas Lippert, IAS/JSC
35
The European Roadmap for Research Infrastructures is the first comprehensive definition at the European level Research Infrastructures are
European Research Area A European HPC service:
communities
development
20.5. 2009 SP-XXL, Barcelona Thomas Lippert, IAS/JSC
36
permanent Research Infrastructure as a single Legal Entity from 2010 on including governance, funding, procurement, and usage strategies.
20.5. 2009 SP-XXL, Barcelona Thomas Lippert, IAS/JSC
37
General Partners Principal Partners
General Partners
tier 1 tier 0
29.4. 2009 Physikalisches Kolloquium Universität Duisburg-Essen
38
– degree of warming, scenarios for our future climate. – understand and predict ocean properties and variations – weather and flood events
– systems, structures which span a large range of different length and time scales – quantum field theories like QCD LHC, FAIR – ITER
– understanding complex materials, complex chemistry, nanoscience – the determination of electronic and transport properties
– system biology, chromatin dynamics, large scale protein dynamics, protein association and aggregation, supramolecular systems, medicine
– complex helicopter simulation, biomedical flows, gas turbines and internal combustion engines, forest fires, green aircraft
General Partners
Further PRACE Activities
BSC Genci EPSRC NCF GCS
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
41
~ 200 Projects
~40 Projects
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
42
User Support Training
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
43
SL Plasma Physics SL Biology SL Earth and Environment
Supercomputing Centre
SL Molecular Systems SL SL
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
44
Blue Gene/L Scaling Week, May 2006 Blue Gene Scaling Workshop, Dec 2006
National Lab.)
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
45
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
46
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
48
Soft Matter Composites DEISA I3HP Jülich Initiative Other 48
Chemistry Many Particle Physics Elementary Particle Physics Biology/Biophysics Material Science Soft Matter Other
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
49
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
50
3
European HPC Centre Topical HPC Centre, Centre with regional tasks HPC Server Aachen, DKRZ, Dresden, DWD, Erlangen, G-CSC Frankfurt, HLRN (Hannover, Berlin), Karlsruhe, MPG/RZG, Paderborn University/Institute
~ 10 ~ 100
National HPC Centre Garching, Jülich, Stuttgart
Gauß Centre for Supercomputing Gauß Alliance
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
52
Methods & Algorithms Parallel Performance
Simulation Laboratories (New)
Earth & Environment Plasma Physics Energy Biology Molecular Systems NanoMikro
Cross-Sectional Teams
Astro- Particle
Research Groups
Quantum Information NIC Group Distributed Computing
Education & Training Programmes
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
53
Research
Support
Groups
Protein 1LQ7
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
54
54
Research
Support
Codes
Outreach
Rossendorf
Laser-ion acceleration Solar flare modelling
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
55
(Dr. Paul Gibbon, FZJ)
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
56
Frontier research on quantum effects in computing
Research
(Gate imperfections, decoherence, error correction)
Cooperations
8-bit ion trap
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
58
time)
toolboxes
Cologne
FMM Evacuation modelling Soil-root interface
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
60
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
61
Motivation
The earthquake safety analysis of the 2nd Bosphorus Bridge in Istanbul is of great practical interest.
the bridge has to be developed
earthquake loads at Istanbul provides input for the simulation
Kandilli Observatory and Earthquake Research Institute Bogazici University, Istanbul
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
62
62
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
64
2008 2007 2006 2005 2004 2003 2002 2001 2000 1999 2009
European research & development and infrastructure projects
2010 2011 UNICORE UNICORE Plus EUROGRID GRIP GRIDSTART OpenMolGRID UniGrids VIOLA DEISA NextGRID CoreGRID D-Grid IP EGEE-II OMII-Europe A-WARE Chemomentum eDEISA PHOSPHORUS D-Grid IP 2 SmartLM PRACE D-MON DEISA2 ETICS2 SLA4D-Grid WisNetGrid
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
65
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
66 UNICORE WS-RF hosting environment XNJS – Site 1 IDB UNICORE Atomic Services OGSA-* Service Registry Local RMS (e.g. Torque, LL, LSF, etc.) Target System Interface – Site 1 Local RMS (e.g. Torque, LL, LSF, etc.) X.509, Proxies, SOAP, WS-RF, WS-I, JSDL OGSA-ByteIO, OGSA-BES, JSDL, HPC-P, OGSA-RUS, UR X.509, XACML, SAML, Proxies DRMAA UCC command- line client URC Eclipse-based Rich client Portal e.g. GridSphere HiLA Programming API Gateway – Site 1 UVOS VO Service External Storage USpace GridFTP, Proxies USpace XUUDB Workflow Engine Service Orchestrator XACML entity UNICORE WS-RF hosting environment XNJS – Site 2 IDB UNICORE Atomic Services OGSA-* Target System Interface – Site 2 XUUDB XACML entity Gateway – Site 2 CIS Info Service OGSA-RUS, UR, GLUE 2.0 Grid services hosting job incarnation web service stack data transfer to external storages authorization authentication scientific clients and applications central services running in WS-RF hosting environments Gateway
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
67
Core D-Grid sites committing parts of their existing resources to D-Grid
LRZ
DLR-DFD
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
68
Consortium of leading national HPC centers in Europe
heterogeneous HPC environment
UNICORE as Grid Middleware
services:
Dedicated network Shared file system Common production environment at all sites
applications
IDRIS – CNRS (Paris, France), FZJ (Jülich, Germany), RZG (Garching, Germany), CINECA (Bologna, Italy), EPCC ( Edinburgh, UK), CSC (Helsinki, Finland), SARA (Amsterdam, NL), HLRS (Stuttgart, Germany), BSC (Barcelona, Spain), LRZ (Munich, Germany), ECMWF (Reading, UK)
www.deisa.eu
20.5.2009 SP-XXL, Barcelona Thomas Lippert, IAS/JSC
70
Objective
parallel codes with highest scalability Research
analysis tool for large-scale systems Current projects
Scalasca-Cube screenshot
20.5.2009 SP-XXL, Barcelona Thomas Lippert, IAS/JSC
71
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
77
Etwas Theorie…
12.1.2009 Physikalisches Kolloquium Ruhr-Universität Bochum
77
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
78
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
79
Proton‟s Mass „Predicted‟
STARTING FROM A THEORETICAL DESCRIPTION OF ITS INNARDS,
physicists precisely calculated the mass of the proton and
aren‟t new; experimenters have been able to weigh the proton for nearly a century. But the new results show that physicists can at last make accurate calculations of the ultracomplex strong force that binds quarks….
20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC
80
Mitglied der Helmholtz-Gemeinschaft