Chicagoland Computational Cosmology Salman Habib High Energy Physics Division Mathematics and Computer Science Division Argonne National Laboratory habib@anl.gov Monday, June 13, 2011
Overview • Primary Target: Cosmological signatures of physics beyond the Standard Model, plus ‘ancillary science’ • Structure Formation Probes: Exploit nonlinear regime of structure formation • Discovery Science: Derive signatures of new physics, search for new cosmological probes • Precision Predictions: Aim to MC2/HPM produce the best predictions and LSST on Cerro Pachon error estimates/distributions for structure formation probes (rough analogy with lattice QCD) • Design and Analysis: Advance ‘Science of Surveys’; contribute to major ‘Dark Universe’ missions: BOSS, DES, LSST , BigBOSS, DESpec -- Monday, June 13, 2011
Cosmological Probes of New Physics • Dark Energy: Properties of DE ROSAT (X-ray) WMAP (microwave) equation of state, modifications of GR, other models? Sky surveys, terrestrial experiments • Dark Matter: Direct/Indirect searches, clustering properties, constraints on model parameters Sky surveys, targeted observations, Fermi (gamma ray) SDSS (optical) terrestrial experiments • Inflation: Probing primordial fluctuations, CMB polarization, non- Gaussianity Sky surveys • Neutrino Sector: CMB, linear and nonlinear matter clustering Explosion of information from sky Sky surveys, terrestrial experiments maps: Precision cosmology requires detailed modeling, simulations, and inverse analysis Monday, June 13, 2011
Chicagoland Opportunities: We Have it All! • Fundamental Science JK ‣ Beyond the Cosmological Standard Model ‣ Precision Predictions for Cosmological Probes • Survey Science HF ‣ Interpretation of data from surveys (DES, SPT , --) ‣ Design of future surveys (DESpec, LSST , --) ‣ Simulation database accessible by local and external community • High Performance Computing and Data-Intensive Supercomputing AP (DISC) ‣ HPC: Petascale to Exascale at Argonne ‣ High-throughput computing at Fermilab DES ‣ Data-oriented science at Computation Institute • Education and Outreach Mira ‣ Student and post-doc involvement SB ‣ Opportunities for cross-disciplinary collaborations Monday, June 13, 2011
Chicagoland: Computational Capabilities JK • Complementary Simulation Codes ‣ Coverage of gravity and hydrodynamics and ‘sub-grid’ physics ‣ Adaptive Refinement Tree (ART) at UofC/Fermilab HF ‣ Hardware Accelerated Cosmology Codes (HACC) framework at Argonne ‣ FLASH at UofC in collaboration with Argonne • Data-Intensive Computing AP ‣ Grid computing at all three institutions ‣ Petascale Active Data Store (PADS) at CI ‣ High-speed data transfers available locally • Next-Generation Computational Platforms and Associated Computer Science ‣ IBM BG/Q Mira (10 PF , 750 TB) coming to Argonne in 2012 SB ‣ Remote and in situ visualization/analysis Monday, June 13, 2011
Chicagoland: Computational Capabilities • Example: High-throughput data transfer JK ‣ Globus Online (CI) used for transferring 22 TB of data from LANL to ANL ‣ Data will be used to build a pilot public cosmology simulation database Data transfers on May 23/24 HF 5 Gb/s AP SB 3.00pm 12.00am 9.00am Monday, June 13, 2011
Cosmology: An Argonne View • Cosmology is Multi-Disciplinary: Modern cosmology needs physics, astronomy, computer science, engineering, statistics, -- • Cosmology is Diverse: Many avenues for people to contribute in different ways BG/Q • Precision Cosmology as Big Science: Small and large team efforts: ‘HEP style science’ • Leadership Computing: Cosmology is one field that can and must exploit next-generation supercomputing (‘science at scale’) • Large Data Sets: Simulations and observations ANL TCS are driving a new approach to data-centric science (potentially good connection with HEP data efforts) • National/International Collaborations: National Labs as collaboration/science hubs Monday, June 13, 2011
New Faces at Argonne HEP/MCS staff: JK Salman Habib Katrin Heitmann HEP post-docs: HF Suman Bhattacharya [joint with UofC] SD Sanghamitra Deb Juliana Kwan Adrian Pope [ANL Named Fellow] BB Amol Upadhye [ANL Director’s Fellow] AP AU ALCF post-doc: Hal Finkel [Mira ESP] ALCF student: Brittney Bullis [Mira ESP] Work very closely with local/area theorists, observers, experimentalists, and computer SB scientists -- Monday, June 13, 2011
Hardware-Accelerated Cosmology Code (HACC) Framework • Architecture Challenge: HPC is rapidly evolving (clusters/BG/CPU+GPU/MIC --) Perfect scaling on Roadrunner, 64 billion particle hi-res run • Code for the Future: Melds optimized now completing performance, low memory footprint, embedded analysis, and scalability • Implementation: Long/short-range force matching with spectral force-shaping (long-range=PM, short-range=PP , tree) • Key Features: Hybrid particle/grid design, particle overloading, spectral operators, mixed-precision, node-level ‘plug-ins’, ~ 50% of peak Flops Mira ‘Titan’ 10 PFlops • Cross-Platform: Designed for all current 20 PFlops and future supercomputing platforms • Embedded Analysis: High performance with low I/O and storage requirement Roadrunner 2 PFlops Habib et al. 2009, Pope et al. 2010 Monday, June 13, 2011
Status and Near-Term Outlook • HACC: Up and running on GPU and Cell accelerated hardware • ESP on Mira: 150M CPU hours, tree BOSS Ly-alpha simulation implementation almost completed, Roadrunner view (halos) of the Universe at z=2 from a 64 billion particle run full-time post-doc and student • Hydrodynamics: HACC optimal for particle-in-cell approach, algorithm development initiated, aim to complete by Mira arrival • HACC and Large Datasets: Simulations and mocks need to be where the data is -- ANL provides state of the art resources via ALCF SDSS LRG mock catalog Mock catalog for SDSS luminous red galaxies (orange) and satellite galaxies (green), in coll. with M. White Monday, June 13, 2011
Ultimate Vision for a Large-Scale Effort • Clean up ‘Discovery Space’: Robust theory with subtle signals • Precision Cosmic Calibration at Scale: ‘All Sky’ solution of the cosmic inverse problems in the nonlinear regime • Cosmology Simulations at the Exascale: Next-generation computing and beyond as essential theoretical and analysis tools - • • Large Data: Simulation & observational datastreams: Archiving, serving, quality assurance, (joint) analyses • Simulation/Data/Analysis: Cross- SDSS LRG mock catalog Observation Science platform, multi-source, analysis and Feed interrogation frameworks Results! Monday, June 13, 2011
Chicagoland Planning • Progress on Report JK ‣ Boroad science case made and targets identified ‣ Scale of computations understood (previous studies and white papers --) ‣ Software and algorithmic development requirements identified (driven by HF next-generation hardware mix and expected ‘data deluge’) • Boundary Conditions ‣ DOE/NNSA exascale co-design initiative future murky (progress too slow?) ‣ DOE HEP has initiated a new planning exercise for computational cosmology AP ‣ Need to get inputs from surveys (DES, LSST , --), increase interaction level ‣ Possible cross-agency collaboration (DOE, NSF , --) • Timescale for Report and Suggested Actions ‣ Need to initiate/continue and investigate local science collaborations and interactions in detail (seminar series, regular task force meetings) ‣ Input from dynamic ‘boundary conditions’ and SWOT/competition analysis SB ‣ Continue work on report, final version with recommendations in the fall Monday, June 13, 2011
Collaborations and Projects Kev Abazajian, Jim Ahrens, Ujjaini Alam, Debbie Bard, Sergei Bashinsky, Andrew Benson, Gary Bernstein, Joe Bernstein, Derek Bingham, Rahul Biswas, Anna Cabre, Jordan Carlson, Joanne Cohn, Andrew Connolly, David Daniel, Nehal Desai, Scott Dodelson, Tim Eifler, Gus Evrard, Patricia Fasel, Hume Feldman, Wu Feng, Josh Frieman, David Higdon, Tracy Holsclaw, Chung-Hsing Hsu, Bhuvnesh Jain, Steve Kahn, Lloyd Knox, Savvas Koushiappas, Steve Kuhlmann, Earl Lawrence, Herbie Lee, Adam Lidz, Zarija Lukic, Chris Miller, Charles Nakhleh, Mike Norman, Alex Pang, Uliana Popov, Darren Reed, Paul Ricker, Robert Ryne, Bruno Sanso, Paul Sathre, Michael Schneider, Sergei Shandarin, Alex Szalay, Tony Tyson, Licia Verde, Alexei Vikhlinin, Christian Wagner, Mike Warren, Martin White, Brian Williams, Tim Williams, Jon Woodring, --- Monday, June 13, 2011
Recommend
More recommend
