Helping the Discovery of New Galaxies on the Worlds Largest - PowerPoint PPT Presentation

Helping the Discovery of New Galaxies on the World’s Largest Telescopes Using a Large GPU Cluster Damien Gratadour 1 and Hatem Ltaief 2 1 LESIA, Observatoire de Paris and Universit´ e Paris Diderot, France 2 Extreme Computing Research Center, KAUST, Saudi Arabia NVIDIA GTC Conference at San Jose, CA March 26-29, 2018 D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 1 / 61

Outline The European Extremely Large Telescope 1 Ubiquitous Taskification 2 Targeting Large-Scale GPU Supercomputer 3 Performance Results 4 Summary and Future Work 5 D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 2 / 61

Acknowledgments Students/Collaborators Extreme Computing Research Center @ KAUST A. Charara and D. Keyes L’Observatoire de Paris, LESIA R. Dembet, N. Doucet, E. Gendron, D. Gratadour, C. Morel, A. Sevin and F. Vidal Innovative Computing Laboratory @ UTK PLASMA/MAGMA/PaRSEC Teams INRIA/INP Bordeaux, France Runtime/HiePACS Teams D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 3 / 61

Acknowledgments Support/Funding Funded partially by the French National Center for Scientific Research (CNRS, 2016) Funded partially by the European Commission (Horizon 2020 program, FET-HPC grant# 671662) For core-hours: Tsubame 3.0, Tokyo, Japan Tokyo Institute of Technology Shaheen 2.0, Thuwal, KSA KAUST Supercomputer Lab D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 4 / 61

E-ELT Outline The European Extremely Large Telescope 1 Ubiquitous Taskification 2 Targeting Large-Scale GPU Supercomputer 3 Performance Results 4 Summary and Future Work 5 D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 5 / 61

E-ELT The effect of atmospheric turbulence The sun observed with a compact camera Disturbs the trajectory of light rays (wavefront perturbations) Reduces astronomical images quality D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 6 / 61

E-ELT Adaptive optics (AO) AO: technique used compensate in real-time the wavefront perturbations providing a significant improvement in resolution The moon observed with a 8m telescope (left: no AO, right: with AO) D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 7 / 61

E-ELT How AO works D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 8 / 61

E-ELT The Top 10 (present and future) Ground-based Telescopes Rank Name Location Diameter Cost Year 10 Large Synoptic Survey Telescope (LSST) Chile 8.4m 450 million 2014 9 South African Large Telescope (SALT) South Africa 9.2m 36 million 2005 8 Keck USA 10m 100 million 1996 7 Gran Telescopio Canarias (GTC) Spain 10.4m 130 million 2009 6 Aricebo Observatory Puerto Rico 305m 9.3 million 1963 D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 9 / 61

E-ELT The Top 10 (present and future) Ground-based Telescopes Rank Name Location Diameter Cost Year 10 Large Synoptic Survey Telescope (LSST) Chile 8.4m 450 million 2014 9 South African Large Telescope (SALT) South Africa 9.2m 36 million 2005 8 Keck USA 10m 100 million 1996 7 Gran Telescopio Canarias (GTC) Spain 10.4m 130 million 2009 6 Aricebo Observatory Puerto Rico 305m 9.3 million 1963 5 Atacama Large Millimeter Array (ALMA) Chile 12m 1.4 billion 2013 4 Giant Magellan Telescope (GMT) Chile 24.5m 2.2 billion 2024 3 Thirty Meter Telescope (TMT) USA 30m 1.4 billion 2030 2 Square Kilometer Array (SKA) Australia 90m 2 billion 2020 1 European Extremely Large Telescope (E-ELT) Chile 39m 1.3 billion 2024 D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 9 / 61

E-ELT The Top 10 (present and future) Ground-based Telescopes Rank Name Location Diameter Cost Year 10 Large Synoptic Survey Telescope (LSST) Chile 8.4m 450 million 2014 9 South African Large Telescope (SALT) South Africa 9.2m 36 million 2005 8 Keck USA 10m 100 million 1996 7 Gran Telescopio Canarias (GTC) Spain 10.4m 130 million 2009 6 Aricebo Observatory Puerto Rico 305m 9.3 million 1963 5 Atacama Large Millimeter Array (ALMA) Chile 12m 1.4 billion 2013 4 Giant Magellan Telescope (GMT) Chile 24.5m 2.2 billion 2024 3 Thirty Meter Telescope (TMT) USA 30m 1.4 billion 2030 2 Square Kilometer Array (SKA) Australia 90m 2 billion 2020 1 European Extremely Large Telescope (E-ELT) Chile 39m 1.3 billion 2024 Consortium: multiple nation initiatives Src: http://www.space.com/14075-10-biggest-telescopes-earth-comparison.html D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 9 / 61

E-ELT The World’s Biggest Eye on The Sky Credits: ESO (http://www.eso.org/public/teles-instr/e-elt/) D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 10 / 61

E-ELT The World’s Biggest Eye on The Sky Credits: ESO (http://www.eso.org/public/teles-instr/e-elt/) The largest optical/near-infrared telescope in the world. It will weigh about 2700 tons with a main mirror diameter of 39m. Location: Chile, South America. D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 11 / 61

E-ELT Multi-object Adaptive Optics (MOAO) Probably the most challenging embedded instrument in the E-ELT. Observe the most distant galaxies in parallel to understand their evolution with a high multiplex Capable of exploiting a Field of View (FoV) of 7 to 10 arcminutes (1/4 of the full moon) with a resolution of few tens of milli-arcsec (1/20,000 of the full moon) Use multiple guide stars (tomographic measurement of the turbulence) and multiple deformable mirrors (direction specific compensation). Extremely compute intensive at full scale. D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 12 / 61

E-ELT Multi-object Adaptive Optics (MOAO) Credits: ESO D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 13 / 61

E-ELT Multi-object Adaptive Optics (MOAO) Creating artificial guide stars. Credits : A. Reeves, Durham University D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 14 / 61

E-ELT Multi-object Adaptive Optics (MOAO) Probably the most challenging embedded instrument in the E-ELT. Extremely compute intensive at full scale. Today: need to simulate the system to provide efficient observations forecast (evaluate the possible science return, perform design studies). Tomorrow (2024): need to drive the system in real-time for routine operations Core component of the system: the real-time controller which provides commands to the deformable mirrors actuators from the measurements of multiple wavefront sensors. D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 15 / 61

E-ELT AO real-time controller Heterogeneous HPC facility D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 16 / 61

E-ELT AO real-time controller Heterogeneous HPC facility Real-time box: 1ms response time, from 100k measurements to several 5k actuators (see J. Bernard presentation @ GTC17 for our implementation on DGX-1). Similar to real-time inference D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 17 / 61

E-ELT AO real-time controller Heterogeneous HPC facility Here we concentrate on the supervisor sub-system Critical component for operations and observations forecast D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 18 / 61

E-ELT AO supervisor sub-system Cost function optimization for parameters identification ( Learn stage) Linear algebra for reconstructor matrix computation ( Apply stage) D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 19 / 61

E-ELT Learn process Fitting a covariance matrix on a model including system and turbulence parameters, using a score function Levenberg-Marquardt algorithm for function optimization Future development: rely on Machine learning approaches Example of fitted turbulence profile D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 20 / 61

E-ELT Learn process Multi-GPU process, including matrix generation and LM fit Time to solution on DGX-1 (P100) for a matrix size of 86k and 40 turbulence layers:240s (4 minutes) Weak and strong scaling for the learn process D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 21 / 61

E-ELT Apply process: tomographic reconstructor Compute the tomographic reconstructor matrix using covariance matrix between direction specific truth sensor and other sensors and the inverse of measurements covariance matrix R ′ = C tm · C − 1 mm Factorize and Solve for R ′ with C mm , a 100k x 100k matrix, is extremely compute intensive At the core of system operations (soft real-time, should be achieved in seconds to update the real-time box) Also a critical component for the numerical simulation of the system behavior (observation forecast for today’s design studies) D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 22 / 61

E-ELT AO simulation pipeline: observations forecast Goal is to produce image quality maps over the instrument’s full FoV depending on turbulence conditions evolution over the night D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 23 / 61

E-ELT AO simulation pipeline: observations forecast From system parameters and turbulence evolution timeline to image quality at the output of the system D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 24 / 61

HPC Tasking Outline The European Extremely Large Telescope 1 Ubiquitous Taskification 2 Targeting Large-Scale GPU Supercomputer 3 Performance Results 4 Summary and Future Work 5 D. Gratadour & H. Ltaief MOAO on Large GPU Cluster 25 / 61