Meshes/no meshes, radiation and tasks Numerical algorithms for the - - PowerPoint PPT Presentation

▶

Oct 20, 2023 259 likes •774 views

Meshes/no meshes, radiation and tasks Numerical algorithms for the future of astrophysical simulations Bert Vandenbroucke bv7@st-andrews.ac.uk Pedro Gonnet (Google), Matthieu Schaller (Leiden), Josh Borrow (Durham) Slide 2 of 51 Why do we

SLIDE 1

Meshes/no meshes, radiation and tasks

Numerical algorithms for the future of astrophysical simulations Bert Vandenbroucke

bv7@st-andrews.ac.uk

Pedro Gonnet (Google), Matthieu Schaller (Leiden), Josh Borrow (Durham)

SLIDE 2

Why do we use simulations?

Sedov, L. I., Similarity and dimensional methods in mechanics, 10th edition (CRC Press, 1993), pp. 261-290

Slide 2 of 51

SLIDE 3

Why do we use simulations?

X-ray: NASA/CXC/Rutgers/J.Hughes; Optical: NASA/STScI

Slide 3 of 51

SLIDE 4

Why do we use simulations?

gravity + hydrodynamics + gas physics (cooling/heating) + star formation + stellar feedback + AGN physics

image: EAGLE Project

Slide 4 of 51

SLIDE 5

SOME HISTORY...

Slide 5 of 51

SLIDE 6

Holmberg experiment

Holmberg, 1941

Slide 6 of 51

SLIDE 7

Hydrodynamics

Grid based 1D methods However (Gingold & Monaghan, 1977):

20 cells in 1D 400 cells in 2D 8000 (!) cells in 3D 1000 cells in 1D 1,000,000 cells in 2D 1,000,000,000 cells in 3D Slide 7 of 51

SLIDE 8

Smoothed Particle Hydrodynamics

Gingold & Monaghan, 1977

Slide 8 of 51

SLIDE 9

SPH

Pongracic, 1987

approximately 13,000 (!) particles Slide 9 of 51

SLIDE 10

Adaptive Meshes

Ruffert, 1992

Berger & Colella, 1989

3 × 643 = 786,432 cells

Saftly et al., 2013

Slide 10 of 51

SLIDE 11

Santa Barbara cluster comparison

Dark matter Gas

Frenk et al., 1999

Slide 11 of 51

SLIDE 12

Blob test

Agertz et al., 2007

Slide 12 of 51

SLIDE 13

Shearing layers test

SPH AMR GADGET2

RAMSES

Slide 13 of 51

SLIDE 14

Bulk movements

BV, public PhD presentation Slide 14 of 51

SLIDE 15

Galilean invariance

Lagrangian Eulerian

EXTRA BOOST VELOCITY BV & De Rijcke, 2016 Slide 15 of 51

SLIDE 16

Angular momentum

Hopkins, 2015 Slide 16 of 51 Springel, 2010

SLIDE 17

Moving mesh

Springel, 2010; BV & De Rijcke, 2016

Slide 17 of 51

SLIDE 18

Moving mesh

BV, public PhD presentation Slide 18 of 51

SLIDE 19

Moving mesh

Slide 19 of 51

SLIDE 20

A NOTE ABOUT COMPUTERS

Slide 20 of 51

SLIDE 21

Do computers get faster?

Slide 21 of 51

SLIDE 22

Do computers get faster?

Slide 22 of 51

SLIDE 23

Do computers get faster?

Verstocken et al., 2017 Slide 23 of 51

SLIDE 24

Do computers get faster?

YES, but only if your software

is incredibly parallel:

– a huge amount of independent computations – the same operation for many different values

uses very little memory at a time
generates very little output to hard disk

Slide 24 of 51

SLIDE 25

Do simulations get faster?

Gonnet, 2013

51,000,000 SPH particles with GADGET2 Short answer: NO

yes no Slide 25 of 51

SLIDE 26

Algorithmic problems

Neighbour finding with an octree

Gonnet, 2013 Slide 26 of 51

SLIDE 27

Algorithmic changes

Gonnet, 2013

Replace tree with regular grid to

exploit symmetries
split the data into small chunks
explicitly avoid conflicts

NON-TRIVIAL!

Slide 27 of 51

SLIDE 28

Task based parallelism

Gonnet, 2013 Slide 28 of 51

SLIDE 29

Task based parallelism

SPH mode (equal to GADGET2)

Slide 29 of 51

SLIDE 30

SWIFT vs GADGET2

Gonnet, 2013 Schaller, 2017 Slide 30 of 51

SLIDE 31

MESH-FREE HYDRODYNAMICS

Slide 31 of 51

SLIDE 32

Moving messes

BV & De Rijcke, 2016 Slide 32 of 51

SLIDE 33

Mesh-free volumes

Hopkins, 2015

Slide 33 of 51

SPH Moving-mesh Mesh-free

SLIDE 34

Mesh-free hydrodynamics

Mesh-free (SPH) Finite volume

Volumes Neighbours “Faces” Predictable: efficient Hydro: Second order reconstruction Riemann problem Accurate

Slide 34 of 51

SLIDE 35

SWIZMO

GADGET2 SPH GIZMO mesh-free

Slide 35 of 51

SLIDE 36

SWIZMO

Slide 36 of 51

SLIDE 37

Is mesh-free worth it?

The SWIFT hydro scheme comparison project

Slide 37 of 51

SLIDE 38

RADIATION

Slide 38 of 51 If out of time, click here…

SLIDE 39

Radiation

NGC 604

HST, 1995

Slide 39 of 51

SLIDE 40

Post-processing

BV et al., submitted to MNRAS

N. Sartorio

https://github.com/bwvdnbro/CMacIonize

BV & Wood, submitted to A&C

Slide 40 of 51

SLIDE 41

Radiation hydrodynamics (RHD)

BV & Wood, submitted to A&C

Slide 41 of 51

SLIDE 42

Algorithmic problems

Slide 42 of 51

SLIDE 43

A task-based alternative

PACKET GENERATION PACKET PROPAGATION PACKET REEMISSION nothing grid (part), grid (part),

IN OUT

Slide 43 of 51

SLIDE 44

A task-based alternative

Each grid (part) has 27 buffers:

1 internal buffer
6 direct (face) neighbours
12 indirect (edge) neighbours
8 indirect (corner) neighbours

The packet traversal task takes an INPUT buffer and deposits photon packets in the 27 OUTPUT buffers, according to the outgoing direction (absorbed photons are put in the internal OUTPUT buffer)

Slide 44 of 51

SLIDE 45

Example: serial run

60 × 60 × 60 cell grid, 60 subgrids 107 photon packets, no reemission 1 core, total run time: 151.757 s

Slide 45 of 51

SLIDE 46

Example: parallel run

32 cores, total run time: 10.1375 s 60 × 60 × 60 cell grid, 60 subgrids 107 photon packets, no reemission

Slide 46 of 51

SLIDE 47

Strong scaling

Naive parallelization Better parallelization

Slide 47 of 51

SLIDE 48

A few lessons I learned so far

One grid vs many subgrids not an issue:

equally efficient

Measuring performance is cheap!
Measuring load-balance is better than

guessing it

Lots of extra tricks (subgrid copies, premature

launching...), BUT mostly computer science

Slide 48 of 51

SLIDE 49

CONCLUSIONS

Slide 49 of 51

SLIDE 50

Which method should I use?

Slide 50 of 51

SLIDE 51

Which code should I use?

Slide 51 of 51