Adaptive Ray Tracing Jeong-Gyu Kim (Princeton) Mar 18, 2019 - PowerPoint PPT Presentation

Las Vegas Athena++ Developers Meeting Adaptive Ray Tracing Jeong-Gyu Kim (Princeton) Mar 18, 2019

Characteristics Method for 
 Point Source Radiative Transfer • Solves RT equation along selected rays and computes radiation 4-force vector directly Long characteristics Short characteristics Rijkhorst+06

Adaptive Ray Tracing • Spatially adaptive long characteristics • Split rays to match the angular resolution to that of grid cell • Angular discretization using HEALPix Abel+02 Gorski 05

Pros and Cons • Conservative of photon energy and momentum • Retains full information about the directionality of radiation flux • Best suited for UV radiation from a small number of massive stars • Photoionization/direct radiation pressure, coupling with chemistry • Ill-suited to handling di ff use sources/scattering • Computational cost scales as N src x N cell x N freq x m ray . • In practice, cost depends on problem geometry, domain decomposition, and parallel algorithm • Di ffi cult to parallelize

Science Applications Kado Fong+19 Rosen+16 Kim, J.-G.+18 • Massive star formation with radiation feedback (Rosen+16, 19) • Destruction of molecular clouds (Kim, J-.G.+17, 18, 19) • Post-processing kpc-scale galactic disk simulations 
 TIGRESS (Kim, C-.G.+17,18) • Di ff use ionized gas (Kado Fong+19, in prep) • H 2 and X CO (Gong+19, in prep) • TIGRESS with ART (see also Peters+17) • Escape of radiation (e.g., Wise+09, Kim, J-.h.+13)

<latexit sha1_base64="u1xw79g3CdR3vnRyiksXJtypqGg=">ACVXicbVFRSxtBEN67WquxrbF97MtiECLY9M4K9qUgLZQWBC0KmRjmNvMJYt7e8fuXCEc9yd9Kf0nfSm4FyOocWDh4/u+YWa+TQqtHEXR3yB8tvJ89cXaemvj5avXm+2tN2cuL63Evsx1bi8ScKiVwT4p0nhRWIQs0XieXH1t9PfaJ3KzS+aFTjMYGJUqiSQp0ZtLTKgaZJW32r+mQtliP/gYgpU3QmrnYG4uTDCfQeO6Ltq6POV5W7wVB2bV7S6279YfugSgUt5f7u6N2J+pF8+LIF6ADlvU6ah9Lca5LDM0JDU4N4ijgoYVWFJSY90SpcMC5BVMcOChgQzdsJqnUvMdz4x5mlv/Flz9n5HBZlzsyzxzmZf91hryKe0QUnp2GlTFESGnk7KC01p5w3EfOxsihJzwAaZXflcspWJDkP6LlQ4gfn7wMzvZ78cde9POgc/RlEcae8e2WZfF7JAdse/slPWZNfsXxAEYfAn+B+uhKu31jBY9LxlDyrcvAHv37LY</latexit> <latexit sha1_base64="SuB+vOaqwXoikfeJzku4R2aL1yI=">ACBXicbVA9SwNBEN3zM8avU0stFoNgFe5U0EYIamEZwXxA7gh7m7lkyd4Hu3NCGls/Cs2ForY+h/s/Ddukis08cHA470ZuYFqRQaHefbWlhcWl5ZLawV1zc2t7btnd26TjLFocYTmahmwDRIEUMNBUpopgpYFEhoBP3rsd94AKVFEt/jIAU/Yt1YhIzNFLbPvBuQCLzkGX0knq8J2iugJRtu+SUnQnoPHFzUiI5qm37y+skPIsgRi6Z1i3XSdEfMoWCSxgVvUxDynifdaFlaMwi0P5w8sWIHhmlQ8NEmYqRTtTfE0MWaT2IAtMZMezpW8s/ue1Mgwv/KGI0wh5tNFYSYpJnQcCe0IBRzlwBDGlTC3Ut5jinE0wRVNCO7sy/OkflJ2T8vO3VmpcpXHUSD75JAcE5eckwq5JVSI5w8kmfySt6sJ+vFerc+pq0LVj6zR/7A+vwBMsCXvQ=</latexit> <latexit sha1_base64="sZQBIRSbs3Qqb+QiL2TlVlvGiRs=">ACInicbVDLSsNAFJ3UV62vqEs3g0VwVRMV1IVQFEVXVrAPaEKZTCbt0MmDmYlQ0nyLG3/FjQtFXQl+jJM0C60eGDicy9z7nEiRoU0jE+tNDM7N79QXqwsLa+srunrGy0RxhyTJg5ZyDsOEoTRgDQlYx0Ik6Q7zDSdobnmd+J1zQMLiTo4jYPuoH1KMYSX19BPLR3KAEUsuUngKLdfjCdmuDUoGE19C1bnzSR8oc57Ol1ym4z3c06tGzcgB/xKzIFVQoNHT3y03xLFPAokZEqJrGpG0E8QlxYykFSsWJEJ4iPqkq2iAfCLsJD8xhTtKcaEXcvVUrFz9uZEgX4iR76jJLKSY9jLxP68bS+/YTmgQxZIEePKRFzMoQ5j1BV3KCZspAjCnKqsEA+QKkmqViuqBHP65L+ktV8zD2rG7WG1flbUQZbYBvsAhMcgTq4Ag3QBg8gCfwAl61R+1Ze9M+JqMlrdjZBL+gfX0Dgqmjow=</latexit> <latexit sha1_base64="bHRimhil2umH7WEB2m7vA85BPY=">AB8nicbVBNS8NAEN3Ur1q/qh69LBbBU0lU0GNRDx4rWFtIQtlsJ+3SzSbsToRS+jO8eFDEq7/Gm/GbZuDtj4YeLw3w8y8KJPCoOt+O6WV1bX1jfJmZWt7Z3evun/waNJc2jxVKa6EzEDUihoUAJnUwDSyIJ7Wh4M/XbT6CNSNUDjIE9ZXIhacoZX84BYkMhqAlN1qza27M9Bl4hWkRgo0u9WvoJfyPAGFXDJjfM/NMBwzjYJLmFSC3EDG+JD1wbdUsQRMOJ6dPKEnVunRONW2FNKZ+ntizBJjRklkOxOGA7PoTcX/PD/H+CocC5XlCIrPF8W5pJjS6f+0JzRwlCNLGNfC3kr5gGnG0aZUsSF4iy8vk8ezunded+8vao3rIo4yOSLH5JR45JI0yB1pkhbhJCXP5JW8Oei8O/Ox7y15BQzh+QPnM8fxKmQ6Q=</latexit> Ray-cell Interaction • Photon Packets ( PPs ) are transported radially outward from the source and attenuated by gas and dust. n · r I = � χ I Z E = 1 Id Ω = | F | /c c Z r Le − τ ( r, ˆ n ) / (4 π r 2 ) I ˆ n d Ω = ˆ F = ∆ ` L in ∂ L ray ∆ L = L in (1 − e − ∆ τ ) = − χ L ray ∂ r opacity per length ∆ ⌧ = � ∆ ` meshblock cell

Flow Chart ray_tracing advance_ray Inject rays inject rays and compute N dest,max traverse a cell calculating Δ r, 
 Δ L=L in (1-e - Δτ ), 𝓕 , and F N exit =0 Ray tracing call advance_ray () until my_PP_list is empty or N exit > N exit,max yes need to create split? children for each neighbor: Send to no if exit_PP_list is not empty: Neighbors update my_N dest or copy MPI_Isend PPs to neighbors yes exit this to exit_PP_list 
 meshblock? with N exit =N exit +1 no 1. check for incoming PPs Receive from no yes τ ν > τ ν ,max ? ( MPI_Iprobe and MPI_Recv ) update my_N dest for all ν ? 2. check for pending MPI_Isend request Neighbors (MPI_Testsome) exit 1. if my_N dest ≠ 0: Check if all rays add my_N dest to N dest,tot of root 
 are destroyed ( MPI_Fetch_and_op ), my_N dest =0 2. If N dest,tot of root eq N dest,max : or exits the update N dest,tot of other processors domain no yes exit N dest,tot =N dest,max ? Kim, J-.G.+17, Rosen+17

<latexit sha1_base64="hlkU2Llhek1T7IhNWf2GNwQU=">AB+HicbVBNS8NAEJ3Ur1o/GvXoZbEInkqiBb0IRS8eK9gPaEPYbDft0t0k7G6EGPpLvHhQxKs/xZv/xm2bg7Y+GHi8N8PMvCDhTGnH+bZKa+sbm1vl7crO7t5+1T47Kg4lYS2Scxj2QuwopxFtK2Z5rSXSIpFwGk3mNzO/O4jlYrF0YPOEuoJPIpYyAjWRvLtqvDzgRI4myKrlHDt2tO3ZkDrRK3IDUo0PLtr8EwJqmgkSYcK9V3nUR7OZaEU6nlUGqaILJBI9o39AIC6q8fH74FJ0aZYjCWJqKNJqrvydyLJTKRGA6BdZjtezNxP+8fqrDKy9nUZJqGpHFojDlSMdolgIaMkmJ5pkhmEhmbkVkjCUm2mRVMSG4y+vks53b2oO/eNWvOmiKMx3ACZ+DCJThDlrQBgIpPMrvFlP1ov1bn0sWktWMXMEf2B9/gCVZJf</latexit> <latexit sha1_base64="obm3n4oG19RFXQWwc07GXcN+hko=">AB+XicbVBNS8NAEJ34WetX1KOXxSJ4KkV9CIUvXisYD+gDWGz3bZLdzdhd1MIof/EiwdFvPpPvPlv3LY5aOuDgcd7M8zMixLOtPG8b2dtfWNza7u0U97d2z84dI+OWzpOFaFNEvNYdSKsKWeSNg0znHYSRbGIOG1H4/uZ35QpVksn0yW0EDgoWQDRrCxUui6Isx7SiCFsym6RTUvdCte1ZsDrRK/IBUo0Ajdr14/Jqmg0hCOte76XmKCHCvDCKfTci/VNMFkjIe0a6nEguogn18+RedW6aNBrGxJg+bq74kcC60zEdlOgc1IL3sz8T+vm5rBTZAzmaSGSrJYNEg5MjGaxYD6TFieGYJorZWxEZYWJsWGVbQj+8surpFWr+pdV7/GqUr8r4ijBKZzBfhwDXV4gAY0gcAEnuEV3pzceXHenY9F65pTzJzAHzifPwVlkpc=</latexit> Radiation in Vacuum m ray = 4 m ray = 20 Kim, J-.G.+17

Weak Scaling Strong Scaling 1pc • Constant amount of work per process • Fixed problem size (256 3 , 512 3 cells) (32 3 cells/proc) Kim, J-.G.+17

Thoughts • Need to minimize the overhead associated with finding neighboring meshblocks when multiple meshblocks are assigned to a single processor • Control flow with chemistry module • Update species abundances based on radiation field • Is operator split method with substepping best approach? • Hybrid RT • Aborption of UV radiation calculated by ART • Moment method to follow di ff use IR emission from dust • ART as an on-the-fly integration tool? • Synthetic observation, column density maps, etc.