Progress towards development of discontinuous Galerkin - PowerPoint PPT Presentation

Progress towards development of discontinuous Galerkin finite-element methods for compressible flows using Charm++ Aditya K. Pandare † , Jozsef Bakosi ∗ , Hong Luo † † Department of Mechanical and Aerospace Engineering, North Carolina State University. ∗ Computer, Computational and Statistical Sciences, Los Alamos National Laboratory. 16th Annual Charm++ Workshop, 11-12 April 2018 AK Pandare, J Bakosi, H Luo DG methods using Charm++ 1 / 19

Overview ◦ Platform ◦ Physical system ◦ Numerical approximation ◦ Considerations for parallel computation ◦ Some examples ◦ Future plan AK Pandare, J Bakosi, H Luo DG methods using Charm++ 2 / 19

Code platform: Quinoa ◦ Written in modern C++11 and native Charm++ ◦ Fully asynchronous parallel programming ◦ Extensive unit and regression tested Capabilities: ◦ Continuous Galerkin solver for compressible fluid flow ◦ Fully unstructured tetrahedral mesh support ◦ Stochastic differential equation particle solver ◦ Dynamic load balancing capabilities using over-decomposition AK Pandare, J Bakosi, H Luo DG methods using Charm++ 3 / 19

The Physical System: from continuum equations to code ◦ Transport phenomena represented by system of (non-linear) PDEs ∂ U ∂t + ∂ F k = S ∂x k ◦ Multiply with a test-function and integrate over the entire domain: � ∂ U � � F k ( U ) ∂W j � ∂t W j dΩ + F k ( U ) n k W j dΓ − dΩ = S ( U ) W j dΩ ∂x k Ω Γ Ω Ω ◦ Choose a triangulation Ω e ∈ Ω with Γ e = ∂ Ω e , and then approximate the solution as: � U ( x ) ≈ U h ( x ) = u i B i ( x ) ∀ x ∈ Ω e i B i is a compact basis for the continuous function U on element Ω e . AK Pandare, J Bakosi, H Luo DG methods using Charm++ 4 / 19

High-order DG Solution polynomial: U h ( x ) = � ndof u i B i ( x ) i Order B i P0 [1] P1 [1 , ˜ x ] x 2 � � P2 1 , ˜ x, ˜ � x 2 , ˜ x 3 � P3 1 , ˜ x, ˜ AK Pandare, J Bakosi, H Luo DG methods using Charm++ 5 / 19

Discrete (weak) form of integral conservation laws ◦ The above weak form can be written per element using this approximation as: � ∂ u i �� � � F k ( U h ) ∂W j � B i W j dΩ ∂t + F k ( U h ) n k W j dΓ − dΩ = S ( U h ) W j dΩ ∂x k Ω e Γ e Ω e Ω e ∂ u i ∂t = R j M ij For Galerkin-type finite-element methods, W i = B i . This is the equation to be solved. ◦ Special case: First order approximation for U (Finite volume) : B 1 = 1 ∂ u h Ω e ∂t + F k ( u h ) n k Γ e = S ( u h )Ω e AK Pandare, J Bakosi, H Luo DG methods using Charm++ 6 / 19

Examples of PDE systems ◦ Scalar advection: U = φ, F = a φ ◦ Compressible Euler equations of gas dynamics:     ρ ρ u  , U = F = ρu i u + p ρu i    ρE ( ρE + p ) u AK Pandare, J Bakosi, H Luo DG methods using Charm++ 7 / 19

Time integration Explicit forward Euler discretization: u n +1 − u n i i = R j ( u n M ij h ) ∆ t ◦ Ensure time-step ∆ t is within CFL restrictions ◦ Compute volume and surface integrals for R j → most time-consuming ◦ Update vector of solution DOFs u h Time-stepping is done without using any global reductions, only using SDAG. AK Pandare, J Bakosi, H Luo DG methods using Charm++ 8 / 19

Riemann solvers ◦ Solution discontinuous at cell boundaries Γ e � ◦ How to uniquely define integrals Γ e F k n k dΓ e ? → Riemann solver. � � Γ e F k n k dΓ e ≈ Γ e F k ( u L , u R ) n k dΓ e ◦ ◦ Design based on physics: · Upwind · Downwind · Central-difference AK Pandare, J Bakosi, H Luo DG methods using Charm++ 9 / 19

Parallel communication for DG methods ◦ Riemann solver requires left and right states ◦ Requires communication at processor boundaries Communicated information 1) “Outer” element-id cor- responding to face-id 2) Geometry information of this element 3) Solution vector of this el- ement 4) → Communication map depends on the derived data AK Pandare, J Bakosi, H Luo DG methods using Charm++ 10 / 19

Derived data structures for unstructured meshes ◦ Elements surrounding elements ◦ Elements surrounding faces ◦ Vertices of faces Additionally, data structures required for parallel communication: ◦ Partition face-ids and corresponding “inner” element-id associated to chare-ids ◦ Mapping for local “outer” element-id to remote element-id AK Pandare, J Bakosi, H Luo DG methods using Charm++ 11 / 19

Preliminary results Linear advection of a Gaussian hump (on 3D Tetrahedral mesh) AK Pandare, J Bakosi, H Luo DG methods using Charm++ 12 / 19

Preliminary results for load balancing ◦ Perform excess computations (fake) on some chares ◦ Sanity-check of the chare-migration and load-balancing AK Pandare, J Bakosi, H Luo DG methods using Charm++ 13 / 19

Roadmap for DG 1) Extension of DG(P0) to coupled systems of PDEs 2) High-order discontinuous Galerkin discretization 3) p -refinement AK Pandare, J Bakosi, H Luo DG methods using Charm++ 14 / 19

p -refinement ◦ Order of solution polynomial raised/lowered based on errors/solution moments ◦ Initial mesh-based distribution on PEs now unbalanced ◦ Example † of unbalanced computational load: → Charm++ load balancers † From a separate research: Pandare A. Luo H., 2018, ”A robust and efficient finite volume method for compressible viscous two-phase flows”, In press . AK Pandare, J Bakosi, H Luo DG methods using Charm++ 15 / 19

High-order DG Ω e F k ( U h ) ∂B j � � ◦ Volume and surface integrals: Γ e F k ( U h ) n k B j dΓ − ∂x k dΩ ◦ Approximate both integrals using Gauss quadrature Example of surface integral over (green) face: � � → G ( U h )dΓ = ( w i G ( U i )) Γ e Γ e igp As order of solution polynomial increases: ◦ Number of integration points for an accurate numerical approximation increases. ◦ Computational effort per-element increases. AK Pandare, J Bakosi, H Luo DG methods using Charm++ 16 / 19

Causes of load-imbalance p -adaptation leads to the following: ◦ More number of unknowns to solve for, and larger problem size ◦ Gaussian quadrature requires more integration points at high-order zones Other possible sources: ◦ Expensive limiting strategies to preserve monotonicity for highly nonlinear problems. AK Pandare, J Bakosi, H Luo DG methods using Charm++ 17 / 19

Summary ◦ Ongoing development of a high-order p -adaptive discontinuous Galerkin method for fluid dynamics ◦ Dynamic load balancing using over-decomposition and migration using Charm++ ◦ Production-style coding AK Pandare, J Bakosi, H Luo DG methods using Charm++ 18 / 19

Thank you! AK Pandare, J Bakosi, H Luo DG methods using Charm++ 19 / 19