A Parallel Strategy for a Level Set Simulation of A Parallel Strategy for a Level Set Simulation of Droplets Moving in a Liquid Medium Oliver Fortmeier, H. Martin Bücker VECPAR’10 June 24 th VECPAR 10, June 24 Berkeley, CA, USA
Motivation – Experimental Setup 2 E. Bertakis, S. Groß, J. Grande, O. Fortmeier, A. Reusken, and A. Pfennig. Validated simulation of droplet sedimentation with finite ‐ element and level ‐ set methods. Chemical Engineering Science , 65(6):2037 ‐ 2051, 2010 A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Outline 3 • Mathematical model for simulating droplets as two ‐ phase flows flows • Parallel simulation of droplets P ll l i l ti f d l t – Hierarchy of tetrahedral grids – Decomposition of adaptively refined tetrahedral grids D i i f d i l fi d h d l id – Distributed unknowns • Performance results up to 256 MPI processes – Load ‐ balancing – Strategies to place MPI processes on nodes A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Mathematical Model 4 • Level set functions describe two phases 1 (e.g., droplet) and 2 (e.g., surrounding water) 1D example for a level ‐ set function 2D example for a level ‐ set function • Coupling the level set equation and the Navier ‐ Stokes equations describes two ‐ phase flows d ib t h fl A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Typical Serial Simulation 5 Evolving Grid ‐ Solution Solution Solution for time for time for time t i t i+1 t i+1 in time mod. Vertical slice through computational domain Vertical slice through computational domain A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Need for Load ‐ Balancing 6 Evolving Grid Load Solution Solution Solution Solution for time for time for time for time t i t i+1 t i+1 t i+1 in time mod. balancing Vertical slice through computational domain Vertical slice through computational domain A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Distributed Hierarchy 7 s. Serial hierarchy of triangulations Serial hierarchy of triangulations esian Grids Lecture ger. r ve to Carte 5976 of L 010. Sprin gorithm fo , Berlin, 20 Parallel Alg om Adaptiv a, volume Level 0 Level 1 A Hybrid P ON, Canada nctions fro ges 48-61, Distributed hierarchy of triangulations lement Fun Kingston, O cience , pag M. Bücker. er and H. M CS 2009, K mputer Sc ng Finite El otes in Com n Proc HPC . Fortmeie ransformin Level 0 Level 1 No In Tr O A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Representing the Hierarchy by Graphs 8 • Dual, reduced Graph: – Vertices Vertices = = Family of tetrahedra T(v) i e all children Family of tetrahedra T(v) , i.e. all children of a parent tetrahedra – Edges Edges = Vertices are adjacent if corresponding = Vertices are adjacent if corresponding tetrahedron families share a common face – Vertex weight = Number of children or one if no children exists Vertex weight = Number of children, or one if no children exists – Edge weight = Number of common faces 2 1 4 A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Updating the Hierarchy of Triangulations 9 efin- ing Re i refine coarse Graph 2 1 4 2 1 4 A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
VECPAR‘10, June 24 th Oliver Fortmeier +Unknowns +Unknowns Updating the Hierarchy of Triangulations A Parallel Strategy for a Level Set Simulation of Droplets Moving in a Liquid Medium coarse refine 1 +Unknowns Re efin- i ing 2 1 4 Graph 2 +Unknowns 4 [ 2 2] O. Fort tmeier, T T. Henrich h, and H. M. Bück ker. Mode eling data a di istributio on for two o-phase f flow prob blems by weighted d gr raphs. In n Proc., 2 23rd Work kshop on n Parallel Sytems and 10 A lgorithms s, Hanno over, Germ many, pa ages 31-3 38. VDE, 2010.
Handling of Distributed Unknowns 11 • Unknowns are stored on vertices and U k d i d edges of the tetrahedra. • We distinguish between local and distributed unknowns. • Distributed unknowns are can be either – Partially assembled , i.e., each process only h ld holds a part of the value f h l P 0 P 1 – Accumulated , i.e., each process holds the “global” value global value • For linear algebra operations transforming partially assembled to transforming partially assembled to accumulated unknowns is needed P 2 P 3 • Transformation involves neighbor • Transformation involves neighbor communication which can be local unknowns overlapped by computations overlapped by computations di distributed unknowns ib d k A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Techniques to Solve Two ‐ Phase Flows 12 • Software – DROPS (http://www.igpm.rwth ‐ aachen.de/drops) • Computational domain – Hierarchy of tetrahedral grids for adaptively refining/coarsening / – Domain decomposition for parallelization • Numerical techniques: – Level set method to represent phases and to capture the interface Level set method to represent phases and to capture the interface – Finite elements (quadratic for velocity and level set function, linear and X ‐ FEM for pressure) p ) – Time integration is based on a linear theta scheme – Continuum surface force (CSF) for describing the surface tension – Parallel Krylov subspace methods (and in serial: iterative solvers based on multigrid methods) A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Is Updating a Bottleneck? 13 • Problem size: Speedup: – 875k tetrahedra, – 4 130k unknowns 30k k – radius 2mm, 5 levels • Total time: – 5 822s on 32 procs. – 893s on 256 procs. 893 2 • Updating the triangulation: – 12.6s (=0.22%) on 32 procs. – 3.7s (=0.41%) on 256 procs. Right now updating is Right now, updating is not a bottleneck Harpertown processors InfiniBand network Intel MPI library Intel compiler Harpertown processors, InfiniBand network, Intel MPI library, Intel compiler A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Placing MPI processes on nodes 14 Schematic view of two “Harpertown” processors on a node Schematic view of two “Harpertown” processors on a node: ssor0 ssor1 Core0 Core1 Core2 Core3 Core4 Core5 Core6 Core7 Proces Proces L2 L2 L2 L2 Bus Memory Two strategies to place MPI processes: 1 process per processor 4 process per processor compact p scatter A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Effect of Placement 15 1 node 4 nodes 16 nodes 32 nodes Problem size: 2 MPI 8 MPI 32 MPI 64 MPI – 155k tetrahedra 2104 s 533 s 277 s 75 s – 725k unknowns – Radius 2mm – Radius 2mm 8 MPI 8 MPI 32 MPI 32 MPI 128 MPI 128 MPI 256 MPI 256 MPI 1713 s 445 s 155 s 119 s – 4 level Factor 1.2 1.2 1.8 0.6 A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Conclusion and Outlook 16 • Conclusion: – Parallel strategy to simulate a raising drop – Parallel strategy to simulate a raising drop – Updating the triangulation is not a bottleneck – Data movement is expensive for Harpertown processors Data movement is expensive for Harpertown processors – Now, focus on optimizing parallel algorithms • Outlook: Outlook: – Parallel iterative mulitgrid solvers for the linear equation systems – Advanced techniques to “re ‐ initialize” the level set function Advanced techniques to re initialize the level set function – Hybrid MPI/OpenMP parallelization A Parallel Strategy for a Level Set Simulation Oliver Fortmeier of Droplets Moving in a Liquid Medium VECPAR‘10, June 24 th
Thank you for your attention! I am looking forward to I am looking forward to questions and discussions! http://www.sc.rwth ‐ aachen.de/fortmeier fortmeier@sc.rwth ‐ aachen.de http://www igpm rwth ‐ aachen de/drops http://www.igpm.rwth ‐ aachen.de/drops
Recommend
More recommend
