PARALLEL SIMULATION IN TUNNEL ENGINEERING APPLICATION Hoang-Giang Bui and Günther Meschke Institute for Structural Mechanics Ruhr University Bochum PETSc User Meeting 2016 Vienna, Austria 28-30 June 2016
Outline • Motivation • Problem Description – Tunnel Simulation Model – Governing Equations – Discretization • Software Overview & Parallellization Implementation • Numerical Examples • Conclusions Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 2 PETSc UG 2016
Motivation • What is a tunnel for urban infrastructure? – Nothing but a tube under ground – Used for train traffic (frequently) – Or urban street (likely for the above-ground tunnel) • Why do we do tunnel simulation? – Understand the impact of tunnel construction on existing urban infrastructure (i.e buildings) – Compute the risk factor – Optimize the construction process parameters – Build a prediction model Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 3 PETSc UG 2016
Modern Tunneling Concept: Mechanized Tunneling Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 4 PETSc UG 2016
PROBLEM DESCRIPTION Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 5 PETSc UG 2016
Geometry Description of the Tunnel Representative Model FEM Model TBM Lining Ground Mortar grouting Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 6 PETSc UG 2016
Governing Equations • PDEs 𝑒𝑗𝑤 𝝉 𝑡 − 𝑞 𝑥 𝟐 + 1 − 𝑜 𝜍 𝑡 + 𝒐𝜍 𝑥 𝒉 = 0 𝐿 𝒗 𝑡 + 𝑒𝑗𝑤 𝜈 𝑥 −𝑠𝑏𝑒 𝑞 𝑥 + 𝜍 𝑥 𝒉 𝑒𝑗𝑤 = 0 • Stress-Strain relationship Elastoplastic Critical State 𝝉 𝑡 = 𝐷 𝑓 : 𝜻 − 𝜻 𝑞 𝝉 𝑡 = 𝐷 𝑓 𝑞 ′ : 𝜻 − 𝜻 𝑞 𝑞 ′ 1 𝑚𝑜 + 𝜃𝐽 3 𝝉 𝑡 − 𝜊𝑑 𝜻 𝑞 ≤ 0 𝑜 ′ 2 𝝉 𝑡 : 𝝉 𝑡 𝐾 2 𝑞 𝑝𝑘 𝑟 + ≤ 0 𝑁𝑞 ′ ln 𝑠 Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 7 PETSc UG 2016
Discretization • Initial boundary value problem 𝑔𝑗𝑜𝑒 𝑣 ℎ , 𝑞 ℎ ∈ 𝑉 ℎ , 𝑄 ℎ , ℒ 𝑣 ℎ , 𝑞 ℎ , 𝑤 ℎ , 𝑟 ℎ = 𝑔 𝑤 ℎ , 𝑟 ℎ ∀𝑤 ℎ ∈ 𝑊 ℎ , 𝑟 ℎ ∈ 𝑅 ℎ • Q2-P1 discretization 𝒲 ℰ 𝑑 𝒲 ℰ 𝑣 ℎ = 𝑞 ℎ = 𝑂 𝑗 𝑣 𝑗 𝑂 𝑗 𝑞 𝑗 • Galerkin method: = 𝐿 𝑣𝑣 𝐿 𝑥𝑣 𝐿 𝑗𝑘 = ℒ 𝑂 𝑗 , 𝑂 𝑘 𝐿 𝑣𝑥 𝐿 𝑥𝑥 𝑗𝑘 In which: 𝑓 = Ω 𝑓 𝐶 𝑈 𝐸 𝑓 𝐶 𝐾 𝑒𝑌 𝑓 Ω 𝑓 𝐶 𝑈 𝐽 𝑤 𝑂 𝑞 𝐾 𝑒𝑌 𝐿 𝑣𝑣 𝐿 𝑣𝑥 = − Ω 𝑓 𝐶 𝑈 𝐿 𝑓 𝑓 𝐿 𝑥𝑣 = 0 𝐿 𝑥𝑥 = − 𝜈 𝑥 𝐶 𝐾 𝑒𝑌 Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 8 PETSc UG 2016
Material Inhomogeneity E ~ 1e6 MPa E ~ 1e3~1e9 MPa Ground Ground E ~ 3e10 MPa Mortar Lining Grouting Support Ground Ground Excavated soil E ~ 2.1e11 MPa Ground Ground Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 9 PETSc UG 2016
PARALELLIZATION Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 10 PETSc UG 2016
Software Infrastructure [1] MortarApplication + Mortar Tying/Contact MetisApplication + Domain Decomposition DistributedBuildersApplication + Parallel Assembly (MPI) PetscSolversApplication + Parallel Linear Algebra (solver/preconditioner) [1] Dadvand & Rossi et al, An Object-oriented Environment for Developing Finite Element Codes for Multi-disciplinary Applications, DOI 10.1007/s11831-010-9045-2 [2] Balay et al, PETSc user manual Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 11 PETSc UG 2016
Software Infrastructure Heavy used of: + boost shared_ptr + Template + boost ublas Python interface Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 12 PETSc UG 2016
Software Design + domain decomposition MortarApplication + Mortar Tying/Contact MetisApplication + Domain Decomposition + wrappers for Mat, Vec DistributedBuildersApplication + preallocation + Parallel Assembly (MPI) + ghost layer communication PetscSolversApplication + Parallel Linear Algebra (solver/preconditioner) + construct preconditioner + call KSPSolve Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 13 PETSc UG 2016
NUMERICAL EXAMPLES Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 14 PETSc UG 2016
Numerical Example 1 • Fracture Simulation using Phase Field method – # nodes: 1.759.004 – # elements: 10.188.671 – # dofs: 5.277.012 – MPI processes = 64 – GMRES + BoomerAMG – Staggered solver • Displacement field: ~120s • Phase field fracture: ~18s Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 15 PETSc UG 2016
Numerical Example 2 • Reference Tunnel Project 1 o One phase (pure displacement) discretization o No contact between the Tunnel Boring Machine and the soil o 433.298 nodes o 303.398 tets o 1.205.521 dofs o 8 Mpi processes o GMRES + BoomerAMG + diagonal scaling Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 16 PETSc UG 2016
Numerical Example 2 • Reference Project 1 Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 17 PETSc UG 2016
Numerical Example 2 • Reference Project 1 MKL Pardiso PETSc - BoomerAMG Saving: 2d -> 3hrs Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 18 PETSc UG 2016
Numerical Example 3 • Reference Project 2 o Two phase discretization o Critical State Soil Model o Penalty Contact between the Tunnel Boring Machine and the soil o 8 Mpi processes o Direct solver: MUMPS Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 19 PETSc UG 2016
Numerical Example 3 • Reference Project 2 Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 20 PETSc UG 2016
Numerical Example 3 • Timing with different mesh #nodes 88724 171882 173024 335202 #elements 10256 20240 20240 40480 Avg Solve Time (s) 66,79 406,1 154,81 1108,5 Normalized Avg Solve Time/#Nodes 1 0,319 0,841 0,228 Normalized Avg Solve Time/#Elements 1 0,325 0,851 0,238 Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 21 PETSc UG 2016
Tunnel Simulation With Isogeometric Method Standard Isoparametric Isogeometric Finite Finite Element Element VS Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 22 PETSc UG 2016
Tunnel Simulation With Isogeometric Method Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 23 PETSc UG 2016
Conclusions • The paralellization works and produces expected results • One phase solve => AMG works as expected • MUMPS does not scale • Future works: – Development of block preconditioner for two-phase and contact problem – Tuning multigrid solver – Integrate SNES Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 24 PETSc UG 2016
THANK YOU ! QUESTIONS ? This work is part of sub-project C1, within Collaborative Research Centre SFB837 - http://sfb837.sd.rub.de - Interaction Modeling in Mechanized Tunneling - Ruhr University Bochum, Germany Hoang-Giang Bui Parallel Simulation in Tunnel Engineering Application 25 PETSc UG 2016
Recommend
More recommend
