Testing & Validation HIGH PERFORMANCE COMPUTING FRAMEWORK FOR - PowerPoint PPT Presentation

Modeling & Simulation, Testing & Validation HIGH PERFORMANCE COMPUTING FRAMEWORK FOR CO-SIMULATION OF VEHICLE – TERRAIN INTERACTION Radu Serban, Nicholas Olsen, Dan Negrut University of Wisconsin – Madison 8/22/2018

Modeling & Simulation, Acknowledgements Testing & Validation • University of Wisconsin – Madison – Antonio Recuero [Goodyear Tire & Rubber Co.] – Hammad Mazhar [NVIDIA] – Michael Taylor [UW and Harley-Davidson] • Outside collaborators – Hiroyuki Sugiyama [University of Iowa] – Bryan Peterson [University of Iowa] 8/22/2018

Modeling & Simulation, Off-Road Vehicle Mobility Testing & Validation 8/22/2018

Modeling & Simulation, Off-Road Vehicle Mobility: A Multi-Physics Problem Testing & Validation • Multibody vehicle system – Rigid body models of ground vehicles – Full vehicle subsystems (suspensions, steering, driveline, anti-roll bars, etc.) – Models for powertrain, driver (open/close loop) • Tire subsystem – Rigid tire – Empirical tire models (Pacejka, Fiala) Chassis – Flexible FEA tire models Suspension Wheels • Deformable terrain system Joints Internal Tire Forces – Tires Semi-empirical soil model (Bekker-Wong) Contact – Granular terrain (DEM) Terrain – Internal Terrain Forces Continuum soil models (FEA-based) • Fluid-Solid Interaction – Lagrangian-Lagrangian approach 8/22/2018

Modeling & Simulation, Project Chrono Testing & Validation • Growing ecosystem of software tools • Multi-physics simulation engine • Open source, under permissive BSD-3 license • Provides support for simulation of – Many-body dynamics – Nonlinear Finite Element Analysis – Fluid-Solid Interaction Problems • Middleware: can be embedded in third-party applications • Modular: based on optional linking of specialized modules • Expandable: via C++ inheritance • Efficient: fast and robust data structures and algorithms • Cross-platform: builds on Windows, Linux, OS X (MSVC, GCC, ICC, clang) Chrono: An open source multi-physics dynamics engine , HPC in Sci. and Eng. – Lecture Notes in CS, Springer, 2016 8/22/2018

Modeling & Simulation, Project Chrono - Organization Testing & Validation Chrono Support for Classical MBD Chrono::Engine API MBD Multi-Body Dynamics OpenMP Hardware Equation CPU, Multicore Formulation Chrono Support for Structural FEA API FEA And Volumetric Elements Equation Solution Hardware Chrono CUDA Support for Fluid-Solid API FSI Collision FSI Multiple GPU Interaction Detection HPC Chrono API Future Chrono Expansion … … Support Hardware MPI Multiple Nodes Pre/Post Processing Advanced Chrono Chrono Use Parallel HPC Chrono API Chrono Distributed Low-Entry Point Chrono Chrono Chrono Robotics Vehicle Granular Chrono Use 8/22/2018

Modeling & Simulation, Bottleneck – Computational Times Testing & Validation But what about everything in one simulation? 8/22/2018

Modeling & Simulation, Why Co-Simulation? Testing & Validation Flexibility and efficiency consideration: • Allow any combination of formulations for physics modeling – no suitable integration scheme for NSC involving FEA • Use different integration schemes as called for by the particular dynamics problem: – implicit, adaptive HHT scheme for FEA tires – a semi-implicit Euler scheme for the granular terrain • Allow each subsystem to advance its state using a suitable integration time step • Leverage different and independent parallelization techniques, as dictated by the structure of each subsystem: – multi-core (OpenMP) evaluation of FEA internal forces and Jacobians – multi-core (OpenMP), GPU (CUDA), or distributed (MPI) granular terrain simulation 8/22/2018

Modeling & Simulation, Co-Simulation of Single Tire on Granular Terrain Testing & Validation • Lower tire inflation pressure leads to enhanced mobility capabilities • Curve corresponding to rigid-mesh tire acts as limit envelope High fidelity approach for vehicle mobility simulation: Nonlinear finite element tires operating on granular material , J. Terramechanics, 2017 8/22/2018

Modeling & Simulation, Co-Simulation of Full Vehicle on Granular Terrain Testing & Validation • Vehicle mass: 2300 kg • Terrain: 932,000 particles (penalty) • Tires: 90x24 ANCF multilayer shell elements • Simulation: 7.6 s • Step size: 0.04 ms • Run time: 5.5 days • 40 core Intel Xeon CPU E5-2650 v3 @ 2.30GHz 8/22/2018

Modeling & Simulation, Co-Simulation of Full Vehicle on Granular Terrain Testing & Validation Initial drop causes Once rear tires run over settled terrain, they have large vertical contact forces larger net forces and lower resistance forces Rear tires fall onto front tires initial footprints A co-simulation framework for high-performance, high-fidelity simulation of ground vehicle — terrain interaction , Intl. J. Vehicle Performance, 2018 8/22/2018

Modeling & Simulation, Performance Bottleneck: Granular Terrain Simulation Testing & Validation Straight line maneuver over granular terrain Moving patch granular terrain option 8/22/2018

Modeling & Simulation, Chrono::Distributed – Philosophy and Design Testing & Validation • Philosophy and Assumptions – Chrono::Distributed is intended first and foremost for providing high- resolution terrain for mobility studies – Restricts to the penalty-based frictional contact model – Assumes load balancing is not a concern – Minimally wrap Chrono::Parallel – Create nearly-disjoint Chrono::Parallel simulations on each MPI rank • Let Chrono::Parallel handle simulation as it normally would with a synchronization step – Critically: Allow each Chrono::Parallel sub-system to perform its usual algorithm for collision detection on only the bodies in its sub-domain 8/22/2018

Modeling & Simulation, Chrono::Distributed – Domain Decomposition Testing & Validation • At setup: – Statically divide a predefined domain into subdomains for each MPI rank – Minimize inter-node communication with: – Non-blocking MPI – Point-to-point communications – No global collective operations • Three levels of parallelism – SIMD vectorization from AVX – Multi-core shared-memory parallelism from OpenMP – Multi-node distributed-memory parallelism from MPI 8/22/2018

Modeling & Simulation, Chrono::Distributed – Strong Scaling Results Testing & Validation Ranks Number Ratio Wall-clock Parallel Particles Time (s / s) Efficiency 1 1,236,372 1 23,490 – 2 1,236,372 1 11,827 0.993 4 1,236,372 1 5,954 0.986 8 1,236,372 1 2,773 1.059 16 1,236,372 1 1,440 1.020 32 1,236,372 1 712 1.031 𝑈(1) Parallel Efficiency 𝐹 𝑇 𝑜 = 𝑜𝑈(𝑜) Cray XC30; dedicated Cray Aries network nodes: 2 x 12 core Intel Xeon CPU E5-2697 v2 @ 2.7 GHz 8/22/2018

Modeling & Simulation, Chrono::Distributed – Weak Scaling Results Testing & Validation Ranks Number Ratio Wall-clock Parallel Particles Time (s / s) Efficiency 1 1,236,372 1 23,490 – 2 2,472,744 2 23,901 0.983 4 4,944,700 4 23,998 0.979 8 9,889,400 8 24,099 0.975 16 19,778,012 16 24,407 0.962 32 39,555,236 32 24,481 0.960 𝑈(1) Parallel Efficiency 𝐹 𝑋 𝑜 = 𝑈(𝑜) Cray XC30; dedicated Cray Aries network 64 nodes; 2 x 12 core Intel Xeon CPU E5-2697 v2 @ 2.7 GHz 8/22/2018

Modeling & Simulation, Chrono::Distributed – Wave Tank Demonstration Testing & Validation 8/22/2018

Modeling & Simulation, Distributed Co-Simulation Framework Testing & Validation 8/22/2018

Modeling & Simulation, Full Vehicle Simulations – Performance Testing & Validation Maneuver Acceleration Acceleration DLC DLC Tire model ANCF ANCF Rigid mesh Rigid mesh Domain size [m x m] 8 x 3 8 x 3 110 x 6 110 x 6 Particle radius [mm] 12.5 10.0 12.5 12.5 Number particles 283,162 591,090 6,513,518 6,513,518 Step size [ms] 0.04 0.04 0.04 0.04 Number MPI ranks 5 + 8 5 + 16 5 + 16 5 + 32 Average timing information [ms] Vehicle 0.93 0.93 0.92 0.92 Terrain 348.03 391.57 3953.67 1992.25 Tire (max overall) 3483.08 3488.10 1.46 1.41 Total 3547.53 3545.19 3966.21 2002.35 8/22/2018

Modeling & Simulation, Double Lane Change Maneuver on Granular Terrain Testing & Validation • Terrain: 6.5 million particles • Tires: rigid mesh • Simulation: 12 s • Step size: 0.05 ms • Run time: 66 hours on 5+64 nodes Extrapolation to higher resolution: • Terrain: 22 million particles • Tires: rigid mesh • Simulation: 12 s • Step size: 0.05 ms • Run time: 223 hours on 5+64 nodes Cray XC30; dedicated Cray Aries network nodes: 2 x 12 core Intel Xeon CPU E5-2697 v2 @ 2.7 GHz 8/22/2018

Modeling & Simulation, Acceleration Maneuver on Granular Terrain Testing & Validation Extrapolation to higher resolution: • Terrain: 22 million particles • Tires: ANCF mesh • Simulation: 10 s • Step size: 0.04 ms • Tire: ~3.4 s / step • Terrain: ~3.4 s / step • Run time: 236 hours on 5+64 nodes *Animation obtained with multi-core granular terrain simulation Cray XC30; dedicated Cray Aries network nodes: 2 x 12 core Intel Xeon CPU E5-2697 v2 @ 2.7 GHz 8/22/2018