Some Organizing Principles for Coupling in Multiphysics and Multiscale Models J. Walter Larson ANU Supercomputer Facility The Australian National University Presented at the Biennial Computational Techniques and Applications Conference (CTAC ’06) Townsville QLD Australia 3 July 2006
Overview • Multiphysics and Multiscale Models • The Coupling and Parallel Coupling Problems • Some Organizing Principles • Conclusions + Future Work CTAC ’06, Townsville QLD Australia
Multiphysics Systems • A bold effort to move the practice of modeling from single systems in isolation to simulation of entire systems comprising multiple mutually interacting models • Examples abound: • Climate and Weather • Fusion device simulation • Space weather • Reactive flow (e.g., combustion) • Fluid structure interactions • Groundwater hydrology CTAC ’06, Townsville QLD Australia
Multiscale Systems • Physical systems that exhibit novel interacting phenomena that is basically the same type of physics on multiple time scales • Fluids exhibit this property • Weather (pictured) is a classic example of such multiscale modeling in operational use today Tropical Cyclone Larry 20 March 2006 03:55UT CTAC ’06, Townsville QLD Australia
The Ties that Bind • Multiphysics and Multiscale systems face a common challenge-- coupling • The Coupling Problem-- Data sharing and transformation in models that execute von Neumann style (i.e., single process) • The Parallel Coupling Problem-- Distributed data description, parallel data transfer and redistribution, and parallel data transformation in models that employ distributed-memory (a.k.a. message-passing parallelism) CTAC ’06, Townsville QLD Australia
Organizing Principle #0: The Parallel Coupling Problem can be split into two distinct parts: coupled model architecture and parallel data processing CTAC ’06, Townsville QLD Australia
Parallel Coupling Problem Coupled Model Architecture Parallel Data Processing • • Science of the system under study: Description of data to be exchanged during coupling • Connectivity--who talks to • Physical fields/variables whom? • • Mesh or representation associated Coupling event scheduling with the data (e.g., periodic?) • Domain decomposition • Domain overlap--lower- • Transfer of data--a.k.a. the MxN dimensional vs. colocation problem • Timescale separation/ • Transformation of data interaction & domain overlap • • Intermesh interpolation/ Tightness transformation between • representations (and associated Implementation choices: conservation issues) • Resource allocation • Time transformation • • Scheduling of model execution Diagnostic/variable transformations • Number of executables • Merging of data from multiple • Mechanism sources CTAC ’06, Townsville QLD Australia
Organizing Principle #1: The Parallel Data Processing part of the parallel coupling problem is largely solved--concentrate instead on parallel coupled model architecture CTAC ’06, Townsville QLD Australia
Organizing Principle #2: Graph Theory is relevant CTAC ’06, Townsville QLD Australia
Notions from Graph Theory • A coupled model has a directed graph representation • Nodes represent individual models/components • A directed arc from model A to model B represents a data dependency of input to model B on output from model A • In-valency of a node corresponds to number of distinct inputs from other models • Out-valency of a node corresponds to the number of distinct outputs to other models • The representation can not be factored into two or more disjoint graphs; otherwise one is representing multiple isolated coupled models • Typically, the directed graph representation is cyclic CTAC ’06, Townsville QLD Australia
A Famous Example C 3 C 1 Atmosphere Sea-Ice C 2 C 4 Ocean Land Coupled Climate Model (CCSM) CTAC ’06, Townsville QLD Australia
Organizing Principle #3: The matrix representation of Directed Graphs is relevant CTAC ’06, Townsville QLD Australia
Connectivity • Connectivity of a coupled model can be viewed as a directed graph, where nodes represent subsystem models and directed edges point from a source of coupling data to a target • Can be represented in terms of a connectivity matrix K � 1 , if C i receives data from C j , K ij = 0 , if i = j, or if C i does not receive data from C j • Model C is directly dependent on C if K = 1 i j ij • Model C is first-order indirectly dependent on C if K = 0 but i j ij C is directly dependent on another model C that is directly i k dependent on C j • Possible to define higher-order indirect dependencies as well CTAC ’06, Townsville QLD Australia
Organizing Principle #4: Coupling tightness can be quantified CTAC ’06, Townsville QLD Australia
The Load Matrix A logical starting point is the cost break-down structure for the coupled model. Define the load matrix L as � cost of running C i in isolation , if i = j, L ij = if i � = j cost of getting output from C j to C i where costs are defined in a suitable unit such as cpu node- seconds, and can be decomposed as L ij = R ij + S ji Cost to model i to Cost to model j to receive from model j send to model i CTAC ’06, Townsville QLD Australia
Tightness vs. Looseness • Tight and loose coupling are often mentioned, but, there is no clear definition • Can be defined following communications/compute ratio from HPC The tightness of the coupling between models i and j τ ij = τ ji = L ij + L ji L ii + L jj The tightness of an individual model’s coupling to the rest of the system is given by its coupling overhead � N j,j � = i ( S ij + R ij ) O i = L ii CTAC ’06, Townsville QLD Australia
Organizing Principle #5: Definition of terms for Resource Allocation and Component Scheduling CTAC ’06, Townsville QLD Australia
Resource Allocation and Component Scheduling • Single component multiple data (SCMD) approach, a.k.a. Serial Composition • Single global resource (e.g. MPI processes) pool • Sequential scheduling of components as an event loop • Example: NCAR/DOE Parallel Climate Model (PCM) • Advantage: Load balancing more straightforward • Disadvantage: “Weakest link” Amdahl’s law scaling limitations and wasted processor time • Multiple component multiple data (MCMD) approach, a.k.a. Parallel Composition • Partition global resource pool into disjoint sets • Components scheduled concurrently, each on its respective processor pool • Example: NCAR Community Climate System Model (CCSM) • Advantage: Mitigation of disparity in component scaling problem • Disadvantage: Intercomponent load balance challenge-- the stalling problem • Hybrid SCMD/MCMD CTAC ’06, Townsville QLD Australia
Organizing Principle #6: Definition of terms for Dynamic Load Balance in Parallel Coupled Models CTAC ’06, Townsville QLD Australia
Levels of Dynamic Load Strategy Intra-component Inter-component Global Level 0 Static Static Static (Static) Level 1 Dynamic Static Static Level 2 Dynamic Dynamic Static Level 3 Dynamic Dynamic Dynamic CTAC ’06, Townsville QLD Australia
Requirements Stemming from Dynamic Load Balance Cumulative growth in requirements as constraints on scheduling and resource allocation are loosened: Level 0: No additional requirements Level 1: Fast handshaking between components to cope with changing decompositions between coupling events Level 2: Ability of framework or coupling mechanism to checkpoint models and re-instantiate/restart them on their new processor pools Level 3: Ability of underlying communications mechanism to cope with dynamically varying global resource pool (e.g., dynamic MPI_COMM_WORLD) CTAC ’06, Townsville QLD Australia
Conclusions + Future Work • Multiscale and Multiphysics models almost always confront the parallel coupling problem • The data processing part of the parallel coupling problem is largely amenable to automation • The Model Architecture part of the problem has some organizing principles • Graph theory offers a good jumping-off point to define a number of quantities for parallel coupled models • Future work is to cement these notions and exploit them in larger systems comprising greater numbers of components CTAC ’06, Townsville QLD Australia
Recommend
More recommend
