MTAGS 2009 Many Task Computing for Multidisciplinary Ocean Sciences: Real-Time Uncertainty Prediction and Data Assimilation Constantinos Evangelinos Pierre F. J. Lermusiaux Chris Hill Jinshan Xu MIT Patrick J. Haley Jr. Earth, Atmospheric and Planetary Sciences MIT/Mechanical Engineering MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Motivation ● Improve the forecasting capabilities of ocean data assimilation and related fields via increased access to parallelism ● Move existing computational framework to a more modern, non-site specific setup ● Test the opportunities for executing massive task count workflows on distributed clusters, Grid and Cloud platforms. ● Provide an external outlet to handle peak- demand for compute resources during live experiments in the field ● Explore educational possibilities in the Cloud MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Ocean Data Assimilation dx = M (x, t) + dη ; M the model operator o = H (x k , t k ) + ε k ; H the measurement operator y k o ; dη, ε k , Q(t), R k ) ; J objective function min x J (x k ,y k Model errors are assumed Brownian: M dη = N(0,Q(t)) with E{dη(t) dη(t) T } = Q(t) dt In fact the models are forced by processes with noise correlated in space and time (meteo) Measurement errors follow white Gaussian: ε k = N(0, R k ) MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Ocean Acoustics Estimate of the ocean temperature and salinity fields (and uncertainties) necessary for calculating acoustic fields and their uncertainties. Sound-propagation studies often focus on vertical sections. Time is fixed and an acoustic broadband transmission loss (TL) field is computed for each ocean realization. A sound source of specific frequency, location and depth is chosen. The coupled physical-acoustical covariance P for the section is computed and non- dimensionalized and used for assimilation of hydrographic and TL data. MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Acoustic climatology maps TL STD over depth TL STD over bearing ( ) Mean Transmission Loss TL . 55dB 1 3 dB 3 dB 77km . 0 1 dB 65km 65dB . 0 1 dB effect of internal Effect of steep tides bathymetry Underwater acoustics transmission loss variability predictions in a 56 x 33 ● km area northeast of Taiwan. 2D propagation over 15km distance at 31x31 = 961 grid points X 8 directions ● Each job a short 3 minute acoustics 2D ray propagation problem ● Distributed on 100 dual-core computer nodes, speed up more than 100 ● times in real time experiment (SGE overhead of scheduling short jobs) MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Canyon Nx2D acoustics modeling – OMAS moving sound source Bathymetry of Mien Hua Canyon MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
AOSN-II Monterey Bay MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Error Subspace Statistical Estimation MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
ESSE Surf. Temp. Error Standard Deviation Forecasts for AOSN-II Leonard and Ramp, Lead PIs Aug 12 Aug 13 Aug 14 Start of Upwelling First Upwelling period Aug 24 Aug 27 Aug 28 End of Relaxation Second Upwelling period MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Serial and Parallel ESSE workflows MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
The ESSE workflow engine ● Is actually (for historical and practical reasons) a heavily modified C-shell script ( master )! – Catches signals to kill all remaining jobs ● Grid Engine, Condor and PBS variants – Submits and tracks singleton jobs ● Or uses job arrays for scalability – Further variants depending on I/O strategy: ● Separate pert singletons ? ● Input/output to shared or local disk (or mixed)? ● Shared directories store files with the execution status of each of the singleton scripts ● Singletons need the perturbation number:tricks! MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Multi-level parallelism in ESSE ● Nested ocean model runs (HOPS) are run in parallel – Limited parallelism – 2 or 3 levels – bi-directional ● SVD calculation is based on parallelizable LAPACK routines ● Convergence check calculation also. MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
ESSE and ocean acoustics ● As things stand ESSE is used to provide the necessary temperature and salinity information for sound propagation studies. ● The ESSE framework can also be extended to acoustic data assimilation. With significantly more compute power one can compute the whole “acoustic climate” in a 3D region – providing TL for any source and receiver locations in the region as a function of time and frequency, – by running multiple independent tasks for different sources/frequencies/slices at different times. MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Canyon Nx2D acoustics modeling Acoustics transmission loss difference in 6 hours (internal tides or other ● uncertainties) In future, incorporate with ESSE for uncertainties estimation, computation ● cost will be 1800 (directions) X 15 locations X HUNDREDS of cases. MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Ocean DA/ESSE/acoustics: MTC ● A minimum of hundreds to thousands (and with increased fidelity tens of thousands) of ocean model runs (tens of minutes or more) preceded by an equal number of IC perturbations (secs) ● File I/O intensive, both for reading and writing ● Concurrent reads to forcing files etc. ● Thousands of short acoustics runs (mins) ● Future directions for ESSE will generate even more tasks: – dynamic path sampling for observing assets – combined physical-acoustical ESSE MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
“Real-time” experiments MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Notable differences From many parameter sweeps and other MTC apps: ● there is a hard deadline associated with the execution of the ensemble worflow, as a forecast needs to be timely; ● the size of the ensemble is dynamically adjusted according to the convergence of the ESSE workflow which is not a DAG; ● individual ensemble members are not significant (and their results can be ignored if unavailable) - what is important is the statistical coverage of the ensemble; ● the full resulting dataset of the ensemble member forecastis required, not just a small set of numbers; IC are different for each ensemble members ● individual forecasts within an ensemble, especially in the case of interdisciplinary interactions and nested meshes, can be parallel programs themselves. MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
And their implications ● Deadline: use any Advanced Reservation capabilities available ● Dynamic: means that the actual total compute and data requirements for the forecast are not known beforehand and change dynamically ● Dropped members: suggests that failures (due to software or hardware problems) are not catastrophic and can be tolerated. Moreover runs that have not finished (or even started) by the forecast deadline can be safely ignored provided they do not collectively represent a systematic hole in the statistical coverage. ● I/O needs: mean that relatively high data storage and network bandwidth constraints will be placed on the underlying infrastructure ● Parallel ensemble members: mean that the compute requirements will not be insignificant either. MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Ocean DA on local clusters ● Local Opteron cluster – Opteron 250 2.4GHz (4GB RAM) compute nodes (single gigabit network connection) – Opteron 2380 2.5GHz (24GB RAM) head node – 18TB of shared disk (NFS) over 10Gbit Ethernet – 200Gbit switch backplane – Grid Engine and Condor co-existing ● Tried both GridEngine and Condor versions of ESSE workflows. Test 600 member ensemble: – I/O optimizations (all local dirs) 86 to 77 mins – SGE 10-20% faster than Condor ● without heroic tuning of the latter MTAGS 2009 MIT/EAPS & Mech.Eng. SC'09, Portland OR, Nov. 16 2009 C. Evangelinos (ce107@computer.org)
Recommend
More recommend
