Multicore job management in the Multicore job management in the Worldwide LHC Computing Grid Worldwide LHC Computing Grid EGI Community Forum, Helsinki, May 20th 2014 EGI Community Forum, Helsinki, May 20th 2014 Antonio Pérez-Calero Yzquierdo and Alessandra Forti for the WLCG Multicore deployment Task Force
Outline Outline ● Multicore applications in WLCG ● Job submission across the WLCG ● The problem of multicore job scheduling ● The WLCG multicore deployment TF ● First results and current status ● Conclusions and Outlook 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 2
Jobs in WLCG Jobs in WLCG LHC experiments need a global computing infrastructure in order to analyze LHC ● collisions: the Worldwide LHC Computing Grig A distributed computing model with data and job submission across the Grid in ● order to process billions of collision events Computation tasks include experimental data reconstruction and analysis as ● well as event simulation . Mainly sequential tasks, one event at a time, with no parallelization, single core ● jobs 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 3
Multicore jobs in WLCG Multicore jobs in WLCG Looking at the restart of the LHC data taking in 2015, experiments are developing multicore applications due to: Hardware evolution : over the last decade architecture design goes in the ● direction of adding processors to the CPU, while individual core performance will probably not increase significantly Evolution of LHC conditions : higher data volumes to be processed, with ● increased event complexity due to higher pileup, causing increasing processing time per event – memory usage – New era for HEP computing: Integration of elements of Grid Computing and High Performance Computing ● going from sequential programming to parallel processing over the Grid: distributed parallel computing Parallel view to other activities: ● LHC upgrades ↔ LHC detectors upgrades ↔ LHC VOs software upgrades 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 4
Multicore jobs in WLCG Multicore jobs in WLCG Advantages for multicore jobs: Fully exploit future CPU capabilities , adapting code to new architecture designs ● Reduced tasks memory consumption per core, as memory may be shared ● between threads Parallel processing is being considered at different levels : Run over events in parallel processes ● Process data modules inside an event in parallel ● Both combined: processing in parallel modules not necessarily from the same ● event Jobs running parallel threads share common data in memory , such as detector geometry, calibration and conditions data, etc. 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 5
Multicore job scheduling problem Multicore job scheduling problem Objectives: Integrate scheduling of both multicore and single- ● core jobs , that will still be used by LHC experiments, as well as other VOs in shared sites. Avoid splitting resources , such as dedicated whole ● node slots and separated queues, which may introduce additional inefficiency and complexity in site resources configuration and management. Maximize CPU usage : No idle CPUs while there is ● job to be done 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 6
Pilot based job submission Pilot based job submission LHC experiments commonly make use of pilot jobs: ● reserve resources at the remote computing centers – once pilots get resources, they start pulling jobs from a general job pool, – pulling jobs to be run at their location In this schema, multicore jobs require multicore pilots ● Ex. glideinWMS based on HTCondor (CMS) 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 7
WLCG Multicore Deployment TF WLCG Multicore Deployment TF Job scheduling involves two main elements : a) Grid-wide job submission by experiments b) Resource allocation at the sites The purpose of the WLCG Multicore Deployment TF is to explore, develop and propose ways to connect a) and b) in the most efficient way, with reasonable effort from sites and experiments, and in a reasonable time in order to achieve our multicore job scheduling objectives. 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 8
WLCG Multicore Deployment TF WLCG Multicore Deployment TF Evaluate: Multicore capabilities of local batch systems ● Compatibility of approaches to multicore job distribution by different ● LHC VOs This contribution: summary of the activities of this task force over the last months. ● Acknowledgements: thanks to all the participating people and sites, which provided the content for this talk! Project twiki: https://twiki.cern.ch/twiki/bin/view/LCG/DeployMultiCore 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 9
Review of batch systems systems Review of batch We have reviewed batch systems in terms of their functionalities useful for ● multicore scheduling Experience related to: ● ATLAS multicore jobs in production since January – CMS limited testing up to now – Mini workshops dedicated to each technology ● HTCondor (RAL), UGE (KIT), Torque/Maui (NIKHEF), SLURM (CSCS) – Main conclusion: most popular batch systems support multicore jobs ● Native functionalities plus sometimes complementary scripts – System configuration ( tuning ) depends on site load composition and running ● conditions: we will need more than one iteration to fully evaluate the performance of each system 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 10
Scheduling multicore jobs Scheduling multicore jobs Key problem: in order for a multicore job to start in a non-dedicated environment, ● the machine needs to be sufficiently drained Creating a multicore slot: prevent single core jobs from taking freed resources ● draining = idle CPUs!! – 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 11
Scheduling with backfilling Scheduling with backfilling However, a well tuned scheduler doing backfilling can reduce the amount of idle CPUs caused by the WN draining: Jobs of lower priority are allowed to utilize the reserved resources only if their ● prospective job end (i.e. their declared wallclock usage) is before the start of the reservation job job job BACKFILLED JOBS 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 12
Scheduling with backfilling Scheduling with backfilling The ability of the scheduler algorithm to perform successful backfilling depends on the concepts of entropy and predictability Entropy : having a variety of jobs with different requirements in the ● queue. There should be a distribution of jobs resources requests in order to increase the likelihood of finding the right "piece" to fill each temporary hole in draining WNs Predictability : reasonably accurate prediction for jobs running time, so ● that the scheduler can make a decision on whether it should run this job in that hole or not. – How accurate this prediction needs to be? 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 13
Job running time estimation Job running time estimation Providing a reliable estimation of job running times is however difficult for various reasons: Inherent to the jobs themselves , as the instantaneous luminosity and pile-up ● determine the complexity of events and thus the job running time different for analysis, MC production and data reconstruction/reprocessing – there are currently ways to mitigate this, for example data reconstruction – workload distributed in a number of jobs with approximately equal running time Access to input data waiting times : unpredictable in a complex environment ● such as the WLCG Variance in CPU power for WNs distributed across the grid and also within sites ● This may not be so much of a problem if the actual different between the faster – and lower machines at a given site still provides an estimation accurate enough to do some backfilling The masking effect of pilots: submission of jobs through pilots introduce some ● other effects, such as running more than one job per pilot, waiting for new jobs to appear, etc. 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 14
Conserving the slots Conserving the slots There are two aspects of the problem: creating and conserving multicore slots ● Once the cost has been paid, avoid multicore slot destruction – VO:1 job VO:2 job 20-05-2014 Multicore job management in WLCG - Antonio Pérez-Calero Yzquierdo 15
Recommend
More recommend
