The 2007 High Performance Computing & Simulation (HPCS'07) Conference PANEL Parallel Processing Systems: Present and Future Trends Prague, Czech Republic, June 4-6, 2007 Experiences using Grid Computing Technologies to Solve Optimization Problems Departamento de Lenguajes y Ciencias de la Antonio J. Nebro Computación PhD in Computer Science Associate Professor Universidad de Málaga SPAIN Introduction Panel: Case study 1: Condor Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Summary • In our research group we are interested in applying parallelism to solve optimization problems using metaheuristics. • In this line of work, grid computing appears as a discipline providing a potential huge computing power that can be used to solve very hard optimization problems. • In this talk, I will comment our experiences using these systems and I will discuss about what we expect concerning grid computing and optimization in the future. 2 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro 1
Introduction Panel: Case study 1: Condor Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Table of Contents • Introduction • Case study 1: Condor and enumerative search • Case study 2: Condor, MW, and DNA sequencing • Case study 3: ProActive and optimizing the VRP • Conclusions and future plans 3 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro Introduction Introduction Panel: Case study 1: Condor Case study 1: Condor Case study 2: Condor and MW Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Case study 3: ProActive Present and Future Trends Conclusions and future plans Conclusions and future plans Experiences in parallel computing • MSc in Computer Science (1991) – Masther thesis: • “ Parallelization of a Branch and Bound Algorithm for Solving the TSP ” • Parallel computer: Transputers (16 nodes machine) • Parallel programming: Occam • PhD in Computer Science (1999) – PhD thesis: • “ A Distributed Runtime System for Implementing Concurrent Objects ” • Parallel computers: LAN of workstations, cluster of four Digital AlphaServer (4 CPU multiprocessors) • Parallel programming: sockets, PVM, POSIX threads (Pthreads) 4 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro 2
Introduction Introduction Panel: Case study 1: Condor Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Motivation • There many computers in the labs of the Computer Science Department of the University of Málaga – Currently about 300 processors • Question: – ¿How can we use them together to solve a problem? • Using known passing message libraries (sockets, PVM, MPI) is not a solution – Easy to use tool: heterogeneity, fault tolerance, dynamicity – Machines are idle in the nights and in the weekends – Variable availability during the day • SOLUTION: using grid technologies 5 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro Introduction Introduction Panel: Case study 1: Condor Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Motivation • Time table 6 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro 3
Introduction Introduction Panel: Case study 1: Condor Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Motivation • Key paper K. Anstreicher, N. Brixius, J.-P. Goux, and J. Linderoth Solving Large Quadratic Assignment Problems on Computational Grids . Mathematical Programming, 2002, Vol. 91, pp. 563–588. • Distributed branch and bound algorithm • 2500 processors • A hard instance of the QAP (Quadratic Assignment Problem) was solved. ESTIMATED TIME: seven years • Grid computing software: Condor 7 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro Introduction Panel: Case study 1: Condor Case study 1: Condor Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Case study 1: Condor and enumerative search A.J. Nebro, E. Alba, F. Luna. Multi-objective Optimization Using Grid Computing . Soft Computing, Vol. 11, No. 6. (April 2007), pp. 531-540. • Goals – To apply grid technologies to an enumerative search algorithm for multiobjective optimization – Heterogeneous distributed system – Grid system: Condor 8 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro 4
Introduction Panel: Case study 1: Condor Case study 1: Condor Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Case study 1: Condor and enumerative search • Condor (http://www.cs.wisc.edu/condor/) – Project of the University of Wisconsin – It is a distributed scheduler • Some features – Checkpointing – Remote system calls – Several universes (working modes): standard, vanilla, PVM, MPI, Java – Interoperable with Globus – Suitable to master/slave applications through the MW library – Opportunistic computing 9 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro Introduction Panel: Case study 1: Condor Case study 1: Condor Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Case study 1: Condor and enumerative search • Our requirements – Our application is composed of hundred/thousands of independent tasks – All the results (output files) are gathered offline – Heterogeneous machines 110 Processors 10 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro 5
Introduction Panel: Case study 1: Condor Case study 1: Condor Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Case study 1: Condor and enumerative search • Obtained results – Two hard problems were solved – Problem 1 • Time needed: 2 days • Total time reported by Condor: 90 days – Problem 2 • Timed needed: 27 days • Total time reported by Condor: 775 days • Analysis: – There is not dynamic load balancing – Condor is unable to migrate works between heterogeneous machines 11 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro Introduction Panel: Case study 1: Condor Case study 1: Condor Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Case study 1: Condor and enumerative search • Our experience – Condor is easy to install and administrate – Very easy to use if the tasks are independent • The programs do not need to be modified nor recompiled • Only linking with the Condor libraries is needed – Very stable • Few problems in this sense – Well documented – Opportunistic computing works – The lack of dynamic load balancing can be a problem 12 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro 6
Introduction Panel: Case study 1: Condor Case study 2: Condor and MW Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Case study 2: Condor, MW and Genetic Algorithms A.J. Nebro, G. Luque, F. Luna, E. Alba. DNA Fragment Assembly Using a Grid Based Genetic Algorithm . Computers and Operations Research. In Press. 2007 • Goals – To design a distributed genetic algorithm to solve an optimization problem from the biotechnology field – Grid system: Condor and the MW (Master/Worker) library • MW: C++ library allowing to develop master/slave applications on top of Condor • Not difficult to use if your are familiar with C++ 13 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro Introduction Panel: Case study 1: Condor Case study 2: Condor and MW Case study 2: Condor and MW Parallel Processing Systems: Case study 3: ProActive Present and Future Trends Conclusions and future plans Case study 2: Condor, MW and Genetic Algorithms • Our requirements – Designing a distributed genetic algorithm (GA) to optimize a biotechnology problem (DNA sequencing) – Ability to use up to 150 processors – Analysis of the master-worker scheme when applied to implement a grid-enabled distributed GA 14 - 24 Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro 7
Recommend
More recommend
