& Grid5000 Grid eXplorer eXplorer Grid Plates-formes de - PDF document

Grid'5000 and Grid eXplorer 1 GdX GdX Grid’5000 & Grid’5000 Grid eXplorer eXplorer Grid Plates-formes de Grilles expérimentales mutualisées à l’échelle nationale ACI GRID & ACI MD Franck Cappello INRIA fci@lri.fr Avec tous les membres participants ACI Masse de ACI GRID Données Grid'5000 and Grid eXplorer 2 GdX GdX Grid’5000 Agenda • Rational • Grid’5000 • Grid eXplorer

Grid'5000 and Grid eXplorer 3 GdX GdX Grid’5000 Grid experimental platforms rational Grid raises a lot of research issues: Security, Performance, Fault tolerance, Scalability, Load Balancing, Coordination, Message passing, Data storage, Programming, Algorithms, Communication protocols and architecture, Deployment, etc. Theoretical models and simulators cannot capture real life or real size conditions Production platforms have strong difficulties to reproduce experimental conditions How to test and compare? • Fault tolerance protocols • Security mechanisms • Networking protocols • etc. Grid'5000 and Grid eXplorer 4 GdX GdX Grid’5000 Tools for Distributed System Studies To investigate Distributed System issues, we need: 1) Tools (model, simulators, emulators, experi. Platforms) 2) Strong interaction between these research tools Tools for Large Scale Distributed Systems log(cost) Real system s Real system s Real applications Real applications “I n- lab” platform s Real platform s Synthetic conditions Real conditions Models: Sys, apps, Key system m ecas. Platform s, Algo, app. kernels conditions Virtual platform s Synthetic conditions log(realism ) m ath sim ulation em ulation live system s

Grid'5000 and Grid eXplorer 5 GdX GdX Grid’5000 Grids are complex dist. Systems. • Systems – nodes, OS, – distributed systems mechanisms (resource discovery, storage, Static scheduling, etc.), – middleware, runtimes, – Fault (crash, transient) – Workload (multiple users/ multiple applications) Dyn. – Heterogeneity (resource diversity, performance) – Malicious users/ behaviors • Networks – routers, links, topology, – protocols, Static – Theoretical features: synchronous, pseudo synchronous or asynchronous – Disconnection Dyn. – Packet loss – Congestion Grid'5000 and Grid eXplorer 6 GdX GdX Grid’5000 Existing Grid Research Tools • SimGRid and SimGrid2 • Discrete event simulation with trace injection • Originally dedicated to scheduling studies • GridSim • Australian competitor of SimGrid • Dedicated to scheduling (with deadline) • Titech Bricks • Discrete event simulation for scheduling and replication studies • MicroGrid • Emulator with MPI communications • Not dynamic � No emulator or real life experimental platform � These tools do not scale (limited to ~ 100 grid nodes) � They do not consider the network issues (almost)

Grid'5000 and Grid eXplorer 7 GdX GdX Grid’5000 We need Grid experimental tools According to the current knowledge: There is no large scale testbed dedicated to Grid experiments � Grid’5000 as a real life system � Grid eXplorer as a large scale emulator log(cost) Grid’5 0 0 0 TERAGrid PlanetLab Grid eXplorer Naregi Testbed WANinLab Emulab Sim Grid MicroGrid Bricks NS, etc. Model Protocol proof log(realism ) m ath sim ulation em ulation live system s Grid'5000 and Grid eXplorer 8 GdX GdX Grid’5000 A New Generation of Research Tools Grid’5000

Grid'5000 and Grid eXplorer 9 GdX GdX Grid’5000 NAREGI Middleware Development Infrastructure • Under Procurement – Installation planned in Dec. 2003 – 3 SMPs, 128 procs total – 6 x 128-proc clusters, with different interconnects – 1 File Server – Multi-gigabit networking to simulate Grid Environment – NOT a production system (c.f. TeraGrid) – Mainly geared towards R&D, but could be used partially for experimental production – ~ 5 Teraflops – To form a Grid with the IMS NAREGI application testbed infrastructure (~ 10 Teraflops, March 2004), and other national centers(voluntary basis) via SuperSINET Grid'5000 and Grid eXplorer 1 0 GdX GdX Grid’5000 Netbed (Emulab) Comme « Modelnet » Description de la topologie => NS script Utilisation de Dummynet Outils de Mapping routeur_logique => machine physique Utilisation du simulateur NSE (ns emulation) Utilisation de noeuds exterieurs client (40 DSL) 3 Sites

Grid'5000 and Grid eXplorer 1 1 GdX GdX Grid’5000 GNET-1 Grid'5000 and Grid eXplorer 1 2 GdX GdX Grid’5000 What do we need for Grid experiments ? 1) Remotely controllable Grid nodes installed in geographically distributed laboratories 2) A « Controllable » and « Monitorable » Network between the Grid nodes � (may be unrealistic in some cases) 3) A middleware infrastructure connecting allowing users to access the Grid sites 4) A playground to prepare experiments 5) A toolkit to deploy, run, monitor, control experiments and collect results We need these components for a nation wide experimental platform

Grid'5000 and Grid eXplorer 1 3 GdX GdX Grid’5000 Agenda • Rational • Grid’5000 • Grid eXplorer Grid'5000 and Grid eXplorer 1 4 GdX GdX Grid’5000 The Grid’5000 Project 1) Building a nation wide experimental platform for Grid researches (like a particle accelerator for the computer scientists) • 8/ 10 geographically distributed sites • every site hosts a cluster (from 256 CPUs to 1K CPUs) • All sites are connected by RENATER (French Res. and Edu. Net.) • RENATER hosts probes to trace network load conditions • Design and develop a system/ middleware environment for safely test and repeat experiments 2) Use the platform for Grid experiments in real life conditions • Address critical issues of Grid system/ middleware: • Programming, Scalability, Fault Tolerance, Scheduling • Address critical issues of Grid Networking • High performance transport protocols, Qos • Port and test applications • Investigate original mechanisms • P2P resources discovery, Desktop Grids

Grid'5000 and Grid eXplorer 1 5 GdX GdX Grid’5000 Participants Grid’5000 Jean-Luc ANTHOINE Jean-Claude Barbet Pierrette Barbaresco Nicolas Capit Eddy Caron Christophe Cérin Olivier Coulaud Georges Da-Costa Yves Denneulin Comité de Pilotage (11) : Benjamin Dexheimer Aurélien Dumez -Franck Cappello (animateur) Gilles Gallot -Thierry Priol (Directeur directeur ACI Grid) David Geldreich -Brigitte Plateau (Directrice CS ACI Grid) Comité Technique Sébastien Georget -Dani Vandrome (Renater) Olivier Gluck -Frédéric Desprez (Lyon) étendu (28) : Claude Inglebert -Michel Daydé (Toulouse) -Yvon Jégou (Rennes) Julien Leduc -Stéphane Lantéri (Sophia) Cyrille Martin -Raymond Namyst (Bordeaux) Jean-Francois Méhaut -Pascale Primet (Lyon) Jean-Christophe Mignot -Olivier Richard (Grenoble) Thierry Monteil Guillaume Mornet Alain Naud Forums: Vincent Néri Gaetan Peaquin Deploym ent/ exploitation: Franck Simon Franck Cappello (AS1, RTP8) Sebastien Varrette Program m ing m odels: Jean-Marc Vincent Raymond Namyst (AS2, RTP8) Grid'5000 and Grid eXplorer 1 6 GdX GdX Grid’5000 www.grid5000.org

Grid'5000 and Grid eXplorer 1 7 GdX GdX Grid’5000 Grid'5000 and Grid eXplorer 1 8 Grid’5000 in November’2004 GdX GdX Grid’5000 Funding: ACI Grid, ACI MD, INRIA, CNRS, Reg. Grid eXplorer Pau Grid 5000 nodes � Connexion des sites à 1 Gbits 3 ~ 4M€ for hardware only (including 2004 fund)

Grid'5000 and Grid eXplorer 1 9 GdX GdX Grid’5000 Planning Grid’5000 today Call for Vendor Instal. Final Fisrt Call for Selection ACI GRID Expression selection First review Demo proposals of 7 sites Funding Of Interest tests (SC04) Grid’5000 Hardware Switch Proto to Grid’5000 System/ middleware Forum Grid’5000 Security Prototypes Control Prototypes Renater connection Grid’5000 Grid’5000 Programming Forum Demo preparation Grid’5000 Experiments March04 Jun/ July 04 Spt 04 Oct 04 Nov 04 Sept03 Nov03 Jan04 Grid'5000 and Grid eXplorer 2 0 GdX GdX Grid’5000 Summary of Grid5000 experiments of Grid’5000 members • Networking – End host communication layer (interference with local communications) – High performance long distance protocols (improved TCP) – High Speed Network Emulation • Middleware / OS – Grid’5000 control/ access – Grid’5000 experiment automation – Scheduling / data distribution in Grid – Fault tolerance in Grid – Resource management – Grid SSI OS and Grid I/ O – Desktop Grid/ P2P systems • Programming – Component programming for the Grid (Java, Corba) – GRID-RPC – GRID-MPI – Code Coupling • Applications – Multi-parametric applications (Climate modeling/ Functional Genomic) – Large scale experimentation of distributed applications (Electromagnetism, multi-material fluid mechanics, parallel optimization algorithms, CFD, astrophysics – Medical images, Collaborating tools in virtual 3D environment