sweep applications P. Kacsuk MTA SZTAKI 1 The EDGI/EDGeS projects - PowerPoint PPT Presentation

Extending gLite VOs with volunteer and institutional BOINC-based desktop grids to execute parameter sweep applications P. Kacsuk MTA SZTAKI 1 The EDGI/EDGeS projects receive(d) Community research funding

Source: http://europa.eu/volunteering Presentation for: ISGC2011 Author: Peter Kacsuk

First great achievement of volunteer computing • SETI@home: analyzes data from the Arecibo radio telescope, looking for Extra-Terrestrial Intelligence in radio signals. – 3.8M volunteers from 226 countries (2004) – 1200 CPU years/day – 38 TFlops sustained performance (Japanese Earth Simulator 32 TFlops) Presentation for: ISGC2011 Author: Peter Kacsuk

Potential of volunteer computing • Number of privately owned PCs: – currently 1.5 billion – grow to 2 billion by 2015 • ~1 million computers are actively participating in volunteer computing à supplying ~10PetaFLOPS of computing power • Fastest supercomputers supply few PetaFLOPS • Largest grids (e.g. EGI) have several hundreds of thousands of hosts. • In the area of so-called bag of tasks, parameter sweep applications volunteer computing is competitive. • Near-term potential of volunteer computing goes well beyond Exa- scale. Presentation for: ISGC2011 Author: Peter Kacsuk

Why volunteer desktop grids are important? Volunteer desktop grids can collect very large number of resources http://knowledgebase.e-irg.eu Presentation for: ISGC2011 5 Author: Peter Kacsuk Introduction � RI-261561 WP4 � Author: Robert Lovas, Ad Emmen � version: 1.0 �

The Grid Ecosystem • Very expensive, • small number of sites and Supercomputer based SGs very large number of cores (TeraGrid, DEISA) • MPI appls • Moderately expensive, Cluster based • moderate number of sites service grids (SGs) and CPUs (EGI, OSG, etc.) EDGeS • any appls • Inexpensive, Desktop grids (DGs) • very large number of (volunteer DGs – home computers, CPUs (~10K – 1M) organizational DGs – institutional desktops) • Param. sweep appls Presentation for: ISGC2011 Author: Peter Kacsuk

EU FP7 projects on desktop grids: EDGeS –> EDGI and DEGISCO EDGI EDGeS • ARC, Unicore, Clouds further • DG<->SG integration: d eveloped by • QoS with Clouds • gLite → BOINC, • Data intensive apps XtremWeb • SG->DG direction • BOINC, support XtremWeb → supported by gLite • Compute intensive DEGISCO applications • Disseminate EDGeS results world-wide • Green IT aspects 2008 - 2010 2010 - 2012 Presentation for: ISGC2011 Author: Peter Kacsuk 7

Types of Desktop Grids • Global (volunteer) Desktop Grid • Aim is to collect resources for grand-challenge scientific problems • Examples: • SETI@home, Folding@home, Shakemovie@home, LHC@home • Community World Grid, IberCivis, SZTAKI Desktop Grid • Local (institutional) Desktop Grid • Aim is to enable the quick, easy and inexpensive creation of grid for any community (company, university, etc.) to solve their own applications • Example: • SZTAKI Desktop Grid (SZDG) local version (used within EDGeS, EDGI, DEGISCO) Presentation for: ISGC2011 8 Author: Peter Kacsuk

Local DGs in practice – University of Westminster DG system based on SZDG 6 1 2 5 4 1. New Cavendish Street 576 nodes 3 2. Marylebone Campus 559 nodes 3. Regent Street 395 nodes Lifecycle of a node: 4. Wells Street 31 nodes 1. PCs basically used by students/staff 5. Little Tichfield Street 66 nodes 2. If unused, switch to Desktop Grid mode 6. Harrow Campus 254 nodes 3. No more work from DG server -> Total: 1881 nodes shutdown (green solution) Presentation for: ISGC2011 9 Author: Peter Kacsuk 9

Results of EDGeS: Production DG ⇒ EGI Infrastructure gLite based EDGeS VO Core UI/ Resource Service Bridge voms lfc XtremWeb- BOINC-based based DGs DGs bdii lb wms myproxy UoW, IN2P3, Correlation Almere XtremWeb-EGI BOINC-EGi Systems bridge (ui2) bridge (ui1) local DGs local DGs se ce ce ce ce SZTAKI 16 cpus CIEMAT 20 cpus CNRS 1800 cpus SZDG, Almere, IN2P3 Ibercivis ce ce ce public DGs public DGs ce ce ce LIP 1080 Sinica 6 UFCG 6 BIFI 36 KFKI 410 IPB 672 Presentation for: ISGC2011 10 Author: Peter Kacsuk

A typical example of the DG=>EGI direction: ViSAGE - Video Stream Analysis in a Grid Environment By Correlation Systems Ltd. - Israel Visage processes Image pairs over the BOINC/EDGeS grid and paints movement in yellow. Video options: forward, backward, pause, play.. Insert video source: Insert Url of Server: Set mode to EDGeS Presentation for: ISGC2011 Author: Peter Kacsuk

ViSAGE - Video Stream Analysis in a Grid Environment By Correlation Systems Ltd. - Israel ViSAGE Small local DG at Israel EDGeS VO DG to SG bridge Presentation for: ISGC2011 Author: Peter Kacsuk

Results of EDGeS: Production EGI ⇒ DG Infrastructure gLite based service grid Core Resource Bridge Service voms lfc AR XtremWeb- BOINC-based based DGs DGs bdii lb wms myproxy IN2P3 bridge bridge UoW EGI-DG bridge (ce1) local DG local DGs SZDGr, bridge IN2P3 Fundecyt, bridge @home bridge Almere, bridge Almere EDGeS@ home public DGs public DGs VO1 VO2 Presentation for: ISGC2011 13 Author: Peter Kacsuk

Presentation for: ISGC2011 14 Author: Peter Kacsuk

Major Application on EDGeS@home • Fusion community application: ISDEP - Integrator of Stochastic Differential Equations in Plasmas • Used in EGI before EDGeS • EDGeS created: – A production quality BOINC implementation deployed on the Ibercivis and EDGeS@home Desktop Grids. – Application runs through the EGI=>BOINC bridge allowing EGI fusion VO users to utilise resources of EDGeS@home. Presentation for: ISGC2011 Author: Peter Kacsuk

BOINC/gLite applications available @ EGI AppDB 22 applications Presentation for: ISGC2011 Author: Peter Kacsuk

XtremWeb/gLite applications available @ EGI AppDB 10 applications Presentation for: ISGC2011 Author: Peter Kacsuk

How to extend gLite VOs with volunteer and institutional desktop grids? Presentation for: ISGC2011 Author: Peter Kacsuk

Security issues when gLite VOs are extended with DGs • In DG projects – Applications are trusted – DG server is trusted – Clients are not trusted • Therefore any EGI application that is intended to run in a volunteer DG should be validated to make it trusted • EDGeS community provides: – Application validation service – Application repository (AR) where trusted applications are stored – EDGeS bridge transfers only those applications to DGs that are stored in the AR Presentation for: ISGC2011 19 Author: Peter Kacsuk

gLite VO  DG system via EDGeS services DG servers BOINC1 EGI BDII EGI WMS BOINC2 EGI LB EGI Submit Watch VOMS job Get output X509 XWeb proxy EGI UI Presentation for: ISGC2011 Author: Peter Kacsuk

gLite VO  DG system via EDGeS services DG servers BOINC plugin WS interface 3G BOINC1 EGI BDII Bridge BOINC plugin WS interface EGI 3G WMS BOINC2 EGI Bridge LB EGI Submit Watch VOMS job Get output WS interface XWeb plugin X509 3G XWeb proxy Bridge EGI UI Presentation for: ISGC2011 Author: Peter Kacsuk

gLite VO  DG system via EDGeS services DG servers EDGeS Bridge BOINC plugin WS interface Services Modified Info provider 3G BOINC1 EGI BDII Bridge CE (with DG-CE) 3GB WS client DG-LRMS BOINC plugin WS interface Send job EGI 3G WMS BOINC2 EGI Bridge LB EGI Check Submit Watch VOMS EXE job Get output WS interface XWeb plugin X509 3G EDGeS XWeb proxy Bridge EGI UI Application Repository Presentation for: ISGC2011 Author: Peter Kacsuk

gLite VO  DG system via EDGeS services DG servers EDGeS Bridge BOINC plugin WS interface Services Modified Info provider 3G BOINC1 Report resources EGI BDII Bridge CE and performance (with DG-CE) QM WS client DG-LRMS BOINC plugin Submit job WS interface Send job EGI 3G Log events WMS BOINC2 EGI Bridge Send output LB EGI Check Submit Watch VOMS EXE job Get output WS interface XWeb plugin X509 3G EDGeS XWeb proxy Bridge EGI UI Application Repository Presentation for: ISGC2011 Get EXE Author: Peter Kacsuk

Target 1: extend your gLite VO with volunteer DG Infrastructure Extend your gLite VO with volunteer DGs Public DG Steps: Connect the modified VO Public DG CE of the EDGeS gLite Bridge services to Bridge services your gLite VO using your BDII service Presentation for: ISGC2011 Author: Peter Kacsuk

Target 2: extend your gLite VO with institutional DGs Extend your gLite VO with institutional DGs Public DG Local DG Steps: 1. Establish e.g. univ. DGs VO Public DG 2. Deploy 3G Bridge on the gLite univ. DG server Bridge services 3. Ask EDGeS Bridge Service provider to connect your Local DG DG server to the EDGeS Local DG Bridge services Presentation for: ISGC2011 25 Author: Peter Kacsuk

Plans for EDGI EDGI scope for both compute and data EDGeS intensive scope only applications • for for EMI/EGI compute (gLite, ARC, intensive Unicore) applications Extend • for gLite Desktop Grids with Clouds for QoS Presentation for: ISGC2011 26 Author: Peter Kacsuk