SZDG, eCom4Com technology, EDGeS-EDGI in large P. Kacsuk MTA - PowerPoint PPT Presentation

SZDG, eCom4Com technology, EDGeS-EDGI in large P. Kacsuk MTA SZTAKI 1 The EDGI/EDGeS projects receive(d) Community research funding

Outline of the talk • SZTAKI Desktop Grid (SZDG) • SZDG technology: eCom4Com • EDGeS • EDGI Presentation for: 6 th IDGF Tutorial 2 Author: Peter Kacsuk

SZTAKI Desktop Grid global version Main objectives: • • Demonstrate the power of the Desktop Grid concept • Support Hungarian scientific applications • Introducing DG technology in Hungary Three steps for everybody to try and use the • technology: 1. Donate one PC to test the client site 2. Port application to the DG server and register PCs for that application 3. Set up a DG server for the community (univ., company, city, etc.) SZTAKI helps in steps 2 and 3 as explained in detail • at http://www.desktopgrid.hu/ Presentation for: 6 th IDGF Tutorial 3 Author: Peter Kacsuk

Presentation for: 6 th IDGF Tutorial 4 Author: Peter Kacsuk

Presentation for: 6 th IDGF Tutorial 5 Author: Peter Kacsuk

Lessons learnt from SZDG project • BOINC is excellent to create volunteer DG projects – Reliable, stable, robust and scalable technology • Drawback – Difficult to port applications to BOINC – There is no user interface through which end- users could easily launch applications with their parameters Presentation for: 6 th IDGF Tutorial 6 Author: Peter Kacsuk

Question: Can we use BOINC to build institutional DGs? • Yes, but in this case we have to solve the following problems: – Abandon the credit system – Enable • Fast deployment • Fast application porting • Easy usage by end-users • Possible extension with – service grids – desktop grids – clouds Presentation for: 6 th IDGF Tutorial 7 Author: Peter Kacsuk

eCom4Com • SZTAKI developed a new technology based on BOINC and SZTAKI Desktop Grid experiences. • The new technology is called eCom4Com (e- Computing for Communities) • The goal of the eCom4Com technology is to quickly build and easily use BOINC-based DG systems • Easily run PS-nodes of workflows in such DG systems. Presentation for: 6 th IDGF Tutorial 8 Author: Peter Kacsuk

A real example: CancerGrid workflow N = 30K, M = 100 => about 0.5 year execution time x1 NxM= 3 million x1 xN xN xN NxM NxM x1 xN xN xN Generator job Generator job N=30K NxM= 3 million Presentation for: 6 th IDGF Tutorial 9 Author: Peter Kacsuk

Accessing Desktop Grids by a Grid portal WS- PGRADE Grid portal Volunteer DG University DG University Public Local Local Faculty DG DG DG DG Presentation for: 6 th IDGF Tutorial 10 Author: Peter Kacsuk

eCom4Com software • eCom4Com technology provides the following components on top of BOINC: – Easy setup: Debian packages for the server • For Volunteer Computing: Global • For Private/Local Desktop Grids: Local – Simplify writing applications: DC-API – Support porting legacy applications: GenWrapper – WS-PGRADE workflow portal for easily run applications – Allow setups peculiar to DGs: • clusters as resources, • 3G Bridge enables hierarchy, bridging to/from other systems (service grids, clouds) – Security enhancements: certificates, virtualisation – VMs to easily deploy DG systems with bridges Presentation for: 6 th IDGF Tutorial 11 Author: Peter Kacsuk

Automatic generation of master and client code • SZTAKI developed the DC-API (Distributed Computing API) that can – automatically generate WUs for PS jobs at the server of the DG system • SZTAKI developed a generic wrapper that can – eliminate the boincification of the client code – automatically generates the client code without modification of the original code Presentation for: 6 th IDGF Tutorial 12 Author: Peter Kacsuk

The eCom4Com architecture to support PS legacy applications DG Server BOINC client WS- BOINC gUSE Server Storage PGRADE Components (User IF) Sche- duler (WF repre- sentation) Data server gUSE BOINC client Local Submitter gUSE BOINC (Workflow Task gUSE enactor) DB WS Submitter Presentation for: 6 th IDGF Tutorial 13 Author: Peter Kacsuk gUSE DesktopGrid

The eCom4Com architecture to support PS legacy applications DG Server BOINC client WS- BOINC gUSE Server Storage PGRADE DC-API cli Components GenWrapper for (User IF) batch execution Sche- Legacy duler Application (WF repre- sentation) Data server gUSE BOINC client Local Submitter DC-API cli gUSE BOINC GenWrapper for (Workflow Task batch execution gUSE enactor) DB WS Legacy Application Submitter Presentation for: 6 th IDGF Tutorial 14 Author: Peter Kacsuk gUSE DesktopGrid

The eCom4Com architecture to support PS legacy applications DG Server BOINC client WS- BOINC gUSE gUSE Server DG Submitter Storage PGRADE DC-API cli Components GenWrapper for Job (User IF) descr. batch execution Sche- Legacy duler Job Database Application (Description of Jobs: Apps, Args, I/O files) Job (WF repre- descr. sentation) Queue Data DC-API master Manager server gUSE BOINC client Local Scheduling Submitter policy DC-API cli gUSE BOINC WU GenWrapper for (Workflow Task batch execution Batch gUSE enactor) DB creation WS Legacy Application Submitter Presentation for: 6 th IDGF Tutorial 15 Author: Peter Kacsuk gUSE gUSE-DG integration DesktopGrid

DC-API: Writing an application • Simple API to hide the grid infrastructure from application developers • Usable with minimal set of functions but has additional features that can be used when needed • Allows application deployment on different grid infrastructures without source code modification via different backends: – Standalone (local) for testing – BOINC – Condor – XtremWeb Presentation for: 6 th IDGF Tutorial 16 Author: Peter Kacsuk

DC-API application Application Master DC-API G r i d BOINC (Public or Condor Pool XtremWeb Local DG) DC-API DC-API DC-API Presentation for: 6 th IDGF Tutorial Application Worker Application Worker 17 Application Worker Author: Peter Kacsuk

Generic Wrapper (GenWrapper) • Objective of GenWrapper – The features of BOINC wrapper is not enough (e.g. patching config files on client machines, generating extra messages, independent jobs in a WU, etc.) – Wanted to be prepared for unknown requirements might be raised by future applications (e.g. Cancergrid) – We did not want to extend the BOINC wrapper to make it an XML-based programming language, we choose to BOINCify an existing language -> Bourne shell Presentation for: 6 th IDGF Tutorial 18 Author: Peter Kacsuk

Wrapping on the DG client • GenWrapper – Interfaces DC-API on behalf of the application BOINC client – prepares environment for the application – unpacks application DC-API cli binaries GenWrapper – executes application – handles multiple ins/outs execute BOINC client DC-API cli Legacy Application Application inputs outputs Presentation for: 6 th IDGF Tutorial 19 19 Author: Peter Kacsuk

SZTAKI Desktop Grid local version Main objective: • • Enable the creation of local DGs for any community • Demonstrate how to create such a system Building production Grids requires huge effort and represents a • privilege for those organizations where high Grid expertise is available SZTAKI Desktop Grid local version is built on the eCom4Com • technology Using the local SZDG package • • Any organization can build a local DG in a day with minimal cost (a strong PC is enough as a server machine) • The applications of the local community will be executed by the spare PC cycles of the local community • There is no limitation for the applied PCs, all the PCs of the organization can be exploited ( heterogeneous Grid ) • Users of the local SZDG can access the local DG system via WS- PGRADE portal Presentation for: 6 th IDGF Tutorial 20 Author: Peter Kacsuk

The CancerGrid System (CSG) WS-PGRADE Portal Local jobs DG jobs Job 1 3G Local Job 2 browsing executing Bridge Resource Job N molecules workflows WU 1 BOINC Portal WU 2 server Storage WU N BOINC client GenWrapper for batch execution WU X Legacy WU Y Application Legacy Application Portal and Desktop Grid server molecule database (4 selectable workflow applications) Presentation for: 6 th IDGF Tutorial 21 DG clients from all partners Molecule database server Author: Peter Kacsuk

Generic Grid-Grid (3G) Bridge 3G Bridge DC-API Plugin BOINC Job source BOINC BOINC client Job Database GridHandler Interface Job Handler Interface Plugin EGEE Job source EGEE EGEE EGEE GRAM XtremWe Plugin Job source ARC Xtrem ARC b Web Queue Manager Job source WS- ARC WS-PGRADE Plugin PGRADE and BES portals submitter Unicor e Presentation for: 6 th IDGF Tutorial Author: Peter Kacsuk

The Grid Ecosystem • Very expensive, • small number of sites and Supercomputer based SGs very large number of cores (DEISA) • MPI appls • Moderately expensive, Cluster based • moderate number of service grids (SGs) sites and CPUs (EGEE, NorduGrid, etc.) EDGeS • any appls • Inexpensive, Desktop grids (DGs) • very large number of (volunteer DGs – home computers, CPUs (~10K – 1M) institutional DGs – institutional desktops) • Bag of task appls Presentation for: 6 th IDGF Tutorial 23 Author: Peter Kacsuk