IBERGRID towards the EOSC Presented by: Joao Pina (LIP-Lisbon), on - PowerPoint PPT Presentation

IBERGRID towards the EOSC Presented by: Joao Pina (LIP-Lisbon), on behalf of IBERGRID

Distributed computing infrastructure Ministério da Ciência, Tecnologia e Ensino Superior 2 Iberian Distributed Computing Infrastructure

Distributed computing infrastructure a) Federates infrastructures from Iberian research and academic organizations (PT + ES) mainly focused; • Grid Computing • Cloud Computing • Data Processing. b) Enables a joint participation in the European Grid Initiative (EGI) and other initiatives (EOSC, ESFRIs, WLCG, etc); • Provides the regional operations coordination for the computing and data processing activities of several user communities; Forum for common R&D activities and sharing of knowledge . c) • Participation in European and cross-border projects; • Conferences and events. 3 Iberian Distributed Computing Infrastructure

EOSC Framework Iberian Distributed Computing Infrastructure

Vision towards EOSC implementation Expanding the Capacity and Capabilities of EOSC by leveraging the experience, effort and resources of national publicly-funded digital infrastructures in a coherent way, therefore acting also as an incentive for national resource providers. • European Framework: project EOSC-synergy • Led by IBERGRID • Funded in infraeosc5-b, • Starts September 1st 2019 (30 months) • 16 partners Iberian Distributed Computing Infrastructure

Iberian Distributed Computing Infrastructure

• Deployment of an EOSC-enabled digital • Definition of an EOSC software quality baseline infrastructure with a critical mass • Service Quality Assurance as a Service: SQAaaS • Designing a rational strategy to deploy • FAIRness checking “as a Service” FAIR National Data Repositories CAPACITY EXPANSION FOSTERING SKILLS SERVICE DEVELOPMENT INTEGRATION BUILDING CAPABILITIES • Enable auto-training in EOSC • Integration of production-level National • Explore and implement a MooC solution Thematic Services in EOSC following a Marketplace approach • Thematic service certification based on SQAaaS. Iberian Distributed Computing Infrastructure

IBERGRID towards the EOSC Integration of Capacity in EOSC 8 Iberian Distributed Computing Infrastructure

Know-how in federating infrastructures: 2001 2004 2010 2017 Building Capabilities via User Community Engagement Iberian Distributed Computing Infrastructure

IBERGRID contribution to Federation work https://ibergrid.eu/federating-clouds/ ● Accounting Portal: ○ http://accounting.egi.eu ● Cloud Accounting software – CASO ○ https://github.com/IFCA/caso ● Processes and Policies: SQA Coordination ○ Software Quality Assurance Processes and Policies ● Author.& Authent. OpenStack-VOMS ○ https://github.com/IFCA/voms-auth-system-openstack 10 Iberian Distributed Computing Infrastructure

Lessons – Grid achievements ❑ Technical ▪ Standards effort is very important ▪ Interoperability (not perfect or complete, but very valuable) ▪ Sophisticated data an resource management ▪ Worldwide authentication for scientific grids ▪ Common usage and security policies ▪ Many developments for privacy and security ❑ European Policy and coordination structure ▪ Creation of National Grid Initiatives supported by the governments ▪ Creation of the European Grid Initiative – EGI in 2009 ▪ Model for Long Term sustainability Iberian Distributed Computing Infrastructure

Making the leap to EOSC Building on what we have 2001 2004 2010 2017 Integration at the AAI level (the Virtual Organization model worked, Monitoring but it has limitations) EGI Federated Model Open Science EOSC Model Grey Zone Integration at the security level Automatization of Resource provisioning Technical and Policy - wise (it always worked manually: VO configuration) In a nutshell • Policy Level: Actions to be supported by technical developments, notably in the front of Security (data privacy policies) and access policies • Technical Integration Level: work to be done/extended, building on the previous experience. Iberian Distributed Computing Infrastructure

IBERGRID towards the EOSC SOFTWARE AND SERVICE DEVELOPMENT FOR EOSC 13 Iberian Distributed Computing Infrastructure

Software activity history as Iberian Federation 2018 - 2020 2014-2017 2010-2013 CSIC coordinated 2005-2007 7 countries / 12 partners 2009… ( 5 Iberian ) @ Isabel Campos & Jorge Gomes EGI.eu is funded in Feb. 2009. Software Quality task EGI-wide 2006-2010 Many… 2002-2005 2005-2010 11 countries / 21 partners “Southwest Federation” @ Jesús Marco ( 5 Iberian ) 50 partners/8-10 Iberian & Jorge Gomes 14 Iberian Grid Initiative

Popular IBERGRID developments: ● CONTAINER SERVICES ○ UDocker (LIP) ● COMPUTING SERVICES ○ OpenStack Preemptible Instances - OPIE (CSIC) ● PaaS SERVICES ○ Infrastructure Manager - IM (UPVLC) ● THEMATIC SERVICES ○ LIFEWATCH (Lifewatch ERIC) ○ OPENCOASTS (LNEC) 15 Iberian Distributed Computing Infrastructure

INDIGO-DC Software Collaboration Agreement Ibergrid teams are involved in the development of software to support innovative services for researchers in the European Open Science Cloud framework. The INDIGO-DC Software Stack Catalogue is a CSIC , LIP and UPVLC are signatories catalogue of open source software components that are follow the Architecture defined by the of the INDIGO-Datacloud Software INDIGO-DataCloud project, funded by the European Collaboration Agreement , a key Union under the Horizon 2020 Framework Program Technology provider for the EOSC with Grant Agreement 653549. ecosystem. This Collaboration Agreement targets to sustain and further develop the INDIGO–DataCloud INDIGO-Datacloud paper (Open Access): architecture, the original INDIGO-DataCloud Software Catalogue and as well as the „INDIGO brand” beyond https://link.springer.com/sharelink/10.1007/ the lifetime of the INDIGO-DataCloud Project, through s10723-018-9453-3 a not-for-profit, liability-free mutual…. 16 Iberian Distributed Computing Infrastructure

Software Quality Management See: http://digital.csic.es/handle/10261/160086 17 Iberian Distributed Computing Infrastructure

Software Quality Assurance in Consolidation of an EOSC Software Maturity Baseline ฀ Consolidation of best practices that seek software quality Implementation of an EOSC Service Integration Platform: SQAaaS Iberian Distributed Computing Infrastructure

IBERGRID towards the EOSC USER COMMUNITIES 19 Iberian Distributed Computing Infrastructure

User Communities: Examples Community Computing resources Type of resources WLCG 7 600M CPU*hours Grid Lifewatch 16M CPU*hours Cloud INSTRUCT On demand Cloud WeNMR and MoBrain 3M CPU*hours Grid (cooperation in integration and packaging) AUGER 12M CPU*hours Grid OPENCoastS 0.5M (recent) Cloud and Grid (EOSC Thematic service) Genetics 90.5M CPU*hours Grid 20 Iberian Distributed Computing Infrastructure

User Communities: Examples 21 Iberian Distributed Computing Infrastructure

Two examples of user support ฀ udocker ฀ event-driven computing Iberian Distributed Computing Infrastructure

udocker: running containers everywhere ● INDIGO, EOSC, DEEP HybridDataCloud ● Execute Docker in batch systems, HPC and HTC transparently ● no docker, no installation, no privileges, download and run ● Open source available in: https://github.com/indigo-dc/udocker Technologies: Usage: • Docker containers • Many communities • PTRACE, • DEEP HybridDataCloud SECCOMP • INDIGO-DataCloud • Library interception • EOSC • OpenContainers • etc Iberian Distributed Computing Infrastructure

OSCAR: supporting event-driven computing. (biodiversity in plant recognition) • OSCAR: open-source serverless computing platform for data-processing applications that can be deployed on multi-Clouds. • https://github.com/grycap/oscar • Creates highly-parallel event-driven file-processing serverless applications: • Executes customized runtime environments provided by Docker containers. • Runs on an elastic Kubernetes cluster (dynamic provision of nodes). • Users upload files to a storage back-end triggering the parallel invocation of a function that runs a user-defined script in charge of processing each file. • Integrated with EGI Applications on Demand, EGI Federated Cloud and EGI DatHub. Iberian Distributed Computing Infrastructure

OSCAR: Open Source based Iberian Distributed Computing Infrastructure

Returning to the origins*: IBERGRID 2019 Co-located with EOSC-synergy KoM When: 23rd-26th September 2019 Where : Santiago de Compostela Opportunities for co-location : drop us an e-mail. More details in: http://www.ibergrid.eu/ (*) IBERGRID 2007 – Santiago de Compostela First Ibergrid Conference http://old.ibergrid.eu/2007 Iberian Distributed Computing Infrastructure