OpenStack on the EGI Federated Cloud Enol En l Fe Fernández Cloud ud Archi hitect – EGI Founda undation eno nol.ferna nande ndez@egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme www.egi.eu www of the European Union under grant number 654142
EGI: Advanced Computing for Re EG Research 12/ 12/15/ 15/16 16 2
http://go.egi.eu/ServiceCatalogue 12/ 12/15/ 15/16 16 OpenStack Summit Barcelona 3
12/ 12/15/ 15/16 16 OpenStack Summit Barcelona 4
EGI Federation, , 2016 QR3 The he largest di distribut buted d comput pute e-Infra worldw dwide de 2.6 Billion 23 Cloud +250 000 providers, 1.7 Million CPU >48 000 instantiated +300 data jobs/day hours/year users, +25% VMs/year +26% centres 12/ 12/15/ 15/16 16 OpenStack Summit Barcelona 5
Interna In natio ional nal Par artne nership hips Canada China Inst. Of HEP Chinese Academy of Sciences USA India Centre for Development of Advanced Comp. Africa and Arabia Council for Scientific and Asia Pacific Region Industrial Research, South Africa Academia Sinica at Taiwan Latin America Ukraine Universidade Federal do Ukrainian National Rio de Janeiro Grid 12/ 12/15/ 15/16 16 6
Serving Se ing resear arche hers and and inno innovators WLCG Size of individual CTA ELIXIR groups EPOS EISCAT_3D BBMRI CLARIN PeachNote LOFAR CEBA Galaxy eLab EMSO VRE projects Semiconductor design ELI WeNMR Agroknow Main-belt comets LifeWatch DRIHM CloudEO Quantum pysics studies ICOS VERCE CloudSME Virtual imaging (LS) EMSO MuG Ecohydros Bovine tuberculosis spread CORBEL AgINFRA gnubila Convergent evol. in genomes ENVRIplus CMMST Sinergise Geography evolution … LSGC SixSq Seafloor seismic waves SuperSites Exploitation TEISS 3D liver maps with MRI Environmental sci. Terradue Metabolic rate modelling neuGRID Ubercloud Genome alignment … … Tapeworms infection on fish … ESFRIs, Multinational communities Industry, ‘Long tail’ FET flagships SMEs 12/15/ 12/ 15/16 16 OpenStack Summit Barcelona 7
EGI Federated Cloud EG EGI Federated Cloud is a • collaboration of communities developing, innovating, operating and using cloud federations for research and education. 23 providers from 14 NGIs • – 16 OpenStack – 6 OpenNebula – 1 Synnefo ~7K CPU cores • 12/ 12/15/ 15/16 16 8
Cloud Cl ud Realms ms Community Community Platform Platform VM image catalogue, Helpdesk Collaboration Platform EGI endorsed images Cloud Realm Cloud Realm Cloud Realm Cloud Realm EGI Core Infrastructure Platform AAI, Service Registry, Accounting, Monitoring, Federated Service Management Cloud Realm Community Platforms provide community-specific data, tools and subset of cloud providers exposing homogeneous cloud management interfaces applications and can be supported by one or more realms. and capabilities which use the the services of the EGI Core Infrastructure Platform for creating a federation 12/15/ 12/ 15/16 16 OpenStack Summit Barcelona 9
OpenStack realm Open Standards Cloud Cl ud Fede deration realm Uniform Harmonised user interfaces OpenNebula operation OpenStack Service registry Information system Synnefo OpenStack Virtual Machine marketplace Usage accounting Access control OpenNebula OpenStack 12/15/ 12/ 15/16 16 10 10
EG EGI AAI Users identified with X.509 certificates (IGTF Federation) with VOMS • extensions VOMS (Virtual Organization Management System) provides attributes on • membership to VOs, groups and roles on the VO Not user-friendly, problematic in web-based GUIs • Now in transition to new EGI AAI (EGI CheckIn) • Federated identity standards (SAML, OpenID Connect) • Allows users to authenticate with their institutional accounts • Integration with Attribute Authorities beyond VOMS • 12/15/ 12/ 15/16 16 OpenStack Summit Barcelona 11 11
Ke Keystone-VO VOMS • WSGI filter for Keystone V2 API • Extracts information from VOMS proxies to perform AuthN/AuthZ • Can manage federation users • Add users to Keystone • Add roles to users in tenants • Mapping VOMS → Keystone defined on file https://github.com/IFCA/Keystone-VOMS 12/ 12/15/ 15/16 16 OpenStack Summit Barcelona 12 12
EGI Che EG CheckIn In + + Ke Keystone First name, EGI UID Mandatory last name User IdP Attributes SAML assertion IdP email affiliation Apache HTTPD EGI Enter credentials CheckIn mod_shib Keystone SAML assertion with claims Attribute Horizon Authority Token Token 12/15/ 12/ 15/16 16 OpenStack Summit Barcelona 13 13
Se Servic ice Regis gistry All Resource Centers must • register their services at the EGI central catalog: https://goc.egi.eu Static information about • services endpoints – org.openstack.nova and org.openstack.swift service types Web frontend and API • access 12/15/ 12/ 15/16 16 14 14
In Informa matio ion n Dis Discovery • Real-time information provided by BDII – Hierarchical information discovery system based on LDAP – Using standard Glue Schema 2 Resource Centers publish actual capabilities • – Available images & flavors – Supported user groups (VOs) – Available resources • Cloud-bdii-provider – Gathers information from OS services using public APIs and puts it into Glue Schema – https://github.com/EGI-FCTF/cloud-bdii-provider 12/15/ 12/ 15/16 16 OpenStack Summit Barcelona 15 15
Acc Accounting Collect, aggregate and display • usage information across the whole federation. OGF Usage Record extended for • Cloud cASO produces accounting • records using nova (and optionally ceilometer) APIs • https://github.com/IFCA/caso 12/15/ 12/ 15/16 16 16 16
Mon Monitor oring • Health monitoring of services • Automatic discovery of services using GOCDB • A/R metrics for SLA/OLAs • Powered by EGI ARGO https://argoeu.github.io/ • 12/15/ 12/ 15/16 16 17 17
VM VM Image Marketplace: Ap AppDB • Open Library of Virtual Appliances • Use on clouds or for personal download Re-use, share, associate • contextualization • EGI endorsed VM images, securely configured and tested • Community curated sets of images Automatic distribution of sets to cloud • providers https://github.com/alvarolopez/atrope • 12/15/ 12/ 15/16 16 18 18
VM VM Image Marketplace: Ap AppDB 12/ 12/15/ 15/16 16 19 19
Ope OpenSt nStack OCCI OCCI interface (ooi ooi) OCCI (Open Cloud Computing Interface, OGF) • OC – RESTFul protocol and API focusing on cloud interoperability – Primarily for IaaS (manage VMs and Block Storage), extensible to other areas https://launchpad.net/ooi • ooi (OpenStack OCCI interface) – Completely written from scratch OCCI implementation – Uses only public OpenStack APIs – Support for VM, volumes and network operations – Can be installed along an existing nova-api endpoint or as a separate WSGI application 12/15/ 12/ 15/16 16 OpenStack Summit Barcelona 20 20
ooi ooi in in the glo global al OCCI I pic icture ooi 12/ 12/15/ 15/16 16 OpenStack Summit Barcelona 21 21
EG EGI – Ope OpenSt nStack integr gration n (I) (I) OpenStack resource center Keystone ceilometer Get access token Keystone- Publish VOMS VOMS x509 proxy accounting EGI Apache+SSL+ records Extract usage cASO + mod_wsgi accounting information SSM repository OpenStack clients IaaS Operations nova-api Publish information cloud-bdii- EGI BDII Extract provider information OCCI clients ooi Subscribe to AppDB image lists atrope EGI (vmcaster) monitoring glance Register/update images 12/15/ 12/ 15/16 16 22 22
EG EGI – Ope OpenSt nStack integr gration n (II) (II) OpenStack resource center EGI CheckIn Keystone ceilometer OS- Publish FEDERATION Get access token accounting EGI Apache+SSL+ records Extract usage cASO + mod_wsgi accounting information SSM repository OpenStack clients IaaS Operations nova-api Publish information cloud-bdii- EGI BDII Extract provider information OCCI clients ooi Subscribe to AppDB image lists atrope EGI (vmcaster) monitoring glance Register/update images 12/ 12/15/ 15/16 16 23 23
Ope OpenSt nStack Fe FedCloud Ap Appliance ce • A single VM with all the components using the public OpenStack interfaces – Accounting, Information discovery, VMI replication – Packaged as Docker containers, available at docker hub https://hub.docker.com/u/egifedcloud/ • Documentation: – https://wiki.egi.eu/wiki/MAN10#Integration_with_EGI_FedCloud_Appl iance • Appliance at AppDB: – https://appdb.egi.eu/store/vappliance/fedcloud.integration.appliance. openstack 12/15/ 12/ 15/16 16 OpenStack Summit Barcelona 24 24
Recommend
More recommend
