fedcloud f2f
play

FedCloud F2F www.egi.eu EGI-Engage is co-funded by the Horizon 2020 - PowerPoint PPT Presentation

Mar 29, 2023 •115 likes •431 views

FedCloud F2F www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142 Agenda for F2F EGI-ENGAGE EGI FedCloud status overview (10) Resource Providers and operations

  1. FedCloud F2F www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142

  2. Agenda for F2F EGI-ENGAGE • EGI FedCloud status overview (10’) • Resource Providers and operations views and feedback (30’) – How can we attract more resources? • Roadmap discussion (1h) – State of the Scenarios – Next steps • AOB (20’) 2

  3. EGI-ENGAGE EGI FedCloud Status 3 10/11/15

  4. The infrastructure EGI-ENGAGE 4

  5. Infrastructure numbers EGI-ENGAGE Nov 2014 – May 2015 May 2015 – Nov 2015 • 21 sites • 20 sites (SZTAKI gone) • 28.3K VMs* • 29.4K VMs* • 335K CPU hours* • 257K CPU hours* • 20 VOs • 24 VOs • CC are starting to Missing clear informa(on from long- apporach fedcloud: running VMs, these may have grown in LifeWatch, MoBrain, the last period BBMRI, ELIXIR… * Accoun(ng numbers removing LHC and local VOs 5 10/11/15

  6. Sites joining EGI-ENGAGE • New sites actively joining in: – BITP (UA) – EBI (UK) • Long-standing integrations – KISTI (KR) • Got stuck with network configuration, no progress since August – CSC (FI) • Proof of concept working, still not integrated into production. – RAL (UK) • Under risk assessment (network separation) 6

  7. Documentation EGI-ENGAGE • Wiki completely reviewed • Technology Section – Services and tools that build the federation • Site-related documentation – Complete installation manuals for sites • OpenNebula and OpenStacl – Several HOWTOS for configuration of specific functionality – Being tested by new sites now • User/Developers related-documentation – OCCI-client FAQ reviewed – API, SDKs, CLIs page 7

  8. EGI images EGI-ENGAGE • Allow users to get started quickly – Cloud-init configured (no user/root passwords) • Automatically built with packer – Easy to get updated versions • Set of basic OS images – Ubuntu 12, Ubuntu 14, Centos 6 – Centos 7 ready, not deployed due to problems with older versions of Xen • FedCloud clients VM – rOCCI – Preconfigured fedcloud.egi.eu + training.egi.eu VOMS • Training images – MoinMoin Wiki – Fractal Application 8

  9. Docker EGI-ENGAGE • Demand for Docker support increasing (PanCancer, HBP, READemption, ) • Docker is supported on FedCloud – Just install it on a VM – A Ubuntu 14.04 + Docker image now available! • Docker container with voms + OCCI clients installed – Run the EGI FedCloud clients without messing with your machine configuration – Will use it in training tomorrow • Next step: – Documentation – Clusters? 9

  10. Training EGI-ENGAGE • Training infrastructure available – CESNET, BIFI, CIEMAT, CATANIA, UKIM • Use of Per User Sub Proxies – No need for certificates for the trainees – Supported in OpenStack and OpenNebula* • Training modules – Introduction with OCCI CLI, already used in three events – For this week: • Preparation of VMIs, VMDIRAC, COMPSs, D4Science – Coming: • Docker *not complete user separa(on 10

  11. Expanding the Federation EGI-ENGAGE • IaaS interfaces – OCCI as main & preferred API – Openstack API now also supported – Swift API coming soon – PROC19 being tested on: • OpenNebula + rOCCI • OpenStack + OCCI-OS • OpenStack Nova • FedCloud as a technology provider for building federations – See wiki, technology section. 11 10/11/15

  12. Last news from the FedCloud Use Cases EGI-ENGAGE • BILS – Swedish ELIXIR Node: – SLA to be signed by the end of Nov 2016 – CPU cores: 324, RAM: 648 GB, Temp Storage: 1425 GB, Permanent Storage: 5400 GB • Chipster (NGS) – CSC Finnish ELIXIR NODE: – In production with VO chipster.csc.fi – CSC/EGI Training events: • NGS event in Thessaloniki (Oct 2015) • 2 tutorials on Wednesday: CHIPSTER deployment (11:00), Usage for NGS analysis (15:30) • Planning a tutorial in March 2016 in Helsinki (collaboration with ELIXIR's EXCELERATE project) FedCloud F2F @ Bari 12

  13. Last news from the FedCloud Use Cases EGI-ENGAGE • EMSO/EMSODev: – First test running in the FedCloud: • Setup an HADOOP cluster • Scipion/INSTRUCT: – It is an image processing framework to obtain 3D models of macromolecular complexes using Electron Microscopy – MoBrain CC – First testing deployments in the Fedcloud ready 13

  14. Main issues (personal view) EGI-ENGAGE • VMI replication to sites is not reliable – Lack of monitoring • OCCI support in OpenStack – OCCI-OS not maintained, ooi not released – FZJ 0% A/R because of this! • Need to find balance between security and usability – We need to properly define who and what can be done and enforce it • Lack of tool ecosystem (e.g. support from libraries like jclouds, lacking support of orchestrators, vagrant, no GUIs, ...) 14 10/11/15

  15. EGI-ENGAGE Resource Providers 15 10/11/15

  16. Sites joining EGI-ENGAGE • New sites joining in: Why integra(on takes so long? – BITP (UA) • Documenta(on? – EBI (UK) • SoKware Quality? • Security? • Long-standing integrations • … – KISTI (KR) • Got stuck with network configuration, no progress since August – CSC (FI) • Proof of concept working, still not integrated into production. – RAL (UK) • Under risk assessment (network separation) 16

  17. We need feedback! EGI-ENGAGE • What are the biggest issues as a resource provider in FedCloud? • How can we get more providers joining? 17 10/11/15

  18. EGI-ENGAGE Roadmap discussion 18 10/11/15

  19. Scenarios EGI-ENGAGE • VM Management • VM Image Management • Data Management • VM endorsement • Information Discovery • Brokering • Accounting • Security • Monitoring • Intra Cloud Networking • Federated AAI 19 10/11/15

  20. VM Management EGI-ENGAGE • PUSP support • OCCI extensions and move to OCCI 1.2 – Compute -- resize – Compute – save • Deployment of OOI for OpenStack • Native API support • Improvements/integration for client tools 20 10/11/15

  21. Information Discovery EGI-ENGAGE • VO specific info – Who can prepare the schema? • Available resources – Should this be published? • Alternatives to BDII? • Leadership of this scenario? 21 10/11/15

  22. Accounting EGI-ENGAGE • Storage, IP, any other resource accounting? • How to make users aware of their usage? 22 10/11/15

  23. Monitoring EGI-ENGAGE • Rework OCCI probe to use an AppDB VMI – Test all of the EGI VMIs? • VMCatcher probe • Block Storage probe • Swift probes 23 10/11/15

  24. AAI EGI-ENGAGE • Need integration with broader EGI AAI plans – Technology? – Roadmap? • Who can act as bridge? 24 10/11/15

  25. VMI management EGI-ENGAGE • Single VMI per VA version • Reliable image distribution – Disk consumption – Documentation – Allow external implementations (HEPiX list format specification) • Banning VA versions – independent of endorsement? 25 10/11/15

  26. VMI endorsement EGI-ENGAGE • Endorsing images not that easy to implement – Signing? Who? When? How? un-endorsement? • Need to find balance between security and usability – Automatic checks? • This does not solve that the may go (insecurely) crazy during execution! 26 10/11/15

  27. Networking EGI-ENGAGE • Is OCCI network enough? – How to deal with Neutron? – Firewalls? • How to deal with multiple sites? – VPNs – DNS as a Service 27 10/11/15

  28. Brokering, user interfaces EGI-ENGAGE • How can users profit from the federation? • GUIs? – AppDB as VM Management tool • No integration with existing cloud ecosystem • Leadership? 28 10/11/15

  29. Security EGI-ENGAGE • Security Threat Risk Assessment with Cloud focus – Need participation from a few others in the EGI Federated Cloud • Security Requirements related to the EGI Fed cloud. – VA and VM drafted, needs iteration, probably also software requirements extracting • Probably also we should write down the EGI Federated Cloud Security Model • Security Monitoring? 29 10/11/15

  30. Data Management EGI-ENGAGE "The Elephant in the Room Banksy-Barely legal-2006" by Bit Boy - Flickr: The Elephant in the Room. Licensed under CC BY 2.0 via Commons - hWps://commons.wikimedia.org/ 30 10/11/15

  31. EGI-ENGAGE AOB 31 10/11/15

More recommend