manager (IM) Introduction General platform to deploy on-demand - PowerPoint PPT Presentation

Infrastructure manager (IM)

Introduction • General platform to deploy on-demand customized virtual computing infrastructures. Web Interface CLI Interface • With the precise software RADL configuration required. Infrastructure Manager XML-RPC API REST API • Complex infrastructures. Conf. VMRC Cloud • Share Infrastructure descriptions. VMRC Manager Selector Ansible ... • No need of pre-baked VMIs. Cloud Connector VMRC OpenNebula OpenStack EC2 • The same complex infrastructure ... can be deployed both on on- MV premises and on public Clouds. VM Master Ficheros Conf & CTX Cntxt. MV Agent Ansible ... MV

Cloud providers • It supports a wide range of cloud providers and other computing back-ends : • Public: Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, Open Telekom Cloud (OTC). • On-premises: OpenNebula, OpenStack. • Federated: EGI FedCloud (OCCI), FogBow. … • Containers: Docker, Kubernetes 10 nodes Memory > 1GB • The list above can be easily Ubuntu Linux Java ver. 1.6 extended by plugins.

Main features • It features DevOps capabilities. • Based on Ansible. • Provides recipes for common deployments. • Also supporting cloud-init scripts. • IM works as a service that offers several interfaces: • XML-RPC and REST APIs. • Command-line application. • Web-based GUI. • It is distributed under a GNU GPL v3.0 open source license and its source code is available on GitHub. https://github.com/grycap/im

Infrastructure Manager • The user can provide an RADL or TOSCA documents as input to the IM, describing the infrastructure: • RADL : • Resource and Application Description Language. • High level Language to define virtual infrastructures and Specify VM requirements. • TOSCA : • OASIS Standard • Open standard language to model application architectures to be deployed on a Cloud.

RADL Document • An RADL document has the following general structure: ansible <ansible_host_id> (<features>) network net (outbound = ‘yes') system node ( network <network_id> (<features>) cpu . arch = 'x86_64' and cpu . count = 1 and memory . size >= 512M and system <system_id> (<features>) net_interface . 0.connection = 'net' and disk . 0.os . name = 'linux ‘ and configure <configure_id> (<Ansible recipes>) disk.0.image.url = ‘ one://onecloud.i3m.upv.es/67 ’ ) contextualize [max_time] ( system <system_id> configure node ( configure <configure_id> [step <num>] ... ) @begin - tasks: deploy <system_id> <num> [<cloud_id>] - user: name = user1 password = 1234 @end ) The keywords ansible, network, system and configure assign some features or recipes to an identity <id>. The features are a list contextualize ( of constraints separated by and, and a constraint is formed system node configure node ) by <feature name> <operator> <value>. deploy node 1

A Sample tosca template: kepler tosca_definitions_version : tosca_simple_yaml_1_0 endpoint: properties: imports : network_name: PUBLIC - indigo_custom_types: custom_types.yaml Network ports: topology_template: requirements vnc_port: node_templates: protocol: tcp kepler: source: 5900 type: tosca.nodes.indigo.Kepler os: properties: requirements: type: linux - host: kepler_server OS requirements distribution: ubuntu kepler_server: version: 14.04 type: tosca.nodes.indigo.Compute image: one://onecloud.i3m.upv.es/67 capabilities: host: outputs: instance_ip: properties: value: { get_attribute: [ kepler_server, public_address, 0 ] } num_cpus: 1 instance_creds: mem_size: 1 GB value: { get_attribute: [ kepler_server, endpoint, credential, 0 ] }

IM Working Scheme • The user specifies the image (or list of images) to use. • URI naming convention to abstract from cloud provider: • one://server:port/image-id • ost://server:port/ami-id • aws://region/ami-id • <site end-point>/<image-id> • The IM can also contact the VMRC to get a list with the most suitable VMIs • http://www.grycap.upv.es/vmrc • Then, the IM obtains the list of IaaS providers available to the user. • From the credentials provided by the user. • Finally, it contacts the IaaS provider selected and deploys the infrastructure.

Contextualization process 1. SSH connection to the Master VM • A GNU/Linux-based VM with a public IP 2. Configure Master VM 2 • Install and configure Ansible Master VM Conf. Check SSH Cntxt. 3. Launch Contextualization Agent Manager Configure Agent Master • Check SSH from VMs Ansible 3 • Call Ansible Get Apps Configure Check In VMI Requested Nodes SSH Apps 1 4 VMRC VM VM VM ... RADL

Client-side Tools: CLI usage: client.py [-u|--xmlrpc-url <url>] [-a|--auth_file <filename>] operation op_parameters options: --version show program's version number and exit -h, --help show this help message and exit -a AUTH_FILE, --auth_file=AUTH_FILE File with authentication data -u XML-RPC, --xmlrpc-url=XML-RPC URL of the InfrastructureManager service. Operations: list create <radl_file> destroy <inf_id> getinfo <inf_id> [radl_attribute] getradl <inf_id> getcontmsg <inf_id> getstate <inf_id> getvminfo <inf_id> <vm_id> [radl_attribute] getvmcontmsg <inf_id> <vm_id> addresource <inf_id> <radl_file> [ctxt flag] removeresource <inf_id> <vm_id> [ctxt flag] alter <inf_id> <vm_id> <radl_file> start <inf_id> stop <inf_id> reconfigure <inf_id> [<radl_file>] [vm_list] startvm <inf_id> <vm_id> stopvm <inf_id> <vm_id> sshvm <inf_id> <vm_id> getversion

Client-SIDE Tools: Web • Publicly-available web interface (also open-sourced). • Easily deploy infrastructures from a web browser • Share RADL/TOSCA documents. • http://servproject.i3m.upv.es/im/

APIs to be consumed by clients • XML-RPC API • API that follows the XML-RPC specification . • REST API • IM Service can be accessed through a REST(ful) API • More info: http://www.grycap.upv.es/im/documentation.php

Using IM in EGI FedCloud • Get data from AppDB. • Get the URI of the correct image to use. • Format: • <site end-point>/<image-id> • Get the instance size. • Use your proxy in IM the authentication data.

A simple node with Ansible network publica (outbound = 'yes') system front ( instance_type = 'extra-large' and net_interface.0.connection = 'publica' and disk.0.os.name = 'linux' and Data from AppDB disk.0.image.url = 'https://fc-one.i3m.upv.es:11443/uuid_image_for_egi_centos_6_centos6kvm_im154_170' ) configure front ( @begin - name: Install some packages action: yum pkg=pkg1,pkg2 state=installed @end )

A simple node with cloud-init network publica (outbound = 'yes') system front ( instance_type = 'extra-large' and net_interface.0.connection = 'publica' and disk.0.os.name = 'linux' and disk.0.image.url = 'https://fc-one.i3m.upv.es:11443/uuid_image_for_egi_centos_6_centos6kvm_im154_170' and ) configure front ( @begin packages: - pkg1 - pkg2 @end ) contextualize ( system front configure front with cloud_init )

VMOps Dashboard • Part of this functionality has been integrated in the EGI VMOps Dashboard . • IM is the OCCI communication layer to create VM topologies. • TOSCA compatible. • https://wiki.egi.eu/wiki/Federated_Cloud_AppDB_VMOps_Dashboard • https://dashboard.appdb.egi.eu/vmops

Where is the IM used? • In the INDIGO-DataCloud project: Deployment of Customized Virtual GUI-based 1. Access Portlets Infrastructures using INDIGO-DataCloud • IM is a key component of 4. Deploy the architecture: 3. Select / Customize • Used at the PaaS Core to provide 2. Authenticate TOSCA deployment of infrastructures to Orchestrator Repository IAM Service Service 5. Status Monitoring TOSCA- Service 9.b.1 Cloud sites external to INDIGO- compliant 8.a Transfer 6. Prioritize Delegate Templates (Optional) Deploy Brokering/ 7. Agree Policy Service DataCloud (including public Cloud Managed Services/ Infrastructure Applications (MSA) Data Manager QoS/SLA Service Management Service sites). Services 9.a Delegate Deploy 8.b. Transfer (Optional) • Used at IaaS to provide TOSCA-based 9.b.2 Provision & Configure deployment of infrastructures for CDMI TOSCA IM HEAT OpenNebula sites. POSIX / OpenNebula OpenStack POSIX / Cloud APIs CDMI CDMI SAGA WebDav SAGA WebDav External HPC Partner IaaS Grid EUDAT Cloud Clusters 10. Deliver VMs ... ... Virtual Infrastructure for User Application/Service

Where is the IM used? • IM is a key component in EC3 , a tool to deploy virtual hybrid elastic clusters that is integrated in the EGI Access service (for the long-tail of science). • EC3: http://www.grycap.upv.es/ec3/ • EGI Access: https://access.egi.eu/start