CERN and Science Clouds in Europe with TOSCA, OpenStack Heat and the Heat Translator Matt Rutkowski Mathieu Velten Sahdev Zala Ricardo Rocha Brad Topol mrutkows@us.ibm.com mathieu.velten@cern.ch spzala@us.ibm.com Ricardo.Rocha@cern.ch btopol@us.ibm.com @mrutkowski @bradtopol @sp_zala
Introduction • Indigo DataCloud Project • Goal is to provide a sustainable European infrastructure for science, spanning multiple computer centers and existing public clouds. • Challenges • Need to effectively exploit heterogeneous cloud infrastructures (OpenStack, OpenNebula, proprietary clouds) as a single cloud • Need a standard orchestration model that works across multiple cloud infrastructures • Project Approach • Leverage TOSCA as a standard based approach for modeling cloud stacks and applications Support for TOSCA is growing in several cloud communities • • Leverage OpenStack components as a means of deploying TOSCA cloud stacks • OpenStack Heat Project provides robust tools to convert alternative orchestration languages to OpenStack Heat’s HOT templates Heat-Translator: Provides a framework for translation orchestration models to Hot • TOSCA-Parser: Production quality tool for parsing TOSCA templates • • In this presentation we will cover • TOSCA-Parser and Heat-Translator • Project Indigo and CERN’s initiative • Demo concrete examples of using TOSCA to model a complex infrastructure covering multiplesites • Latest on TOSCA specification
Mitaka: TOSCA-Parser Node Types Relationships • What is the TOSCA-Parser? A parser for TOSCA Simple and Network Functions Virtualization (NFV) YAML based • specifications. Policies Groups Capabilities Produces in-memory graph of TOSCA nodes and relationship among them • Subproject of OpenStack Heat • Custom Types Interfaces • Two point releases – 0.4.0 and 0.3.0 Validation Tests • 0.4.0 • Release date: Feb 04, 2016 • Introduced support for TOSCA Simple Profile for NFV v1.0 Node • Support for TOSCA Groups and TOSCA Policies App • Support for TOSCA namespace, nested properties, bug fixes etc. • 0.3.0 Mongo Node. DB js Release date: Dec 02, 2015 • Full validation of TOSCA templates • HostedOn HostedOn Support for nested imports in service template and type definition • Shell entry point, bug fixes etc. • App Mongo • New features available on master Server DBMS Support for LoadBalancer node type • Mongo Support for range data type etc. • Server HostedOn • PyPi releases are available at https://pypi.python.org/pypi/tosca-parser
Mitaka: Heat-Translator TOSCA • What is the Heat-Translator? An OpenStack project to map and translate non-Heat (e.g. TOSCA) • templates to Heat Orchestration Template (HOT) Subproject of OpenStack Heat • • Two point releases – 0.4.0 and 0.3.0 • 0.4.0 Release date: March 02, 2016 • Support for TOSCA NFV templates translation • Introduced support for Policies translation • Completed OpenStack Client plug-in work with new test suites • Handle Nova server specific key_name property, bug fixes etc. • • 0.3.0 Heat Orchestration Template (HOT) Release date: Dec 17, 2015 • Querying Nova flavors to find match to TOSCA constraints • New CLI option --validate-only to only validate template without • translation New CLI option --output-file to save translated output to file etc. • • New features available on master New CLI option --deploy to automatically deploy translated template • Support for Ansible and Puppet scripts in translation etc. • PyPi releases are available at, https://pypi.python.org/pypi/heat-translator •
TOSCA OpenStack Integration is Maturing Rapidly Tacker Translate Deploy Heat-Translator TOSCA-Parser TOSCA Plugin TOSCA Integration OSC Plugin apps.openstack.org
CERN and the Large Hadron Collider • 21 member states Many other involved in other ways • • Large Hadron Collider (LHC) 27km long • 100m underground • • Particle collisions at specialized detectors ~600 million times / sec • Generating ~30PB of data / year • • CERN Cloud in numbers ~6000 nodes • ~150k cores • 2000 users • 16000 VMs running •
Indigo DataCloud • Towards a sustainable European SW infrastructure for science • Funded under the European Union Horizon 2020 program • Multiple partners, heterogeneous infrastructure (OpenStack, OpenNebula) INFN, CERN, ATOS, UPV, CSIC, DESY, UPV, LIP, T-Systems, CNRS, Reply, … • • Main strengths Support for multi-disciplinary scientific communities • Exploits available, general solutions • Low learning curve, rely on popular software tools • Hybrid, distributed environment • • Credits German Molto (UPV), Miguel Caballer (UPV), Marica Antonacci (INFN), Alvaro Lopez Garcia • (CSIC), Giacinto Donvito (INFN) And many others... • https://www.indigo-datacloud.eu
Why TOSCA? • First step was to evaluate available options • HEAT, CloudFormation • Both tied to specific implementations • TOSCA appeared as a viable common denominator • Topology definition • End user applications • Existing codebase (TOSCA parser, HEAT translator) • Reusable in other contexts too • Growing support in different communities
Current Use Cases • Deploy simple VM with a given image • Elastic cluster for batch processing (infrastructure) • Supported systems: torque, slurm, condor • Deploy a Mesos cluster (infrastructure) • Galaxy portal (end user) • Data intensive application for biomedical research • Indigo specific jobs (infrastructure / end user) • Package and deploy applications in docker containers, on Mesos/Marathon/Chronos
Basic example: Indigo Compute Node simple_node: tosca_definitions_version: tosca_simple_yaml_1_0 type: tosca.nodes.indigo.Compute properties: imports: public_ip: yes - indigo_custom_types: custom_types.yaml capabilities: scalable: description: > properties: TOSCA test for launching compute node with a specified image and getting count: 1 as an output the IP and SSH credentials to access host: topology_template: properties: node_templates: num_cpus: 1 mem_size: 1 GB os: outputs: properties: node_ip: type: linux value: { get_attribute: [ simple_node, public_address, 0 ] } distribution: ubuntu version: 12.04
Basic example: Derived Types tosca.nodes.indigo.Compute: derived_from: tosca.nodes.indigo.MonitoredCompute tosca.nodes.indigo.MonitoredCompute: interfaces: derived_from: tosca.nodes.Compute Standard: properties: create: zabbix_server: implementation: zabbix_agent_install.yml type: string configure: required: no implementation: zabbix_agent_configure.yml default: 90.147.170.165 inputs: zabbix_server_port: zabbix_server: { get_property: [ SELF, zabbix_server ] } type: PortDef zabbix_server_port: { get_property: [ SELF, zabbix_server_port ] } required: no zabbix_server_metadata: { get_property: [ SELF, zabbix_server_metadata ] } default: 32314 start: zabbix_server_metadata: implementation: zabbix_agent_start.yml type: string required: no default: Linux 668c875e-9a39-4dc0-a710-17c41376c1e0
simple_node: Basic example: HOT type: OS::Nova::Server properties: flavor: m1.small image: ubuntu-12.04-software-config-os-init public_ip: true user_data_format: SOFTWARE_CONFIG simple_node_create_config: heat_template_version: 2013-05-23 ... description: > simple_node_create_deploy: TOSCA test for launching compute node with a specified ... image and getting as an output the IP and simple_node_configure_config: SSH credentials to access type: OS::Heat::SoftwareConfig parameters: {} properties: resources: config: get_file: zabbix_agent_configure.yml outputs: group: ansible node_ip: simple_node_configure_deploy: value: type: OS::Heat::SoftwareDeployment get_attr: properties: - simple_node config: - networks get_resource: simple_node_configure_config - private input_values: - 0 zabbix_server: 90.147.170.165 zabbix_server_metadata: Linux 668c875e-9a39-4d... zabbix_server_port: 32314 server: get_resource: simple_node
Basic example: Ansible Integration - hosts: localhost connection: local tasks: interfaces: ... Standard: - name: Apt install Zabbix agent create: apt: name=zabbix-agent implementation: zabbix_agent_install.yml when: ansible_os_family == "Debian" configure: Implementation: zabbix_agent_configure.yml inputs: zabbix_server: { get_property: [ SELF, zabbix_server ] } zabbix_server_port: { get_property: [ SELF, zabbix_server_port ] } zabbix_server_metadata: { get_property: [ SELF, zabbix_server_metadata ] } start: implementation: zabbix_agent_start.yml tasks: - lineinfile: | dest=/etc/zabbix/zabbix_agentd.conf regexp="{{ item.var }}=" line="{{ item.var }}={{ item.value }}" with_items: - { var: "ServerActive", value: "{{zabbix_server}}:{{zabbix_server_port}}"}
Basic example: Ansible Integration (Galaxy Roles) - hosts: localhost connection: local roles: - role: indigo-dc.zabbix-agent zabbix_agent_server: "{{ zabbix_server }}" zabbix_agent_server_port: "{{ zabbix_server_port }}" zabbix_agent_metadata: "{{ zabbix_server_metadata }}" interfaces: Standard: configure: implementation: zabbix_agent_configure.yml inputs: zabbix_server: { get_property: [ SELF, zabbix_server ] } zabbix_server_port: { get_property: [ SELF, zabbix_server_port ] } zabbix_server_metadata: { get_property: [ SELF, zabbix_server_metadata ] }
Recommend
More recommend
