CloudKitty Hands-on 1 / 56 Lets meet your hosts! 2 / 56 Lets meet - PowerPoint PPT Presentation

CloudKitty Hands-on 1 / 56

Let’s meet your hosts! 2 / 56

Let’s meet your hosts! Today’s speakers Luka Peschke (Objectif Libre) Cloud Consultant / CloudKitty PTL Flavien Hardy (Objectif Libre) Cloud Consultant / CloudKitty contributor Christophe Sauthier (Objectif Libre) CEO of Objectif Libre / CloudKitty co-father 3 / 56

Today’s tools 4 / 56

Today’s tools 5 / 56

Today’s tools Ceilometer Openstack measurement project Ceilometer (part of the Telemetry project) collects the usage of all resources in an OpenStack cloud. It stores metrics , like CPU and RAM usage, amount of volume storage used… 6 / 56

Today’s tools → Ceilometer Architecture Ceilometer is composed of several parts. The main ones are: ceilometer-agent-notification (controller): reads AMQP messages from other components. ceilometer-agent-central (controller): polls some metrics directly. ceilometer-agent-compute (compute node): polls information related to instances. 7 / 56

Today’s tools Gnocchi Timeseries Database Gnocchi was initially created as a part of the Telemetry project to address Ceilometer’s storage and performance issues. It is independent since March 2017. It stores and aggregates measures for metrics. Gnocchi has a resource notion. Each resource can have several associated metrics. (For example, an instance resource has cpu , vcpus and memory metrics associated). Metadata is linked to resources and not to metrics. Ceilometer publishes measures to Gnocchi (but it does also support other databases). 8 / 56

Today’s tools → Gnocchi Architecture Gnocchi is composed of the following parts: An HTTP REST API: Used to push and retrieve data. A processing daemon ( gnocchi-metricd ): Performs aggregation, metric cleanup… A statsd-compatible daemon (optional): Receives data via TCP rather than the API. 9 / 56

Today’s tools CloudKitty Rating component for OpenStack and co CloudKitty was initially created in order to allow rating of Ceilometer metrics. Today, CloudKitty can be used with Gnocchi, Monasca and Prometheus. It is possible to use it outside of an OpenStack context since Rocky. 10 / 56

Today’s tools → CloudKitty Architecture Processor Storage API Orchestrator Modular component Rating Collector Fetcher Fixed component module(s) 11 / 56

Today’s tools → CloudKitty CloudKitty works the following way: The fetcher retrieves scopes from which information should be gathered (in our case, these will be projects for openstack and job IDs for prometheus). The collector collects measures from somewhere (gnocchi and prometheus in our case) for the given scopes. The collected data is passed to CloudKitty’s rating module(s) (several rating modules can be used simultaneously). The modules apply user-defined rating rules to the data. We will use the HashMap module for today’s workshop. The rated data is pushed to CloudKitty’s storage backend (InfluxDB in our case). 12 / 56

Installing our components 13 / 56

Installing our components Get your browser Slides: https://olib.re/denver-ck-handson Do not open them in your browser or you’ll experience copy/paste issues ! Pick an IP: https://olib.re/denver-ck-handson-ip 14 / 56

Installing our components SSH into your instance Start with SSHing into your instance. The user is centos and the password is d3nv3r . Once you’re connected, identify yourself: $ source ~/admin.sh 15 / 56

Installing our components Create some data $ for i in {1..3}; do openstack server create --image cirros --flavor m1.nano instance${i} openstack server create --image windows --flavor m1.nano instance-win${i} openstack image create --file ~/handson_files/cirros.img image${i} done Once you’ve populated the admin tenant, do the same for summit : $ source ~/summit.sh $ for i in {1..3}; do openstack server create --image cirros --flavor m1.nano instance${i} openstack server create --image windows --flavor m1.nano instance-win${i} openstack image create --file ~/handson_files/cirros.img image${i} done When you’re done, identify as an admin again: $ source ~/admin.sh 16 / 56

Installing our components Getting the Stein RDO repositories First, let’s install the RDO Stein repositories. $ sudo yum remove -y centos-release-ceph-luminous $ sudo yum install -y centos-release-openstack-stein 17 / 56

Installing our components Installing CloudKitty, its Horizon dashboard & client Daemons: $ sudo yum -y install openstack-cloudkitty-{api,processor} Client: $ sudo yum -y install python-cloudkittyclient Dashboard: $ sudo yum -y install openstack-cloudkitty-ui $ sudo systemctl restart httpd Note Restarting httpd can take a minute: on restart, the service collects and compresses static assets for horizon dashboards. 18 / 56

Installing our components → Note Adding CloudKitty to Keystone $ openstack service create --name cloudkitty rating +---------+----------------------------------+ | Field | Value | +---------+----------------------------------+ | enabled | True | | id | ef63cce9085443d5b95d9558b6176f90 | | name | cloudkitty | | type | rating | +---------+----------------------------------+ 19 / 56

Installing our components → Note Creating the cloudkitty user $ openstack user create --project service --password password cloudkitty +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | default_project_id | 8eba65aaa579413fb1dd7ff252caa0a4 | | domain_id | default | | enabled | True | | id | faa6b264724946c7bec4fc64376687a4 | | name | cloudkitty | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+ $ openstack role add --user cloudkitty --project service admin 20 / 56

Installing our components → Note Creating CloudKitty’s endpoints $ for i in public internal admin; do openstack endpoint create --region RegionOne rating $i http://127.0.0.1:8889/ done 21 / 56

Installing our components → Note Creating CloudKitty’s database $ mysql -uroot -popenstack << EOF CREATE DATABASE cloudkitty; GRANT ALL PRIVILEGES ON cloudkitty.* TO 'cloudkitty'@'localhost' \ IDENTIFIED BY 'cloudkittydbpassword'; GRANT ALL PRIVILEGES ON cloudkitty.* TO 'cloudkitty'@'%' \ IDENTIFIED BY 'cloudkittydbpassword'; EOF 22 / 56

Installing our components Configuring CloudKitty Note Please edit these files before you copy them: change period = 3600 to period = 300 . $ sudo cp ~/handson_files/cloudkitty.conf ~/handson_files/cloudkitty-prometheus.conf /etc/cloudkitty/ Most options of this file ( keystone_authtoken , transport_url, database…) are classical OpenStack options. However, some of them are specific to cloudkitty. CloudKitty supports several storage drivers (SQLAlchemy in v1, InfluxDB in v2). For this session, we will use the InfluxDB storage driver . [storage] backend=influxdb version=2 23 / 56

Installing our components → Note Configuring CloudKitty (gnocchi) For the processor collecting information from gnocchi, we’ll use the gnocchi fetcher. This fetcher discovers scopes from unique attributes on a given resouce type. By default, it returns all unique project_id attributes from the generic resource type. The fetcher section contains general fetcher options (which fetcher to use), and fetcher_gnocchi contains options specific to the gnocchi fetcher: [fetcher] backend = gnocchi [fetcher_gnocchi] auth_section=authinfos We use the gnocchi collector, so we specify authentication options in the collector_gnocchi section. [collect] collector = gnocchi period = 3600 metrics_conf = /etc/cloudkitty/metrics-gnocchi.yml # DO NOT USE IN PRODUCTION wait_periods = 0 [collector_gnocchi] auth_section=authinfos 24 / 56

Installing our components → Note Configuring CloudKitty (prometheus) For prometheus, we will use the source fetcher, which reads a list of scopes to process form the configuration file: [fetcher] backend = source [fetcher_source] # Our sources are job IDs sources = monitoring-1, monitoring-2 [collect] collector = prometheus period = 3600 metrics_conf = /etc/cloudkitty/metrics-prometheus.yml # DO NOT USE IN PRODUCTION wait_periods = 0 # The key at which the scope can be found in prometheus scope_key = job 25 / 56

Installing our components → Note Copy the metric collection configuration files to /etc/cloudkitty : $ sudo cp ~/handson_files/metrics-gnocchi.yml ~/handson_files/metrics-prometheus.yml /etc/cloudkitty/ 26 / 56