Use Saltstack to deploy a full monitoring and supervision stack - PowerPoint PPT Presentation

Use Saltstack to deploy a full monitoring and supervision stack #cfgmgmtcamp18 Arthur Lutz ‐ Logilab #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

(part of) who am I Arthur Lutz Logilab Python dev / Sysadmin / Debian / Devops SaltStack Certi�ed Engineer Paris Salt Meetup Python Nantes Meetup / Nantes Monitoring Meetup #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Big picture ‐ scien�fic cloud applica�on Complex scaling parameters salt-based auto-scaling, scale up and down based on application logic scaling is based on metrics collected by salt automatically deploy application-oriented monitoring #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

dashboards from diverse data grafana provides data source diversity we're looking for data origin diversity, and agile ways to deploy them system (internal) data cloud (external) data application-speci�c data autoscaling state, etc... #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

agility by design consistency by design easily de�ne and add metrics compare and test metrics from different points of view #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Overview of components SaltStack - huge toolkit to manage infrastructure Netdata - collect metrics, visualise "realtime" status of server Sensu - devops oriented monitoring system Graphite / Carbon - to collect and query time series data Grafana - to build and display dashboards on the collected data #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Components used salt pillars salt scheduler salt mine salt returners salt runners salt reactor salt-cloud ... "which ones did we not use?" #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Salt formulas salt-formula (what else?) nginx-formula nfs-formula nagios-formula sensu-formula rabbitmq-formula redis-formula #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Role based approach Pillars (with pillar_merge_list: True ) : base: '*': - role.sensu-client 'saltmaster.local': - role.sensu-server States : base: 'I@role:sensu-client': - sensu.client 'I@role:sensu-server': - sensu.server #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Salt mine send unique ID used by Netdata to stream clients to central instance states : {{ salt['mine.get']('role:netdata-master', 'network.interface_ip' expr_form='pillar').values()[0] }} use to retrieve location of Rabbitmq server used by Sensu . pillars : {% set rabbitmq = salt.saltutil.runner("mine.get", tgt='role:rabbitmq', fun="network.interface_ip", tgt_type='pillar').values() %} #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Salt returners Carbon returner to push metrics from application and infrastructure returner address is con�gured in pillars (cf extraction from mine) #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Salt custom modules Custom modules are shipped by salt master from _modules folder celery statistics (use app.inspect.stats ) core application metrics using HTTP API sentry (error collector) metrics using the sentry python binding (which uses the sentry HTTP API) #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Custom module ‐ celery inspect def stats(): ''' Provide stats from celery tasks via app.inspect.stats() [snip] ''' ret = __salt__['redis.sentinel_get_master_ip']('name') r_url = 'redis://{}:{}/0' app = Celery(broker=r_url.format(ret['master_host'], ret['master_port'])) stats = app.control.inspect().stats() _clean(stats) return stats #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Salt‐cloud enables custom auto-scaling system portable between cloud infrastructures enables to run infrastructure on local openstack (or even devstack) #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Salt reactor ‐ event driven! react to new VM connecting get info about VM (for example kernelversion ) upgrade kernel to linux-backports reboot system deploy monitoring deploy application Ended up being replaced by Packer generated AMI. #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Let's zoom in #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Visualising complexity #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

monitoring ‐ how launch a salt runner to pool the metrics salt.execute runner is used to get infrastructure view of a given metric cloud.profile runner is used to provision new VMs cloud.destroy runner is used to scale down unused VMs #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

monitoring ‐ how Netdata sends system metrics central Netdata sends data to Graphite Salt sends custom metrics to Graphite Sensu executes checks (subscriber based model) Sensu collects metrics and sends them to Graphite Grafana uses the metrics stored in Graphite to display dashboards #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

compute node minion MASTER minion client sensu­server compute node sensu­api minion web client storage #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Demo �me! install netdata role based architecture auto con�gure streaming by pushing the API id to the mine install graphite deploy netdata graphite con�guration install grafana con�gure grafana datasource with salt con�gure grafana dashboard with salt #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Demo �me! install rabbitmq with rabbitmq-formula for sensu install sensu with sensu-formula use roles to con�gure subscribed sensu checks con�gure uchiwa for web ui con�gure sensu metrics to send data to graphite update grafana dashboard with salt #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

What's next / Roadmap convert schedulers to beacons various optimisations use reactor instead of polling look into salt's thorium (complex reactor system) #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

End Thanks for your attention Questions ? Comments ? Improvements ? #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

sources http://slides.logilab.fr/2018/cfgmgmtcamp_saltsta ck_monitoring.pdf Demo : https://github.com/arthurlogilab/salt- vagrant-demo Contact / Follow us : https://www.logilab.fr @logilab @arthurlutz arthurlutz@social.logilab.org #cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab