Towards Omnia: a Monitoring Factory for Quality-Aware DevOps Apr 27 - PowerPoint PPT Presentation

Towards Omnia: a Monitoring Factory for Quality-Aware DevOps Apr 27 th , 2017 Marco MIGLIERINA Damian A. TAMBURRI

Dev- to Ops: We are moving faster and faster 2 Figures from: http://martinfowler.com/articles/microservices.html and https://www.slideshare.net/CiscoDevNet/enabing-devops-in-an-sdn-world

Observability is essential 3

Monitoring lagging behind …back in 2011 4

Since then… proliferation of tools and solutions 144 practitioners surveyed 5 M. Miglierina. Monitoring Modern Distributed Software Applications: Challenges And Solutions. PhD thesis, Politecnico di Milano, 2017.

Main perceived drawback in monitoring? 1. Lack of standards 2. Too many tools 3. Lack of usability 6 M. Miglierina. Monitoring Modern Distributed Software Applications: Challenges And Soloutions. PhD thesis, Politecnico di Milano, 2017.

Omnia • Main objectives • reduce learning curve and entry cost to monitoring • an attempt of standardization • How? • One, interoperable, self-service monitoring interface for devs • A simple monitoring factory for ops Monitoring OPS DEVS Stack 7 use manage

Reference team organization • Product teams • each responsible of its microservice • independent workflows • Platform team • provide infrastructure support • cross functional wrt product teams Product Team Product Team Platform Team A P Product Team I Prod DB Sys Net SAN 8 UX Dev QA Mgr Admin Admin Admin Admin Team organzation at Netflix: https://www.nginx.com/blog/adopting-microservices- at-netflix-lessons-for-team-and-process-design/

The Omnia components Monitoring Monitoring Interface factory Product Team Omnia Libs Omnia CLI Product Team Monitoring Platform Team omnia.yml Stack Product Team omnia.admin.yml Omnia Product Team Vocabulary 9

Classical Approach (1) The Platform Team decides to use the monitoring stack X Agents X TDB X Service Alfa Platform Dashboard Team X maintains learn, configure, deploy Alerting X Product Team Alfa 10 automated action manual action

Classical Approach (2) Product Teams have to learn how to use X and Agents X instrument their code TDB X Service Alfa v0.1 TDB X Lib Platform Dashboard Team X learn, code, release learn, configure learn, configure Alerting X Product Team Alfa 11 automated action manual action

Classical Approach (3) Agents X pushData TDB X Service Alfa pushData v0.1 TDB X Lib Platform Dashboard Team X view Monitoring stack X is now up and Alerting X Product Team running Alfa 12 sendAlerts automated action manual action

Classical Approach (4) The Platform Team decides to migrate to monitoring stack Y Agents Y Service Alfa v0.1 TDB X Lib Platform Team Monitoring Tool Y learn, configure, deploy Product Team Alfa 13 automated action manual action

Classical Approach (5) Product Teams have to Agents Y learn how to use Y and re- instrument their code Service Alfa v0.2 AgY Lib Platform Team Monitoring learn, code, release Tool Y learn, configure Product Team Alfa 14 automated action manual action

Classical Approach (6) Agents Y pushData Service Alfa pullData v0.2 AgY Lib Platform Team Monitoring Tool Y view Monitoring stack Y is now up and running Product Team sendAlerts Alfa 15 automated action manual action

Omnia-based approach (1) By using Omnia a single omnia.admin.yml learning step is required v1 for all teams omnia.yml learn, code Service Alfa v0.1 Omnia Lib Platform learn, code, release Team Product Team 16 Alfa automated action manual action

Omnia-based approach (2) parse omnia.admin.yml Omnia CLI parse v1 Agents X omnia.yml use TDB X Service Alfa v0.1 Omnia Lib Dashboard X Platform Team Omnia deploys and configures the monitoring stack X from Alerting X configure, deploy Product Team code 17 Alfa automated action manual action

Omnia-based approach (3) omnia.admin.yml Omnia CLI v1 Agents X pushData omnia.yml TDB X Service Alfa pushData v0.1 Omnia Lib Dashboard X Platform Team view Monitoring stack X sendAlerts Alerting X is now up and Product Team running 18 Alfa automated action manual action

Omnia-based approach (4) omnia.admin.yml Omnia CLI v2 Agents X omnia.yml code TDB X Migration to a new Service Alfa v0.1 monitoring stack is a Omnia Lib simple code change made by the Platform Team Dashboard X Platform Team Alerting X Product Team 19 Alfa automated action manual action

Omnia-based approach (5) parse omnia.admin.yml Omnia CLI parse v2 Agents Y omnia.yml use Service Alfa v0.1 Omnia Lib configure, deploy Monitoring Tool Y Platform Team Omnia takes care of reconfiguring and Product Team redeploying the stack 20 Alfa automated action manual action

Omnia-based approach (6) The Product Teams did not have to omnia.admin.yml change a line Omnia CLI v2 Agents Y pushData omnia.yml Service Alfa v0.1 pullData Omnia Lib Monitoring Tool Y Platform Team view Monitoring stack Y sendAlerts is now up and Product Team 21 running Alfa automated action manual action

Omnia Libs • Requirements: • independent of the monitoring stack • minimize instrumentation overhead • Example: Omnia Lib for Spring Boot 22

Under the hood • Convention over configuration: • statsd (w/ influxdb tag ext.) protocol • automated meta-data decoration • default endpoint • Example: payments 34 Spring Boot Omnia Lib 23 POST “payments,host=vm1,service=payservice 34” http://collector:8125

Omnia.yml: monitoring config as code • Requirements • independent of the monitoring stack • versionable with the code • Example: dashboard.tmpl Go Template Library 24 dashboard

The Omnia vocabulary • Shared vocabulary for resources • and for metrics 25

Monitoring infra as code omnia.admin.yml example: v2 v1 Reusing existing monitoring tools Reusing existing provisioning tools Reusing existing adapters 26

Conclusion • Contributions • Reduce entry cost and learning curve to monitoring • Application of DevOps practices to monitoring • Attempt of standardization • Threats to validity • A common interface may simplify tools characteristics • However: • initial approach to monitoring has simple requirements • it could push tool vendors to implement missing features • the approach could be applied to existing or new tools • Future work • Extensive evaluation using real world examples 27