the logs you're looking for Alicja Kwasniewska, Intel Eric - - PowerPoint PPT Presentation

These aren't

the logs

you're looking for

Alicja Kwasniewska, Intel Eric Lemoine, Mirantis Michal Jastrzebski, Intel

Alicja Kwasniewska Eric Lemoine Michal Jastrzebski

Agenda

Kolla Logging in dockers Hacking Rsyslog Central logging service Elk or Ehk? Demo

Kolla

Docker containers Ansible playbooks

Kolla

• Production-ready Docker containers and deployment tools for
• perating OpenStack clouds
• Both out of the box configuration and possibility for complete

customization

• Images built in the kollaglue namespace for every tagged

release, but you can also build images on your own Docker containers Deployment tools Out of the box configuration Kollaglue Customization

Logging in

docker containers

Docker logs

• What? Anything that is written out to the container’s standard output/error
• Logging capabilities available in the Docker are exposed as drivers

Default

• How to get path to this file? docker inspect -f "{{.LogPath}}" containerName
• The docker logs command retrieves logs present at the time of execution.

NOTE: available only for containers with json-file and journald logging drivers

json syslog journald gelf fluentd

Hacking

rsyslog

Previous solution based on rsyslog

• Hacky
• Shared /dev/log
• Not meant for containers
• New versions of docker breaks
• Syslog doesn’t work well with tracebacks
• Removed in both Liberty and Mitaka

Central logging service

advantages

“One Ring to rule them all, One Ring to find them...”

Which node? Which container? Where should I look first?

No need to search many places ! → data stored in one location and accessed from single interface

One more second and I will find the right place in one of the 10000+ lines...

“Developer looking at production logs after a regression with downtime”

• il canvas, circa 1580; source: http://classicprogrammerpaintings.tumblr.com

No need to search for one line in million ! → interface provides filtering options and visual representation of data using graphs and charts

Kolla Central logging service

New feature in Mitaka

How it works?

Heka

Feed data

Elasticsearch

View

Kibana

/var/lib/ kolla/... Search Interact

Heka

F e e d d a t a /var/lib/k

• lla/...

Heka

Feed data /var/lib/k

• lla/...

Heka

Unified data processing

Introduction to Heka

Stream processing software

• Open source
• Developed by Mozilla
• Used in production (by Mozilla)
• Written in Go
• http://hekad.readthedocs.org

Why Heka?

→“Unified Data Processing” One tool for all the data:

• Ops data/metrics
• Business data
• Logging output
• Error

reports/tracebacks

Ops data/ metrics Business data Logging

• utput

Error reports/ tracebacks

Heka Pipeline

Plugins at each stage of the pipeline

Heka Highlights

• Lightweight
• Flexible
• Configurable
• Easily and safely extendable

Lightweight Flexible Configurable Easily and safely extendable

ELK or EHK?

Logstash vs Heka

Why did we choose Heka over Logstash?

• Avoid running a JVM on each node
• Heka is faster and more lightweight*
• Heka is very flexible (plugins defined w/ code)

* https://github.com/elemoine/heka-logstash-comparison

→ EHK

Elasticsearch

Open-source text search and analytics engine

Introduction to Elasticsearch

• Open source product from Elastic group - https://github.

com/elastic/elasticsearch

• Highly-scalable full-text search and analytics engine
• Used by applications with complex search requirements
• Written in Java

Elasticsearch Highlights

Highly Scalable Highly Available RESTful API Full-Text Search Document-

• riented

Kibana

Open-source analytics and visualization platform

Introduction to Kibana

• Open source product from Elastic group - https://github.

com/elastic/kibana

• Written mainly in JavaScript, comes with Node.js server app
• Server component has a built-in proxy to Elasticsearch (designed to

work with Elasticsearch)

Kibana concepts

• Allows to search, view, and interact with data stored in Elasticsearch indices
• Possibility to analyse and visualize data in a variety of charts, tables, and

maps Search Interact View Visualize Analyse

Kibana - authentication

Elastic Shield - provides security for entire stack, including Kibana plugin for session support and login screen, but requires a license… Nginx - use it as proxy with simple auth mechanism for Kibana HAProxy - add HTTP Basic authentication with ACL that uses the http_auth criterion and http-request auth rule Shield From Elastic Group Nginx HAProxy HTTP Basic authentication with ACL

Deploy

EHK

with

Kolla

3 easy steps

Build images with one command Adapt configuration files and enable central logging Deploy Kolla with one command

For details please watch webinar presented by Steven Dake - link provided at the last slide

Demo →

Thank you for your attention

Alicja Kwasniewska alicja.kwasniewska@intel.com Eric Lemoine elemoine@mirantis.com Michal Jastrzebski michal.jastrzebski@intel.com

Irc: #openstack-kolla Webinar: https://vimeopro.com/midokura/345kl392