Monitoring Kubernetes with Prometheus Henri Dubois-Ferriere - PowerPoint PPT Presentation

Monitoring Kubernetes with Prometheus Henri Dubois-Ferriere @henridf Percona Live, 2018-11-06

Hello. Henri Dubois-Ferriere Technical Director, Sysdig Doing “observability” for many many years, from network to web apps via many startups. PhD in CS from EPFL Repatriate from San Francisco to Switzerland

Outline ● Kubernetes ● Prometheus ● Kubernetes metrics & sources ● Deployment

Monitor why? ● Know about outages before users tell me ● Understand my production environment (or try…) ● Plan/trend/forecast

Kubernetes

Kubernetes - Container orchestration system - aka “OS for your cluster” - Abstracts away the underlying infra - declarative APIs with control loops

https://commons.wikimedia.org/wiki/File:Kubernetes.png

Prometheus

Prometheus ❏ Started at SoundCloud in 2012 ❏ Motivated by challenges with monitoring dynamic environments ❏ Made public 2015, now second CNCF “graduate”

More than a TSDB https://prometheus.io/assets/architecture.png

It’s all about the pull - Prom scrapes targets to get metrics - Nice side effect: know when target down - Needs to know what to scrape

What should Prometheus scrape? - Service discovery provides answer - Azure, Consul, GCE, K8S, EC2, ... - Can also watch a file containing target list

Dimensional data model Query: http_requests_total{code=”200”, method=”get”} Metric name Selector (aka filter)

Dimensional data model Query: http_requests_total{code=”200”, method=”get”} Response : http_requests_total {code="200", method=”get”, route="/api/users"} 1528706829.115 1741 http_requests_total {code="200", method=”get”, route="/api/objects"} 1528706829.115 1920 Label/value pairs (aka dimensions)

Dimensional data model Query: http_requests_total{code=”200”, method=”get”} Response : http_requests_total{code="200", method=”get”, route="/api/users"} 1528706829.115 1741 http_requests_total{code="200", method=”get”, route="/api/objects"} 1528706829.115 1920 Timestamp value

Metadata discovery - SD also provides metadata - Metadata can be mixed in with metrics - Powerful relabelling feature for label manipulation at ingest

Instrumentation

Off-the-shelf or write your own

Kubernetes metrics

Monitoring resources and methods - For resources like memory, queues, CPUs, disks… - USE Method: Utilization, Saturation, Errors - http://www.brendangregg.com/usemethod.html - For services - “RED” Method: Request rate, Error rate, Duration - https://www.weave.works/blog/the-red-method-key-metrics-for-micr oservices-architecture/

node_exporter: node metrics - Host metrics - CPU - Memory - Disk - Network - ... - Not K8S specific, but useful as referential and for totals

cAdvisor: container metrics - Runs in kubelet (usually, for now..) - Resource stats about running containers - Mostly container and node-level labels… - (k8s: plus namespace and pod_name)

Sample cAdvisor metric queries Percent of total cluster memory used: sum(container_memory_rss) / sum(machine_memory_bytes) Memory used by kubernetes namespace: sum(container_memory_rss) by (namespace) Top 5 pods by network I/O: topk(5, sum by (pod_name) (rate(container_network_transmit_bytes_total[5m])))

Kube-state metrics $ kubectl get deploy my-app -o yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-app ... spec: replicas: 4 ... status: replicas: 4 ...

Kube-state metrics $ kubectl get deploy my-app -o yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-app ... spec: kube_deployment_spec_replicas{ deployment= "my-app" , ... } replicas: 4 ... Metrics created by kube-state-metrics With label set from this deployment status: replicas: 4 kube_deployment_status_replicas{ deployment= "my-app" , ... } ...

Sample kube-state-metrics queries Deployments with issues kube_deployment_spec_replicas != kube_deployment_status_replicas_available Top 10 longest-running pods (“reverse uptime”) topk(10, sort_desc(time() - kube_pod_created))

Kube core service metrics - API Server - etcd3 - kube-dns - scheduler, controller-manager

Metrics recap Deployment mode How many Metrics about node_exporter daemonset 1 per node node resources cAdvisor inside kubelet 1 per node container resources kube-state-metrics deployment singleton k8s object state etcd, Api Server, core service singleton or HA group Itself controller manager, ...

Deploying

Monitoring from the inside - Monitoring runs inside thing being monitored? - Yes. It’s fine really. Really, it’s fine. - (And being outside has own challenges)

Deployment outline - Metrics services - node_exporter - kube-state-metrics - (cAdvisor usually enabled out of box) - Prometheus running - Storage - Read access to API server (for service discovery) - Service discovery config for above - Service discovery config for apps/services

Helm-based install helm fetch stable/prometheus vi prometheus/values.yaml # configure install helm upgrade -i # or manually deploy yaml

Prometheus operator - Use Kubernetes API facilities to make Prometheus “native” - new Prometheus-related objects: ` kubectl get prometheus` - PrometheusRule, ServiceMonitor, AlertManager, AlertingSpec, ... - Prometheus configuration abstracted via all these objects - Young but promising - Consider more direct route first (hand-rolled or Helm), and Operator once more familiar with challenges of direct route

Thank You. Henri Dubois-Ferriere @henridf

Pointers Prometheus SD for Kubernetes: - https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config KSM metrics: https://github.com/kubernetes/kube-state-metrics/tree/master/Documentation - Prometheus Helm chart: https://github.com/helm/charts/tree/master/stable/prometheus - Prometheus operator: https://github.com/coreos/prometheus-operator - “A deep dive into Kubernetes metrics” blog series: - https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-66936addedae