Goutham Veeramachaneni @putadent Me Goutham Veeramachaneni | 2 - PowerPoint PPT Presentation

Goutham Veeramachaneni @putadent

Me Goutham Veeramachaneni | 2

Debugging slow queries Cortex | 3

| 4

A few large queries are slow | 5

RED Method | 6

Soooo whats slow? Hypothesis Experiment Metrics | 7

Saving my arse since June 2018 | 8

Soooo whats slow? | 9

Soooo whats slow? | 10

| 11

Is Jaeger the solution to everything? | 12

My new flow | 13

Per environment traces? Several clusters but central jaeger ● | 14

Enriching traces blog ● | 15

Enriching traces Just add namespace and clustername, use env vars ● jaeger_mixin:: if $._config.jaeger_agent_host == null then {} else container.withEnvMixin([ container.envType.new('JAEGER_AGENT_HOST', $._config.jaeger_agent_host), container.envType.new('JAEGER_TAGS', 'namespace=%s,cluster=%s' % [$._config.namespace, $._config.cluster]), container.envType.new('JAEGER_SAMPLER_MANAGER_HOST_PORT', 'http://%s:5778/sampling' % $._config.jaeger_agent_host), ]), | 16

Per environment traces? Several clusters but central jaeger ● Dashboard Jaeger Alerts | 17

Monitoring Jaeger with Prometheus | 18

We’re only scratching the surface | 19

Exemplars: Seamless flow | 20

Exemplars: Seamless flow | 21

Exemplars: Seamless flow | 22

More future Grafana has tracing built in (Jaeger being first) ● ● Integrate the logs and traces and metrics Built in UI for visualising Prometheus queries ● ● Conprof | 23

Cortex | 24

Recap Jaeger is awesome for debugging latency ● Alerts —> Dashboard —> Jaeger, make sure the flow works ● The jaeger_mixin helps you tame Jaeger ● Exemplars are coming! ● | 25

@putadent