Containerizing Databases at New Relic (What We Learned) Bryant Vinisky and Joshua Galbraith Santa Clara, California | April 23th – 25th, 2018
Safe Harbor This presentation and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an o ff er, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward- looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import. Actual results may di ff er materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to di ff er materially from those expressed or implied in any forward-looking statement. Further information on factors that could a ff ect such forward-looking statements is included in the filings New Relic makes with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this presentation or otherwise, with respect to the information provided. � 2
Introductions New Relic • A cloud platform to make every aspect of modern software and infrastructure observable! Bryant Vinisky • Senior Site Reliability Engineer • Database Engineering Team Joshua Galbraith • Senior Software Engineer • Database Engineering Team � 3
Agenda • Where We Started • Research • Prerequisites • Megabase • Monitoring • Lessons Learned • Outcomes � 4
Where We Started Databases at New Relic circa 2016
Database Management Issues 📧 Using Puppet for configuration and deployment 🚛 Slow delivery time due to timing of hardware orders 💹 Inefficient hardware use � 6
Why Containers? • Why not use virtual machines instead? • Why not use AWS? • Why not multi-tenant logical databases? � 7
Preparing for the Future 🤕 New Container Fabric for deploying our stateless apps 🚣 Future regions will be entirely container-based! 🛥 Incremental delivery of containerized databases? � 8
Setting Goals 📧 Packaging and Deployment • consistent and repeatable 🚛 Database Delivery Time • from months to minutes 💹 Cost Efficiency • reduce wasted resources � 9
Compressed Timeline 😆 Managing hundreds of existing, busy databases Need to ship an MVP and gain traction quickly Dev work is blocked on database delivery time � 10
Research A Survey of Open-Source Orchestration Tools
Open-Source Container Orchestration � 12
Stateful Containers: Mesos and Marathon Key Concepts • Dynamic Provisioning • Reservation Labels • Local Persistent Volumes • External Volumes � 13
Stateful Containers: Kubernetes Key Concepts • Stateful Sets • Pods • Headless Service • Persistent Volumes • Operators � 14
Stateful Containers: Nomad Key Concepts • Jobs • Task Groups • Allocations • Sticky Volumes • Volume Plugins • FS Drivers � 15
Joyent Blog: Autopilot for Databases � 16
Stateful Containers: Emergent Patterns Abstract Patterns • Application-aware orchestration • Autopilot pattern for lifecycle management • Storage fabrics and networked storage 😟 � 17
Stateful Containers: Current Status � 18
Making Trade-Offs • Dynamic Scheduling vs. Manual Placement • Custom Orchestration Framework vs. Client-Server • Distributed Consensus? • Local Object Storage? • Lifecycle Management logic inside or outside Container? � 19
Prerequisites Dynamic Inventory and/or Service Discovery
Problem: Database Inventory What systems do we have? What logical databases? What team is the owner? Naming standards? API and CLI access? � 21
Megabase Prerequisite: Inventory System Metadata on DB systems and services Percona containers on Megabase Golang service HTTP interface to DB Update on event and scheduled jobs Seed automation tasks � 22
Inventory System: Database • Query read only data from other systems and authorities • Megabase container deployment information • Provide basic service discovery � 23
Inventory System: HTTP Interface (JSON) Query over HTTP Return JSON � 24
Inventory System: HTTP Interface (text) � 25
Inventory System: Dynamic Lookups � 26
Megabase A platform for containerized databases at New Relic
Megabase: What We Built � 28
Megabase: Ingredients Bare Metal: • Kernel 4.4 and 4.14 • CentOS 7 and CoreOS Docker: 1.12 and 1.17 Image OS: • Alpine (Postgres and Redis) • Debian (Percona) Data stores: • Percona-server 5.6 • Percona-server 5.7 • Postgresql-server 9.3, 9.5 and 9.6 • Redis-server 3.2 Golang: 1.9 � 29
Megabase: Docker Image Building Dockerfile • Upstream base image • Custom labels • Package dependencies • Add binaries: - rclone - configuration sync - replication bootstrap • Entrypoint script � 30
Megabase: Docker Entrypoint Tasks • Validate data mount • Sync base configs from S3 • Apply dynamic configuration • No data => replication bootstrap/initdb • Start server processor � 31
Megabase: Injecting Configuration Strategies • Environment variables - via deploy config • Configuration files - via object storage - version controlled • Dynamic computation from cgroup limits • Docker images - via image registry � 32
Megabase API Server Pre/Post deployment dependencies Manage container runtime dependencies Interface to docker over https Support custom workflows for database services megabase server � 33
Megabase API: TLS Authentication Authenticate all requests with TLS client certificates � 34
Authentication Failure � 35
Authentication Success � 36
Megabase API: Endpoints � 37
Megabase API: Dependencies Containers expect and validate bind mount under data directory Megabase deploy pre-step to docker run: • Carve off extents for logical volume and make filesystem • Create systemd unit file for persistence and mount volume � 38
Megabase API: Create Logical Volume � 39
Megabase API: Container Runtime Reduce client config burden Offload predictable defaults: • Resource Limits • Bind mount volume to data directory • Host networking • Inject environment secrets Example docker run on cli => � 40
Megabase API: Failover Tasks � 41
Megabase API: Xtrabackup Stream to Peer � 42
Megabase… Client? “So we’re just supposed to type out those nasty, long curl commands to operate everything? Really? And TLS client auth too? Are you trying to be annoying?” - anonymous New Relic db-team member mb client � 43
Megabase Client: mb � 44
Megabase Client: mb mb: Command Line Interface • wrap up server API functionality into command line interface • transparently handle TLS authentication • query state, health and configuration across servers and deployments • coordinate deployments across servers • leverage inventory to resolve host and container names � 45
Megabase Client: mb � 46
Megabase Client: Deployment Manifest Minimum info required: • Container name • Image path and tag • CPU, memory and disk resources • VIP for load balancer and DNS • Owning team Auto select values if absent Define one or more containers � 47
Megabase Client: mb Deployment Process • Target environment and cluster • Specify a deployment manifest • Generate port and passwords • Send requests to API servers - new logical volume - run container • Check responses • Validate deployment - health � 48
Monitoring Using New Relic to Monitor New Relic
Recommend
More recommend
