Nomad HASHICORP Armon Dadgar @armon HASHICORP HASHICORP Cluster - PowerPoint PPT Presentation

Nomad HASHICORP

Armon Dadgar @armon HASHICORP

HASHICORP

Cluster Manager Scheduler Nomad HASHICORP

Cluster Manager Scheduler Nomad HASHICORP

Schedulers map a set of work to a set of resources HASHICORP

Work (Input) Resources Web Server -Thread 1 CPU - Core 1 Web Server -Thread 2 CPU Scheduler Redis -Thread 1 CPU - Core 2 Kernel -Thread 1 HASHICORP CPU Scheduler

Work (Input) Resources Web Server -Thread 1 CPU - Core 1 Web Server -Thread 2 CPU Scheduler Redis -Thread 1 CPU - Core 2 Kernel -Thread 1 HASHICORP CPU Scheduler

Type Work Resources CPU Scheduler Threads Physical Cores AWS EC2 / OpenStack Nova Virtual Machines Hypervisors Hadoop YARN MapReduce Jobs Client Nodes Cluster Scheduler Applications Servers HASHICORP Schedulers In the Wild

Higher Resource Utilization Decouple Work from Resources Better Quality of Service HASHICORP Advantages

Higher Resource Utilization Bin Packing Decouple Work from Resources Over-Subscription Better Quality of Service Job Queueing HASHICORP Advantages

Higher Resource Utilization Abstraction Decouple Work from Resources API Contracts Better Quality of Service Standardization HASHICORP Advantages

Higher Resource Utilization Priorities Decouple Work from Resources Resource Isolation Better Quality of Service Pre-emption HASHICORP Advantages

HASHICORP

Nomad HASHICORP

Cluster Scheduler Easily Deploy Applications Operationally Simple Built for Scale Nomad HASHICORP

example.nomad job "redis" { datacenters = ["us-east-1"] task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 dynamic_ports = ["redis"] } } } }

Declares what to run HASHICORP Job Specification

Nomad determines where and manages how to run HASHICORP Job Specification

Nomad abstracts work from resources HASHICORP Job Specification

Docker Containerized Rkt Windows Server Containers Qemu / KVM Virtualized Xen Hyper-V Java Jar Standalone Static Binaries C# HASHICORP

Declarative Job Specification Infrastructure-As-Code Removes Imperative Logic External Dependencies? Nomad HASHICORP

Service Discovery? Health Monitoring? Application Secrets? Stateful Applications? Nomad HASHICORP

example.nomad job “my-app" { … task “my-app" { service { port = “http” check { type = “http” path = “/health” interval = “5s” } } } }

Client Nomad Server Nomad Consul Consul Server Register Service Schedule App Monitor Health App 1 App N HASHICORP

Secret Distribution: API Keys DB Credentials SSL/TLS Certificates Nomad HASHICORP

example.nomad job “my-app" { … task “my-app" { env { DB_USERPASS = “foo:bar” } } }

Secure secret storage Dynamic secrets Leasing, renewal, and revocation Auditing Rich ACLs Vault Multiple client authentication methods

Login  Vault Token Vault Token + Operation  Op Response HASHICORP

example.nomad job “my-app" { … task “my-app" { env { VAULT_TOKEN = “b6a10b96-9060-11e6-9c6f-67a52bc6b8d3” } } }

example.nomad job “my-app" { … task “my-app" { vault { policies = [“my-app-role”] } } }

 Submit Job + Vault Token Verify Vault Token Nomad Server Schedule App Client Generate + Renew Vault Token Nomad App 1 App N HASHICORP

Native Vault Integration No Secrets in Jobs No Secrets on Client Disk Minimize Trust Nomad HASHICORP

Stateless Stateful HASHICORP Stateful Applications

Stateless Stateful API Web Cache HASHICORP Stateful Applications

Stateless Stateful API HDFS Web Cassandra Cache MongoDB HASHICORP Stateful Applications

Stateless Stateful API HDFS *SQL Web Cassandra Cache MongoDB HASHICORP Stateful Applications

EASY MEDIUM HARD Stateless Stateful API HDFS *SQL Web Cassandra Cache MongoDB HASHICORP Stateful Applications

example.nomad job “my-app" { … task “my-app" { ephemeral_disk { sticky = true } } }

Moves data between tasks on the same machine HASHICORP

Copies data between tasks on di ff erent machines HASHICORP

Easily Deploy Apps: Declarative Jobs Flexible Workloads Consul Integration Vault Integration Sticky Volumes Nomad HASHICORP

Operationally Simple HASHICORP

Client Server HASHICORP

Built on Experience GOSSIP CONSENSUS

Cluster Management Gossip Based (P2P) Membership Failure Detection Serf Event System

Large Scale Production Hardened Simple Clustering and Federation Serf

Service Discovery Configuration Coordination (Locking) Central Servers + Distributed Clients Consul

Multi-Datacenter Raft Consensus Large Scale Production Hardened Consul

Operational Simplicity: Single Binary No Dependencies Highly Available Nomad HASHICORP

Built for Scale HASHICORP

Built on Experience Mature Libraries Proven Design Patterns GOSSIP CONSENSUS Lacking Scheduling Logic

Built on Research GOSSIP CONSENSUS

CLIENT CLIENT CLIENT DC1 DC2 DC3 RPC RPC RPC SERVER SERVER SERVER REPLICATION REPLICATION FORWARDING FORWARDING FOLLOWER LEADER FOLLOWER HASHICORP Single Region Architecture

 REGION A SERVER SERVER SERVER REPLICATION REPLICATION FORWARDING FOLLOWER LEADER FOLLOWER  GOSSIP REGION FORWARDING REGION B SERVER SERVER SERVER REPLICATION REPLICATION FORWARDING FOLLOWER LEADER FOLLOWER HASHICORP Multi Region Architecture

Region is Isolation Domain 1-N Datacenters Per Region Flexibility to do 1:1 (Consul) Scheduling Boundary Nomad HASHICORP

Hundreds of regions Tens of thousands of clients per region Thousands of jobs per region HASHICORP

Inspired by Google Omega Optimistic Concurrency State Coordination Service & Batch workloads Nomad Pluggable Architecture

Data Model NODE EVALUATION ALLOCATION JOB

Evaluation ~= State Change

Evaluations Create / Update / Delete Job Node Up / Node Down Allocation Failed / Finished

Evaluations SCHEDULER func(Evaluation) => []AllocationUpdates

Evaluations SCHEDULER func(Evaluation) => []AllocationUpdates Service, Batch, System

External Event EvaluaBon CreaBon EvaluaBon Queuing EvaluaBon Processing OpBmisBc CoordinaBon State Updates HASHICORP

Omega Architecture Optimistically Schedule 100’s of Jobs in Parallel Controls for Correctness Nomad HASHICORP

Nomad Million Container Challenge 1,000 Jobs 1,000 Tasks per Job 5,000 Hosts on GCE 1,000,000 Containers

“ 640 KB ought to be enough for anybody. – Bill Gates

2nd Largest Hedge Fund 18K Cores 5 Hours 2,200 Containers/second

Cluster Scheduler Easily Deploy Applications Operationally Simple Built for Scale Nomad HASHICORP

Thanks! Q/A HASHICORP

HASHICORP