Containers @ Wrapp Powered by AWS EC2 Container Service Jude DSouza - PowerPoint PPT Presentation

Containers @ Wrapp Powered by AWS EC2 Container Service Jude D’Souza Solutions Architect @ Wrapp Phone: +46 767085740 Email: jude@wrapp.com

About Me … q Jude D’Souza – Stockholm, Sweden ß Karachi, Pakistan jude@wrapp.com q MS – Distributed Systems KTH Royal Institute of Technology, Stockholm, Sweden q Solutions Architect @ Wrapp www.wrapp.com q Passion for building large scale distributed systems

About Wrapp … Wrapp delivers personal offers and rewards based on where you shop, dine and the brands you follow. q Microservice Architecture q Infrastructure q 13 ECS Clusters q 50+ Instances q 90+ Services q Technology Stack q Operating System - Ubuntu q Consul, Registrator, AWS(EC2, ECS, Lambda), Docker, HAProxy q Monitoring – Sensu, Riemann q Logs - Logentries and Sumo q Languages - Go, Python, Java

Agenda Architecture before ECS • – Service Orchestration – Service Discovery – Service Deployments – Service Availability & Scalability – Container Registry Transition towards ECS • – Microservice Architecture @ Wrapp Why ECS? • Next Challenges • QA •

ü Service Orchestration Service Deployments Service Availability & Scalability How we did it … Before ECS (1/3) Container Registry Service Orchestration Via runlist configs q On instance boot: Determine autoscaling group q Pull runlist for autoscaling group q Run containers specified in config q Add these to supervisor q PULL RUNLIST CONFIGS Sample Runlist configuration file misc ops api Autoscaling group Autoscaling group Autoscaling group misc ops api

ü Service Orchestration ü Service Deployments Service Availability & Scalability How we did it … Before ECS (2/3) Container Registry Service Deployments Find instances where service is running (via serf) q SSH into these instances q Do rolling deploy (Terminate container, Launch container q with new version) ops misc api Autoscaling group Autoscaling group Autoscaling group misc ops api Sample serf data for services - misc-i-abc 10.0.0.1 rewards/http/9090 - misc-i-def 10.0.1.1 rewards/http/9090 - misc-i-ghi 10.0.2.1 rewards/http/9090 - .... SERF CLUSTER - ....

ü Service Orchestration ü Service Deployments ü Service Availability & Scalability How we did it … Before ECS (3/3) ü Container Registry Service Availability ops misc api q Supervisord Autoscaling group Autoscaling group Autoscaling group Service Scalability q Service itself can't scale misc ops api q Scale only by scaling autoscaling group misc SERF CLUSTER Container Registry Hosting our own container registry q Backed by s3 q

Transition to ECS Service Orchestration q Runlists q Supervisor Service Orchestration Service Deployments Service Deployments Service Availability q Supervisor Service Scalability (with lambda) Service Availability AWS Container Registry q Supervisor Service Scalability + Service stats (cpu + memory) q EC2 Autoscaling + Service monitoring Container Registry + Service Provisioning q Self-hosting Service Discovery q Consul Service Discovery q Registrator q Serf q HAProxy q HAProxy

Wrapp Microservice Architecture DNS api.wrapp.com Elastic Load Balancer PUBLIC SUBNET PRIVATE SUBNET api api api Microservices ZONE A ZONE B ZONE C

Opsweb – Tooling around controlling our Infrastructure Cluster Overview

Opsweb – Tooling around controlling our Infrastructure Services Overview

Opsweb – Tooling around controlling our Infrastructure Instances Overview

Service Scalability (Concept from Amazon) Trigger Lambda function from SNS Topic CloudWatch Alarm SNS Topic AWS Lambda +1 DesiredCount in Service Trigger CloudWatch alarm misc from ECS Metrics (CPU/ Autoscaling group Memory) EC2 Container Service misc ECS Instances Scaling Amazon ECS Services Automatically Using Amazon CloudWatch and AWS Lambda: https://aws.amazon.com/blogs/compute/scaling-amazon-ecs-services-automatically-using-amazon-cloudwatch-and-aws-lambda/

Service (CPU/ Memory) Provisioning Memory allocation by Service ü Allocate CPU Units by Service ü Efficient Resource utilization (Container placement) ü eventbus notifications api rewards auth emails Typical ECS Instance

Container Logging Containers launched via ECS Logspout q Rsyslog q logspout Log router for Docker LogEntries q rsyslog Forwards logs to logentries Typical ECS Instance Logentries: https://logentries.com Logspout: https://github.com/gliderlabs/logspout

Why ECS? Our Experience q ü Supports built-in service discovery, loadbalancing and routing Generally good service (ECS is oK) q Great support q ü Fleet - Container Orchestration We don’t need to host these services q ourselves on AWS Always improving to make their services q ü Docker Swarm: Supports orchestrating containers better and better ü Weave Flux: Supports built-in service discovery, loadbalancing and routing

Service Discovery @ Wrapp (1/2) Key/value store HA Consul Cluster Service catalog Consul server-a Consul Consul server-b server-c Consul Agent Consul Agent Consul Agent Registrator Registrator Registrator users users api auth 8082 8082 8080 8081 10.0.0.1 10.0.0.2 10.0.0.3

Service Discovery @ Wrapp (2/2) HA Consul Cluster >> curl users/v1/users/1 { "name": "jude" Consul server-a "gender": "male" } Consul Consul server-b server-c frontend users-in DNS Hack Consul bind users:80 HAProxy (192.168.x.x) Template mode http default_backend users backend users Consul Agent server Server-a 10.0.0.1:8080 … server Server-c 10.0.0.3:8080 … #### Registrator frontend auth-in bind auth:80 mode http default_backend auth auth users backend auth 8082 8083 server Server-a 10.0.0.1:8081 … 10.0.0.3 server Server-c 10.0.0.3:8081 …

Next Challenges (1/2) – Short comings perceived … Built-in (like kube-proxy, weave-proxy) ? SINGLE ELB FOR MULTIPLE SERVICES APPROACH Service Discovery q Service Routing q Elastic Load Balancer Service Load balancing q 8080 8081 8082 Approach 1: ELB Per service? Costly - 90+ services, 70+ ELBs q Approach 2: Single ELB for all services? ECS Boto3: create_service() Service Routing works ü Service Load balancing works ü No health check per service q No way to return HTTP 503 q (but possible via haproxy http check option) Route53: ELB.dns = elb-services frontend users-in Benefits: bind users:80 Removes tools like Consul and Registrator q mode http default_backend users backend users server users elb-services:8080 … #### >> curl users/ping

Next Challenges (2/2) – Short comings perceived … Cron scheduling Ø q Setup ecs tasks to run on cron schedule ECS Event stream Ø q Actionables on certain events (when deploy completes) ECS Container Logging Ø Optionally Integrate ECS Cluster with EC2 Autoscaling Group Ø

Summary q The Wrapp Architecture before ECS q Transition to ECS q Service Discovery @ Wrapp with Consul, Registrator and HAProxy q Our perceived shortcomings from ECS q Built-in Service Discovery q Cron scheduling on Tasks q Event stream

Questions ? Contact: Jude D’Souza Email: jude@wrapp.com Phone: +46767085740 Twitter: Wrapp Tech @wrapp_tech