Deploy Early, Deploy Often, Deploy Safely Andy Lowe
From User Story to Production Feature Basic Process… Commit Automated UAT Deploy Write Some Acceptance Prod Tests Code Deploy Then There’s All The Other Stuff… Staging Code Security Change OSS Performance Integration Integration Deploy Review Scans Management License Testing Deploy Testing Review @LoweKeyOne #AgileAZ
Why Automate Deployments? • Multiple steps that happen exactly the same way each time • May need to deploy to multiple systems • Automated steps far easier to test Manual deploys will go badly, your team will work nights and weekends, your users will be upset, and your business will be negatively impacted. @LoweKeyOne #AgileAZ
Would you rather fix an incident… • By manually deploying • Deploying using the same process you always do • Changing configs by hand on multiple servers • With automated rollback Or… • Documenting later what • Using a fully tested and changed understood process • Without security controls • With your usual controls Deploying fixes can look just like deploying any other change. Always have a widely understood exception process for any controls that may be in place. @LoweKeyOne #AgileAZ
How Often Do You Deploy? Max Deploy Duration (minutes) Allowed Yearly SLA Downtime Once per Month Once per Week Once per Day (minutes) 99% 5256 438 105.12 30.918 99.9% 525.6 43.8 10.512 3.092 99.99% 52.56 4.38 1.051 0.309 99.999% 5.256 0.438 0.105 0.031 99.9999% 0.526 0.044 0.011 0.003 Do you want to not hit your SLA because of maintenance windows? @LoweKeyOne #AgileAZ
Blue/Green Deploys • Zero downtime deployments of a new version of an app. • Rollback strategy is simple. • Requires the hardware resources to have two versions running simultaneously. @LoweKeyOne #AgileAZ
Blue/Green Deploys MyBlueApp.Example.com MyBlueApp MyApp.Example.com Router Your app is running on on your servers. @LoweKeyOne #AgileAZ
Blue/Green Deploys MyBlueApp.Example.com MyBlueApp MyApp.Example.com Router MyGreenApp.Example.com MyGreenApp Deploy a new version. @LoweKeyOne #AgileAZ
Blue/Green Deploys MyBlueApp.Example.com MyBlueApp Router MyApp.Example.com MyGreenApp.Example.com MyGreenApp Update main route to point both apps. @LoweKeyOne #AgileAZ
Blue/Green Deploys Router MyApp.Example.com MyGreenApp.Example.com MyGreenApp Remove main route from blue app. @LoweKeyOne #AgileAZ
Blue/Green Deploys Router MyApp.Example.com MyGreenApp.Example.com MyGreenApp Delete old version and route. @LoweKeyOne #AgileAZ
Blue/Green Deploys $ cf push MyBlueApp –n MyApp $ cf push MyGreenApp –n MyGreenApp $ cf map-route MyGreenApp example.com –n MyApp $ cf unmap-route MyBlueApp example.com –n MyApp $ cf delete MyBlueApp @LoweKeyOne #AgileAZ
Multi-Availability Zone Deploys – Baseline MyBlueApp MyBlueApp Router Router Load Balancer MyBlueApp MyBlueApp Router Router @LoweKeyOne #AgileAZ
Multi-Availability Zone Deploys – Push Green MyBlueApp MyBlueApp Router Router MyGreenApp MyGreenApp Load Balancer MyBlueApp MyBlueApp Router Router MyGreenApp MyGreenApp @LoweKeyOne #AgileAZ
Multi-Availability Zone Deploys – Map New Routes MyBlueApp MyBlueApp Router Router MyGreenApp MyGreenApp Load Balancer MyBlueApp MyBlueApp Router Router MyGreenApp MyGreenApp @LoweKeyOne #AgileAZ
Multi-Availability Zone Deploys – Unmap Routes MyBlueApp MyBlueApp Router Router MyGreenApp MyGreenApp Load Balancer MyBlueApp MyBlueApp Router Router MyGreenApp MyGreenApp @LoweKeyOne #AgileAZ
Multi-Availability Zone Deploys – Delete Blue Router Router MyGreenApp MyGreenApp Load Balancer Router Router MyGreenApp MyGreenApp @LoweKeyOne #AgileAZ
Health Check That Aliveness checks Functional Checks • Check if app is responding • Check if app is functional • Should be very fast • Should be fast • Run on startup and every 30 • Check for connections to seconds external systems, environment • TCP, Process, and http checks validity, basic functionality • Run on deploy and as part of monitoring solution @LoweKeyOne #AgileAZ
Health Checks During Deploy MyBlueApp MyBlueApp Router Router HealthChecker MyGreenApp MyGreenApp Load Balancer MyBlueApp MyBlueApp Router Router MyGreenApp MyGreenApp @LoweKeyOne #AgileAZ
Health Check Each Green App MyBlueApp MyBlueApp Router Router HealthChecker MyGreenApp MyGreenApp Load Balancer MyBlueApp MyBlueApp Router Router MyGreenApp MyGreenApp @LoweKeyOne #AgileAZ
Health Check Failed - Rollback MyBlueApp MyBlueApp Router Router HealthChecker Load Balancer MyBlueApp MyBlueApp Router Router @LoweKeyOne #AgileAZ
Health Check Succeeded - Continue Router Router HealthChecker MyGreenApp MyGreenApp Load Balancer Router Router MyGreenApp MyGreenApp @LoweKeyOne #AgileAZ
Units of Deployment MyBlueApp Router SomeOtherSystem MyGreenApp SomeOtherSystem has to support MyBlueApp and MyGreenApp simultaneously. Options: 1. Have immutable API. 2. Have versioned API with support for two versions. 3. Combine deployment of new version of SomeOtherSystem with MyGreenApp. @LoweKeyOne #AgileAZ
Devs Are More Focused Now… Basic Process… Commit Automated UAT Deploy Write Some Acceptance Prod Tests Code Deploy All The Other Stuff… Staging Code Security Change Third Party Performance Integration Integration Deploy Review Scans Management License Testing Deploy Testing Review @LoweKeyOne #AgileAZ
Security Scans • Static scans should happen for each artifact deployed to Security prod Scans • May happen as part of deploying to lower environments • Make asynchronous, due to time required • Active scans, taking care with environment in which they run • Set criteria for deploy to succeed Security Scans • Start with scan results must exist for prod deploy • Add criteria around severity of issues over time • Always have an exception process in place @LoweKeyOne #AgileAZ
OSS Scans Scan for OSS license compliance Third Party License • Are you using GPL or LGPL? Review • What about BSD, MIT, WTFPL, APL, others? Know which libraries are in use • Are you using outdated versions with security issues? Third Party License • When a zero-day comes out, know which apps to patch Review @LoweKeyOne #AgileAZ
Change Management Automated deployments mean automatically you know: Change Management • Who initiated the deployment • What artifacts and what configuration • When the deploy happens • Health checks on deploy give you deployment validation. Change Management @LoweKeyOne #AgileAZ
Even More Focus Basic Process… Commit Automated Dev Deploy Write Some Acceptance Prod Tests Code Deploy All The Other Stuff… UAT Deploy Code Security Change Third Party Performance Integration Integration Review Scans Management License Testing Deploy Testing Review @LoweKeyOne #AgileAZ
Why Does All This Matter? • This is real pain felt across the industry. • Freezes are a legitimate business decision. • All those freezes leave about 170 days on which you can deploy. Make the most of them. @LoweKeyOne #AgileAZ
Case Study: Allstate Need: Centralized, opinionated deployment tool for deployments • Deployadactyl – Open source tool for multi-available zone blue/green deploys • Conveyor – Allstate-specific deployment logic • Metrics • Change management • Security scans • Compliance checks @LoweKeyOne #AgileAZ
Case Study: Allstate Artifact CI/CD Repository Dev Dev UAT UAT Source Control Int Int Staging Staging Prod Prod Pipelines include a full automated test suite run prior to publishing to • artifact repository and deployment. Dev pipelines run on commit to master in source control. • Prod pipelines are manually initiated. •
Case Study: Allstate Easy verification that 32% of prod deploys had One month: 87 product teams security scans. 4000+ deployments Security scans on deploy success. 1000+ production deployments Easy verification that >90% of prod deploys had security scans. Teams don’t wait for the weekend to deploy. @LoweKeyOne #AgileAZ
Deploy Early, Deploy Often, Deploy Safely Andy Lowe
Recommend
More recommend
