remediation patterns Jez Humble QCon London 2011 - PowerPoint PPT Presentation

remediation patterns Jez Humble QCon London 2011 jez@thoughtworks.com @jezhumble #continuousdelivery Thursday, March 10, 2011 1

remediation ITIL: “Recovery to a known state after a failed Change or Release.” Recovery: “Returning a Configuration Item or an IT Service to a working state.” Jez: “Fixing shit when it breaks” Thursday, March 10, 2011 2

strategies for remediation prevention patterns for low-risk release patterns for incremental delivery Thursday, March 10, 2011 3

1oz of prevention Thursday, March 10, 2011 4

deployment pipeline Thursday, March 10, 2011 5

the hard bits testing on production environments creating maintainable acceptance tests testing cross-functional requirements Thursday, March 10, 2011 6

reducing release risk automate provisioning and deployment ensure devs, testers and ops collaborate throughout Thursday, March 10, 2011 7

canary releasing Thursday, March 10, 2011 8

canary releasing Thursday, March 10, 2011 9

canary releasing reduce risk of release multivariant testing performance testing Thursday, March 10, 2011 10

immune system what if someone replaced your “buy” button with spacer.gif? T cells http://www.flickr.com/photos/gehealthcare/3326186490/ Thursday, March 10, 2011 11

monitoring Business metrics - revenue, # orders, # users Ops metrics - changes, incidents, TTD, TTR, TBF Technical metrics - TPS, response time, hits http://www.flickr.com/photos/wwarby/3296379139/ Thursday, March 10, 2011 12

the hard bits root cause analysis collboration data Thursday, March 10, 2011 13

incremental delivery John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr Thursday, March 10, 2011 14

incremental delivery develop on mainline feature toggles and branch by abstraction incremental deployments dark launching Thursday, March 10, 2011 15

feature toggles Config File [featureToggles] some.jsp wobblyFoobars: true flightyForkHandles: false <toggle name=wobblyFoobars> ... various UI elements </toggle> other.java forkHandle = (featureConfig.isOn(‘flightlyForkHandles)) ? new FlightyForkHander(aCandle) : new ForkHandler(aCandle) Stolen from Martin Fowler <link to his bliki entry> Thursday, March 10, 2011 16

branch by abstraction Component A Component A Seam Component B Thursday, March 10, 2011 17

branch by abstraction Component A Component A Abstraction layer Component B Component B’ Thursday, March 10, 2011 18

incremental deployment DEPENDENT SERVICE 1.0 1.1 Database Abstraction layer Abstraction layer STATIC CONTENT /static/1.1 APPLICATION /static/1.0 Router / Load balancer Interwebs Thursday, March 10, 2011 19

dark launching Thursday, March 10, 2011 20

dark launching Thursday, March 10, 2011 21

measuring effectiveness How long would it take you to release a change to a single line of code? Ops metrics - changes, incidents, TTD, TTR, TBF If your data center blew up, how long would you take to restore service? Thursday, March 10, 2011 22

questions Jez Humble jez@thoughtworks.com @jezhumble #continuousdelivery Thursday, March 10, 2011 23