HOW DEVOPS HELPS SHAPE SMALL TEAMS AND ARCHITECTURE MATHIAS MEYER, @ROIDRAGE
DEVOPS
PRODUCTION AND CULTURE
PRODUCTION = ARCHITECTURE
!
PRODUCTION = ARCHITECTURE + OPERATIONS(PRODUCTION)
P = A + O(P)
ARCHITECTURE & OPERATIONS
MONOLITH
BIG BALL OF MUD
TWO PARTS
WEB FRONTEND
HUB
(WORKER)
HUB'S TODO LIST ▸ Create Builds ▸ Talk to GitHub API ▸ Schedule Builds ▸ Process Build Results ▸ Process Log Updates ▸ Send Build Notifications
HUB BROKE ! A LOT !
SCALE
THE PROFESSIONALIZATION OF TRAVIS CI
METRICS
By a de ing just a few metrics, you get a be tu er idea what you n ef d to track.
MO METRICS, MO PROBLEMS
THE BIG TRAVIS CI OUTAGE OF 2012
PULL REQUESTS
PROBLEMS ▸ Minor changes in the GitHub API ▸ Liberal timeouts ▸ Serial processing ▸ Bug in JRuby
Because failure requires multiple faults, there is no isolated ‘cause’ of an accident. — Richard Cook, How Complex Systems Fail
Each of these faults is nece st ary insu fg icient, only jointly are these causes su fg icient to create an accident. — Richard Cook, How Complex Systems Fail
WHAT DID WE LEARN?
TREAT EXTERNAL APIS LIKE FOES RATHER THAN FRIENDS.
Fail gracefu lm y.
Fail fast.
HOW YOU HANDLE FAILURES IS A BUSINESS DECISION.
Simplify, simplify, simplify. — Henry David Thoreau
Simplify. — Paraphrased
SINGLE RESPONSIBILITIES SEPARATE PROCESSES
INCREASED COMPLEXITY AND INCREASED VELOCITY
A complex system is shaped more by its behaviour in production than its initial design.
COMPLEX ≠ COMPLICATED
THE TRUTH LIES ONLY IN PRODUCTION
ALERTING
BYOC BUILD YOUR OWN COLLECTOR
Every organization ru no ing a software system in production is bound to implement their own monitoring system around it. — Mathias' 1st Law of Monitoring
Every custom monitoring solution is bound to implement cycle checks. — Mathias' 2nd Law of Monitoring
SMALL SERVICES LOOSELY JOINED
SIMPLIFICATION IS THE HALLMARK OF SCALABILITY
MORE COMPONENTS = MORE RESPONSIBILITIES
WHO YOU GONNA CALL?
THE GOAL OF ALERTING ISN'T TO WAKE PEOPLE UP, IT'S TO MAKE SURE THEY CAN SLEEP
Any new alert wi lm wake people up more often than nece st ary initia lm y. — Mathias' 3rd Law of Monitoring:
GIVE ROOM TO IMPROVE
Any organization that designs a system wi lm produce a design whose structure is a copy of the organization's co mn unication structure. — Conways' Law
BIG TEAMS = BIG SYSTEMS?
DISTRIBUTED TEAMS = DISTRIBUTED SYSTEMS?
ASYNCHRONOUS COMMUNICATION = ASYNCHRONOUS SYSTEMS?
FLEXIBILITY + ADAPTABILITY
CONTINUOUS IMPROVEMENT
EVERYONE'S ON CALL
Any alert that tri gh ers more than once a w ef k wi lm become part of normal procedure. — Mathias' 4th Law of Monitoring
MONITORING IS A PROCESS NOT A FEATURE
OUR ALERTING IS FAR FROM IDEAL
LEARN FROM UNEXPECTED EVENTS
ANDON CORD
BLAMELESS POSTMORTEMS
DEVOPS?
PRODUCTION IS A SHARED RESPONSIBILITY
You build it, you run it. — Werner Vogels
FOCUS ON CONTINUOUS LEARNING
DEVOPS DIDN'T SHAPE OUR TEAM
OUR TEAM EVOLVED INTO SOMETHING THAT LOOKS LIKE DEVOPS
mathias@travis-ci.com
Recommend
More recommend
