preparing for the unexpected
play

Preparing for the Unexpected Samuel Parkinson - PowerPoint PPT Presentation

Dec 09, 2022 •137 likes •1.96k views

Preparing for the Unexpected Samuel Parkinson samuel.parkinson@ft.com #qconlondon #prepfortheunexpected Photo by Hush Naidoo on Unsplash #qconlondon #prepfortheunexpected #qconlondon #prepfortheunexpected Lets start with a story

  1. Preparing for the Unexpected Samuel Parkinson samuel.parkinson@ft.com #qconlondon #prepfortheunexpected Photo by Hush Naidoo on Unsplash #qconlondon #prepfortheunexpected

  2. #qconlondon #prepfortheunexpected

  3. Let’s start with a story #qconlondon #prepfortheunexpected

  4. What’s the worst thing that could happen? #qconlondon #prepfortheunexpected

  5. #qconlondon #prepfortheunexpected

  6. #qconlondon #prepfortheunexpected

  7. #qconlondon #prepfortheunexpected

  8. #qconlondon #prepfortheunexpected

  9. #qconlondon #prepfortheunexpected

  10. ************* #qconlondon #prepfortheunexpected

  11. #qconlondon #prepfortheunexpected

  12. The FT.com zone was missing #qconlondon #prepfortheunexpected

  13. #qconlondon #prepfortheunexpected

  14. FT.com has over 5,100 subdomains 😭 #qconlondon #prepfortheunexpected

  15. This impacted the whole company #qconlondon #prepfortheunexpected

  16. #qconlondon #prepfortheunexpected

  17. #qconlondon #prepfortheunexpected

  18. 😲 #qconlondon #prepfortheunexpected

  19. We have never prepared for such an incident #qconlondon #prepfortheunexpected

  20. It’s a classic data loss situation #qconlondon #prepfortheunexpected

  21. #qconlondon #prepfortheunexpected

  22. Our provider had a partial backup #qconlondon #prepfortheunexpected

  23. But critical records we used for DNS load balancing were missing 👼 #qconlondon #prepfortheunexpected

  24. About 10 people worked to resolve the incident #qconlondon #prepfortheunexpected

  25. And over 30 people were online to follow along #qconlondon #prepfortheunexpected

  26. Most were not called, but still volunteered their time #qconlondon #prepfortheunexpected

  27. #qconlondon #prepfortheunexpected

  28. 4h 30m The first hour was a total outage. #qconlondon #prepfortheunexpected

  29. Lack of panic in the moment #qconlondon #prepfortheunexpected

  30. It was a slick operation and we recovered #qconlondon #prepfortheunexpected

  31. It took restoring from a backup and manual entry to get there #qconlondon #prepfortheunexpected

  32. We were focused on recovery, not what happened #qconlondon #prepfortheunexpected

  33. People were joining the incident to learn #qconlondon #prepfortheunexpected

  34. #qconlondon #prepfortheunexpected

  35. #qconlondon #prepfortheunexpected

  36. This is where we are today #qconlondon #prepfortheunexpected

  37. #qconlondon #prepfortheunexpected

  38. #qconlondon #prepfortheunexpected

  39. #qconlondon #prepfortheunexpected

  40. Photo by Victor Garcia on Unsplash #qconlondon #prepfortheunexpected

  41. Photo by Markus Spiske on Unsplash #qconlondon #prepfortheunexpected

  42. 0. How do we do on-call? 1. Our incident management challenges 2. Making out-of-hours sustainable 3. The results and takeaways #qconlondon #prepfortheunexpected

  43. 0. How do we do on-call? 1. Our incident management challenges 2. Making out-of-hours sustainable 3. The results and takeaways #qconlondon #prepfortheunexpected

  44. Internal FT Core Products Enterprise Services Customer Products Operations & Reliability FT Group Products #qconlondon #prepfortheunexpected

  45. Customer We are Products #qconlondon #prepfortheunexpected

  46. 45 engineers and counting 📉 #qconlondon #prepfortheunexpected

  47. And we own about 180 systems #qconlondon #prepfortheunexpected

  48. #qconlondon #prepfortheunexpected

  49. Split into 9 teams #qconlondon #prepfortheunexpected

  50. #qconlondon #prepfortheunexpected

  51. #qconlondon #prepfortheunexpected

  52. Operations monitor our entire estate 24/7 #qconlondon #prepfortheunexpected

  53. #qconlondon #prepfortheunexpected

  54. Our systems are a drop in the pond #qconlondon #prepfortheunexpected

  55. You build it, you run it #qconlondon #prepfortheunexpected

  56. Supporting our systems out-of-hours #qconlondon #prepfortheunexpected

  57. This is our approach to DevOps #qconlondon #prepfortheunexpected

  58. Our engineers wear many hats Photo by Joshua Coleman on Unsplash #qconlondon #prepfortheunexpected

  59. We’re putting on our incident management hat #qconlondon #prepfortheunexpected

  60. How do we do support out-of-hours? #qconlondon #prepfortheunexpected

  61. Our engineers volunteer to be part of the out-of-hours team #qconlondon #prepfortheunexpected

  62. We don’t have shifts #qconlondon #prepfortheunexpected

  63. We don’t have shifts #qconlondon #prepfortheunexpected

  64. Which means, we could all be unavailable #qconlondon #prepfortheunexpected

  65. What do we care about? #qconlondon #prepfortheunexpected

  66. We’re talking about our business capabilities #qconlondon #prepfortheunexpected

  67. We’re talking about our business capabilities #qconlondon #prepfortheunexpected

  68. What is an incident at the FT? #qconlondon #prepfortheunexpected

  69. Customer Products has two really important business capabilities #qconlondon #prepfortheunexpected

  70. 1. Our users can always read the news #qconlondon #prepfortheunexpected

  71. 2. Journalists must be able to publish the news #qconlondon #prepfortheunexpected

  72. If either of these go wrong we declare an incident #qconlondon #prepfortheunexpected

  73. #qconlondon #prepfortheunexpected

  74. #qconlondon #prepfortheunexpected

  75. 0. How do we do on-call? 1. Our incident management challenges 2. Making out-of-hours sustainable 3. The results and takeaways #qconlondon #prepfortheunexpected

  76. What were our challenges? #qconlondon #prepfortheunexpected

  77. We were not immediately productive on call → #qconlondon #prepfortheunexpected

  78. We were not immediately productive on call We had an engineering mindset in an operations situation #qconlondon #prepfortheunexpected

  79. We were not immediately productive on call Because we don’t have any SRE or DevOps specialists #qconlondon #prepfortheunexpected

  80. “ ” I always start with the impact and the comms, they kinda jump in at the Tech. #qconlondon #prepfortheunexpected

  81. We were not immediately productive on call Our incident management process wasn’t second nature #qconlondon #prepfortheunexpected

  82. We had very few incidents in the first half of the year #qconlondon #prepfortheunexpected

  83. We had very few incidents in the first half of the year #qconlondon #prepfortheunexpected

  84. And we were down to 5 people on the out-of-hours support team #qconlondon #prepfortheunexpected

  85. So we needed to make out-of-hours team sustainable #qconlondon #prepfortheunexpected

  86. 0. How do we do on-call? 1. Our incident management challenges 2. Making out-of-hours sustainable 3. The results and takeaways #qconlondon #prepfortheunexpected

  87. We surveyed engineers about helping out during an incident #qconlondon #prepfortheunexpected

  88. There were many people on the fence #qconlondon #prepfortheunexpected

  89. There were many people 7 people on the fence 3 people 6 people #qconlondon #prepfortheunexpected

  90. And they told us why #qconlondon #prepfortheunexpected

  91. “ ” I will need much more confidence in systems and domains knowledge. #qconlondon #prepfortheunexpected

  92. “ ” If I were to have a better understanding of how it works and what I would need to do, I would very likely join. #qconlondon #prepfortheunexpected

  93. We set out to convince people to join our out-of-hours team #qconlondon #prepfortheunexpected

  94. We built and ran incident workshops #qconlondon #prepfortheunexpected

  95. So our engineers are better prepared to take on incidents #qconlondon #prepfortheunexpected

  96. And we wrote a generic runbook for our microservices #qconlondon #prepfortheunexpected

  97. So engineers knew what they can do, and apply it to our ~180 systems #qconlondon #prepfortheunexpected

  98. We set out in the last 6 months of 2019 to address the situation #qconlondon #prepfortheunexpected

  99. Building your incident workshop → #qconlondon #prepfortheunexpected

  100. Building your incident workshop Don’t Panic! #qconlondon #prepfortheunexpected

Recommend

Dialogue Managing the Unexpected Managing the Unexpected the Role of the Regulatory Body

Dialogue Managing the Unexpected Managing the Unexpected the Role of the Regulatory Body

Swiss Federal Nuclear Safety Inspectorate ENSI Dialogue Managing the Unexpected Managing the Unexpected the Role of the Regulatory Body Cornelia Ryser, ENSI, Switzerland IAEA Technical Meeting Vienna, 25 29 June 2012 Assumptions

543 views • 4 slides
Pitfalls in PID: Expect the unexpected Faculty Disclosure X No, nothing to disclose Yes, please

Pitfalls in PID: Expect the unexpected Faculty Disclosure X No, nothing to disclose Yes, please

Pitfalls in PID: Expect the unexpected Faculty Disclosure X No, nothing to disclose Yes, please specify: PITFALLS IN PID: EXPECT THE UNEXPECTED First daughter of non-consanguineous parents, no relevant family history. 2 months old

258 views • 10 slides
Managing the Unexpected: Interactions between, Human Excellence and Organization Safety Culture

Managing the Unexpected: Interactions between, Human Excellence and Organization Safety Culture

Managing the Unexpected: Interactions between, Human Excellence and Organization Safety Culture Dr. Abdul Nasir Afghan (IBA) Mr. Shahbaz Ali Nasir Bhatti (PNRA) IAEA TM on Managing the Unexpected From the Perspective of the interaction

250 views • 21 slides
Testing for the Unexpected: An Automated T ti f th U t d A A t t d Method of Injecting

Testing for the Unexpected: An Automated T ti f th U t d A A t t d Method of Injecting

Testing for the Unexpected: An Automated T ti f th U t d A A t t d Method of Injecting Faults for Engine Management Development M t D l t Scott James Applied Dynamics International Sh Shaun Fuller Pickering Interfaces F ll

353 views • 18 slides
Scaling for the Expected and Unexpected Brian Moon Senior Web Engineer dealnews.com

Scaling for the Expected and Unexpected Brian Moon Senior Web Engineer dealnews.com

Scaling for the Expected and Unexpected Brian Moon Senior Web Engineer dealnews.com http://brian.moonspot.net/ twitter.com/brianlmoon Tuesday, June 23, 2009 Tuesday, June 23, 2009 Unexpected Likely one page.... at first Likely coming

420 views • 13 slides
Computing the unexpected and Unpredicted and Deceptive and Interesting Reflections

Computing the unexpected and Unpredicted and Deceptive and Interesting Reflections

Computing the unexpected and Unpredicted and Deceptive and Interesting Reflections Fifty year prediction about home computers Computing the unexpected and Unpredicted and Deceptive and Interesting Reflections

501 views • 10 slides
Preparing for Cascadia 9.0 Preparing for Cascadia 9.0 Individual, Household, and Community

Preparing for Cascadia 9.0 Preparing for Cascadia 9.0 Individual, Household, and Community

Preparing for Cascadia 9.0 Preparing for Cascadia 9.0 Individual, Household, and Community Preparedness Bill Hasan Hillsdale Neighborhood Emergency Team members hillsdaleornet@gmail.com March 10, 2019 | Portland Bureau of Emergency

352 views • 31 slides
How to Make a Formal Presentation Contents Preparing Content ( Written ) Theory

How to Make a Formal Presentation Contents Preparing Content ( Written ) Theory

How to Make a Formal Presentation Contents Preparing Content ( Written ) Theory Practice Preparing Content ( Visual ) Preparing Yourself Body Language Preparing for your Presentation Video Preparing Written

233 views • 8 slides
Flash Reliability in Produc4on: The Expected and the Unexpected

Flash Reliability in Produc4on: The Expected and the Unexpected

Flash Reliability in Produc4on: The Expected and the Unexpected Bianca Schroeder Raghav Lagisetty, University of Toronto Arif Merchant Google Inc. Why flash? More and more

210 views • 20 slides
Preparing IRB Submissions for Human Subjects Research Tips for Preparing IRB Protocols IRB

Preparing IRB Submissions for Human Subjects Research Tips for Preparing IRB Protocols IRB

Preparing IRB Submissions for Human Subjects Research Tips for Preparing IRB Protocols IRB Educational Session-Psychological Sciences September 15, 2017 Post Approval Monitor-Joan Levine, MPH, CIP Research Compliance Services Objectives You

386 views • 25 slides
Preparing Preparing for a for a Succe Successful ssful HAZOP/LO HAZOP/LOPA PA (Making or

Preparing Preparing for a for a Succe Successful ssful HAZOP/LO HAZOP/LOPA PA (Making or

Preparing Preparing for a for a Succe Successful ssful HAZOP/LO HAZOP/LOPA PA (Making or Breaking Quality & Efficiency) (GCPS-2018 Paper 513595) Steven T. Maher, PE CSP / Carine M. Black / John M. Johnson Risk Management

397 views • 29 slides
Creating Chaos: Engineering for the Unexpected Presented by:

Creating Chaos: Engineering for the Unexpected Presented by:

DW6 Microservices & Cloud Wednesday, November 7th, 2018 1:30 PM Creating Chaos: Engineering for the Unexpected Presented by:

509 views • 14 slides
Preparing For The Rising Tide Designing with Water Preparing For The Rising Tide Published

Preparing For The Rising Tide Designing with Water Preparing For The Rising Tide Published

Preparing For The Rising Tide Designing with Water Preparing For The Rising Tide Published February 2013 Maps Bostons vulnerability to coastal flooding Encourages property owners in today and tomorrows flood zone to know

283 views • 26 slides
Investigation of Sudden Investigation of Sudden and Unexpected Infant and Unexpected Infant

Investigation of Sudden Investigation of Sudden and Unexpected Infant and Unexpected Infant

Investigation of Sudden Investigation of Sudden and Unexpected Infant and Unexpected Infant Deaths Deaths Melanie Newbould Melanie Newbould Paediatric and Perinatal Paediatric and Perinatal Pathologist Pathologist RMCH RMCH Every year

963 views • 62 slides
Preparing for New School Year This session u What is the Preparing for the new school year

Preparing for New School Year This session u What is the Preparing for the new school year

5/12/16 Preparing for New School Year This session u What is the Preparing for the new school year process u Key term review u How to use the Prepare for Next School year administrative tool u Options for moving students 1 5/12/16 What

406 views • 11 slides
10 tips for preparing a presentation

10 tips for preparing a presentation

10 tips for preparing a presentation 1. Purpose What is the reason for the presentation you are preparing ? This sounds obvious but stay focused on this as you build the

298 views • 3 slides
Protecting Against Unexpected System Calls C. M. Linn, M. Rajagopalan, S. Baker, C.

Protecting Against Unexpected System Calls C. M. Linn, M. Rajagopalan, S. Baker, C.

Protecting Against Unexpected System Calls C. M. Linn, M. Rajagopalan, S. Baker, C. Collberg, S. K. Debray, J. H. Hartman Department of Computer Science University of Arizona Presented By: Mohamed Hassan Code Injection What

227 views • 12 slides
Things I Wish Id Known Rod Johnson My Journey An Unexpected Career Memorable Highs

Things I Wish Id Known Rod Johnson My Journey An Unexpected Career Memorable Highs

Things I Wish Id Known Rod Johnson My Journey An Unexpected Career Memorable Highs SpringOne 2010 Spring community: 1000 attendees Realization that my cofounders had also become good friends Changing the world

938 views • 47 slides
The Risk and Protective Factors associated with Sudden Unexpected Death in Infancy

The Risk and Protective Factors associated with Sudden Unexpected Death in Infancy

The Risk and Protective Factors associated with Sudden Unexpected Death in Infancy Blackpool, Blackburn with Darwen and Lancashire have a higher infant mortality rate than England Not easily identifiable they are often medically

635 views • 14 slides
Swiss Cadastre Preparing for Swiss Cadastre Preparing for E- -Government Government E Dr.

Swiss Cadastre Preparing for Swiss Cadastre Preparing for E- -Government Government E Dr.

Bundesamt fr Landestopographie Office fdral de topographie Ufficio federale di topografia Uffizi federal da topografia Swiss Cadastre Preparing for Swiss Cadastre Preparing for E- -Government Government E Dr. Daniel Steudler Swiss

411 views • 39 slides
Swiss Cadastre Preparing for Swiss Cadastre Preparing for E- -Government Government E Dr.

Swiss Cadastre Preparing for Swiss Cadastre Preparing for E- -Government Government E Dr.

Bundesamt fr Landestopographie Office fdral de topographie Ufficio federale di topografia Uffizi federal da topografia Swiss Cadastre Preparing for Swiss Cadastre Preparing for E- -Government Government E Dr. Daniel Steudler Swiss

641 views • 36 slides
Ethical Challenges in Preparing Ethical Challenges in Preparing Witnesses and Working with Experts

Ethical Challenges in Preparing Ethical Challenges in Preparing Witnesses and Working with Experts

Presenting a live 90 minute webinar with interactive Q&A Ethical Challenges in Preparing Ethical Challenges in Preparing Witnesses and Working with Experts Best Practices to Avoid Unethical Coaching of Lay and Expert Witnesses WEDNES DAY,

666 views • 62 slides
Preparing for Open Enrollment August 15, 2017 Preparing for Open Enrollment Agenda Special

Preparing for Open Enrollment August 15, 2017 Preparing for Open Enrollment Agenda Special

Preparing for Open Enrollment August 15, 2017 Preparing for Open Enrollment Agenda Special Guest: Julie Hammon from Department of Health and Welfare YHI Certification Requirements & Deadlines Live Training, Attendance

163 views • 13 slides
Case #1: Unexpected Pre-op Labs 62 yo man scheduled for bilateral Total Hip Arthroplasty has the

Case #1: Unexpected Pre-op Labs 62 yo man scheduled for bilateral Total Hip Arthroplasty has the

10/25/16 Management of the Hospitalized Patient Case Discussions in Hematology: Musings from the UCSF Hematology (e)Consult Service Andrew D. Leavi4, MD October 21, 2016 Case #1: Unexpected Pre-op Labs 62 yo man scheduled for bilateral Total

382 views • 22 slides

More recommend