failure sketching a technique for automated root cause
play

Failure Sketching: A Technique for Automated Root Cause Diagnosis - PowerPoint PPT Presentation

Mar 28, 2023 •7 likes •937 views

Failure Sketching: A Technique for Automated Root Cause Diagnosis of In-Production Failures Baris Kasikci, Benjamin Schubert, Cristiano Pereira, Gilles Pokam, George Candea Debugging In-Production Software Failures Today 2 Debugging

  1. Failure Sketching: A Technique for Automated Root Cause Diagnosis of In-Production Failures Baris Kasikci, Benjamin Schubert, Cristiano Pereira, Gilles Pokam, George Candea

  2. Debugging In-Production Software Failures Today 2

  3. Debugging In-Production Software Failures Today 2

  4. Debugging In-Production Software Failures Today #0 0x00007f51abae820b in raise (sig=11) at ../nptl/ sysdeps/unix/sysv/linux/pt-raise.c:37 #1 0x000000000042d289 in ap_buffered_log_writer (r=0x7f51a40053d0, handle=0x20eeba0, strs=0x7f51a4003578, strl=0x7f51a40035e8, nelts=14, len=82) at mod_log_config.c:1368 #2 0x000000000042b10d in config_log_transaction (r=0x7f51a40053d0, cls=0x20b9d50, default_format=0x20ee370) at mod_log_config.c:930 #3 0x000000000042aad6 in multi_log_transaction (r=0x7f51a40053d0) at mod_log_config.c:950 #4 0x000000000046cb2d in ap_run_log_transaction (r=0x7f51a40053d0) at protocol.c:1563 #5 0x0000000000436e81 in ap_process_request (r=0x7f51a40053d0) at http_request.c:312 #6 0x000000000042e9da in ap_process_http_connection (c=0x7f519c000b68) at http_core.c:293 #7 0x0000000000465cdd in ap_run_process_connection (c=0x7f519c000b68) at connection.c:85 #8 0x00000000004661f5 in ap_process_connection (c=0x7f519c000b68, csd=0x7f519c000a20) at connection.c:211 #9 0x0000000000451ba0 in process_socket (p=0x7f519c0009b8, sock=0x7f519c000a20, my_child_num=0, my_thread_num=0, bucket_alloc=0x7f51a4001348) at worker.c:632 #10 0x0000000000451221 in worker_thread (thd=0x210fa90, dummy=0x7f51a40008c0) at worker.c:946 #11 0x00007f51ac87c555 in dummy_worker (opaque=0x210fa90) at thread.c:127 #12 0x00007f51abae0182 in start_thread (arg=0x7f51aa8ef700) at pthread_create.c:312 #13 0x00007f51ab80d47d in clone () at ../sysdeps/ unix/sysv/linux/x86_64/clone.S:111 2

  5. Debugging In-Production Software Failures Today Understand root cause #0 0x00007f51abae820b in raise (sig=11) at ../nptl/ sysdeps/unix/sysv/linux/pt-raise.c:37 #1 0x000000000042d289 in ap_buffered_log_writer (r=0x7f51a40053d0, handle=0x20eeba0, strs=0x7f51a4003578, strl=0x7f51a40035e8, nelts=14, len=82) at mod_log_config.c:1368 #2 0x000000000042b10d in config_log_transaction (r=0x7f51a40053d0, cls=0x20b9d50, default_format=0x20ee370) at mod_log_config.c:930 #3 0x000000000042aad6 in multi_log_transaction (r=0x7f51a40053d0) at mod_log_config.c:950 #4 0x000000000046cb2d in ap_run_log_transaction (r=0x7f51a40053d0) at protocol.c:1563 #5 0x0000000000436e81 in ap_process_request (r=0x7f51a40053d0) at http_request.c:312 #6 0x000000000042e9da in ap_process_http_connection (c=0x7f519c000b68) at http_core.c:293 #7 0x0000000000465cdd in ap_run_process_connection (c=0x7f519c000b68) at connection.c:85 #8 0x00000000004661f5 in ap_process_connection (c=0x7f519c000b68, csd=0x7f519c000a20) at connection.c:211 #9 0x0000000000451ba0 in process_socket (p=0x7f519c0009b8, sock=0x7f519c000a20, my_child_num=0, my_thread_num=0, bucket_alloc=0x7f51a4001348) at worker.c:632 #10 0x0000000000451221 in worker_thread (thd=0x210fa90, dummy=0x7f51a40008c0) at worker.c:946 #11 0x00007f51ac87c555 in dummy_worker (opaque=0x210fa90) at thread.c:127 #12 0x00007f51abae0182 in start_thread (arg=0x7f51aa8ef700) at pthread_create.c:312 #13 0x00007f51ab80d47d in clone () at ../sysdeps/ unix/sysv/linux/x86_64/clone.S:111 2

  6. Debugging In-Production Software Failures Today Understand root cause #0 0x00007f51abae820b in raise (sig=11) at ../nptl/ sysdeps/unix/sysv/linux/pt-raise.c:37 #1 0x000000000042d289 in ap_buffered_log_writer (r=0x7f51a40053d0, handle=0x20eeba0, strs=0x7f51a4003578, strl=0x7f51a40035e8, nelts=14, len=82) at mod_log_config.c:1368 #2 0x000000000042b10d in config_log_transaction (r=0x7f51a40053d0, cls=0x20b9d50, default_format=0x20ee370) at mod_log_config.c:930 #3 0x000000000042aad6 in multi_log_transaction (r=0x7f51a40053d0) at mod_log_config.c:950 #4 0x000000000046cb2d in ap_run_log_transaction (r=0x7f51a40053d0) at protocol.c:1563 #5 0x0000000000436e81 in ap_process_request (r=0x7f51a40053d0) at http_request.c:312 #6 0x000000000042e9da in ap_process_http_connection (c=0x7f519c000b68) at http_core.c:293 #7 0x0000000000465cdd in ap_run_process_connection (c=0x7f519c000b68) at connection.c:85 #8 0x00000000004661f5 in ap_process_connection (c=0x7f519c000b68, csd=0x7f519c000a20) at connection.c:211 #9 0x0000000000451ba0 in process_socket (p=0x7f519c0009b8, sock=0x7f519c000a20, my_child_num=0, my_thread_num=0, bucket_alloc=0x7f51a4001348) at worker.c:632 #10 0x0000000000451221 in worker_thread (thd=0x210fa90, dummy=0x7f51a40008c0) at worker.c:946 #11 0x00007f51ac87c555 in dummy_worker (opaque=0x210fa90) at thread.c:127 #12 0x00007f51abae0182 in start_thread (arg=0x7f51aa8ef700) at pthread_create.c:312 #13 0x00007f51ab80d47d in clone () at ../sysdeps/ unix/sysv/linux/x86_64/clone.S:111 Reproduce the failure 2

  7. Debugging In-Production Software Failures Today Understand root cause #0 0x00007f51abae820b in raise (sig=11) at ../nptl/ sysdeps/unix/sysv/linux/pt-raise.c:37 #1 0x000000000042d289 in ap_buffered_log_writer (r=0x7f51a40053d0, handle=0x20eeba0, strs=0x7f51a4003578, strl=0x7f51a40035e8, nelts=14, len=82) at mod_log_config.c:1368 #2 0x000000000042b10d in config_log_transaction (r=0x7f51a40053d0, cls=0x20b9d50, default_format=0x20ee370) at mod_log_config.c:930 #3 0x000000000042aad6 in multi_log_transaction (r=0x7f51a40053d0) at mod_log_config.c:950 #4 0x000000000046cb2d in ap_run_log_transaction (r=0x7f51a40053d0) at protocol.c:1563 #5 0x0000000000436e81 in ap_process_request (r=0x7f51a40053d0) at http_request.c:312 #6 0x000000000042e9da in ap_process_http_connection (c=0x7f519c000b68) at http_core.c:293 #7 0x0000000000465cdd in ap_run_process_connection (c=0x7f519c000b68) at connection.c:85 #8 0x00000000004661f5 in ap_process_connection (c=0x7f519c000b68, csd=0x7f519c000a20) at connection.c:211 #9 0x0000000000451ba0 in process_socket (p=0x7f519c0009b8, sock=0x7f519c000a20, my_child_num=0, my_thread_num=0, bucket_alloc=0x7f51a4001348) at worker.c:632 #10 0x0000000000451221 in worker_thread (thd=0x210fa90, dummy=0x7f51a40008c0) at worker.c:946 #11 0x00007f51ac87c555 in dummy_worker (opaque=0x210fa90) at thread.c:127 #12 0x00007f51abae0182 in start_thread (arg=0x7f51aa8ef700) at pthread_create.c:312 #13 0x00007f51ab80d47d in clone () at ../sysdeps/ unix/sysv/linux/x86_64/clone.S:111 Reproduce the failure 2

  8. Related Work • Collaborative approaches • WER [SOSP’09], CBI [PLDI’05], CCI [OOPSLA’10] • Identifying differences of failing and successful runs • Delta debugging [TSE’02], Symbiosis [PLDI’15] • Record & replay, checkpointing • ODR [SOSP’09], Triage [SOSP’07] • Hardware support • PBI [ASPLOS’13], LBRA/LCRA [ASPLOS’14] 3

  9. Related Work • Collaborative approaches • WER [SOSP’09], CBI [PLDI’05], CCI [OOPSLA’10] • Identifying differences of failing and successful runs • Delta debugging [TSE’02], Symbiosis [PLDI’15] • Record & replay, checkpointing • ODR [SOSP’09], Triage [SOSP’07] • Hardware support • PBI [ASPLOS’13], LBRA/LCRA [ASPLOS’14] 3

  10. Contributions 4

  11. Contributions Goal: automate the manual detective work of debugging 
 4

  12. Contributions Goal: automate the manual detective work of debugging 
 Failure sketching Complements in-house static analysis with in-production dynamic analysis Automatically and efficiently builds accurate failure sketches that show root causes of failures 4

  13. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... ... 5 5 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

  14. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... ... 5 5 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

  15. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... ... 5 5 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

  16. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... ... 5 5 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

  17. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... ... 5 5 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

  18. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... Root ... 5 5 cause 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

Recommend

Getting the most out of the ROOT tutorials Automated conversion from ROOT macros to Jupyter

Getting the most out of the ROOT tutorials Automated conversion from ROOT macros to Jupyter

Getting the most out of the ROOT tutorials Automated conversion from ROOT macros to Jupyter notebooks Pau Miquel, Summer 2016 Supervisors: Pere Mat, Danilo Piparo 1 / 24 Outline 1. Overview 2. ROOT Tutorials 3. Conversion a. Steps

746 views • 30 slides
EE3CL4: Sketching the Root Locus, Introduction to Linear Control Systems Steps 14 Steps 1

EE3CL4: Sketching the Root Locus, Introduction to Linear Control Systems Steps 14 Steps 1

EE 3CL4, 5 1 / 65 Tim Davidson Preliminary examples Principles EE3CL4: Sketching the Root Locus, Introduction to Linear Control Systems Steps 14 Steps 1 and 2 Section 5: Root Locus Procedure Review of Principles Review of Steps 1,

936 views • 58 slides
Plan of the Lecture Review: rules for sketching root loci; introduction to dynamic

Plan of the Lecture Review: rules for sketching root loci; introduction to dynamic

Plan of the Lecture Review: rules for sketching root loci; introduction to dynamic compensation Todays topic: lead and lag dynamic compensation Plan of the Lecture Review: rules for sketching root loci; introduction to dynamic

1.13k views • 97 slides
a culture of failure mathias meyer, @roidrage travis-ci.org / travis-ci.com failure risk 28

a culture of failure mathias meyer, @roidrage travis-ci.org / travis-ci.com failure risk 28

a culture of failure mathias meyer, @roidrage travis-ci.org / travis-ci.com failure risk 28 january 1986 sts-51-l 73 seconds there is no root cause "What you call root cause is simply the place where you stop looking any further"

610 views • 48 slides
Iterative Sketching Agile Arizona 2017 Agenda Who am I? The Power of Sketching When

Iterative Sketching Agile Arizona 2017 Agenda Who am I? The Power of Sketching When

Iterative Sketching Agile Arizona 2017 Agenda Who am I? The Power of Sketching When to Sketch? Who Sketches? Sketching Practice: Software Elements Lets Sketch! Wrap-up The Power of Sketching Create Shared

416 views • 9 slides
INC 342 Lecture 2: Root Locus (cont.) Dr. Benjamas Panomruttanarug Benjamas.pan@kmutt.ac.th

INC 342 Lecture 2: Root Locus (cont.) Dr. Benjamas Panomruttanarug Benjamas.pan@kmutt.ac.th

INC 342 Lecture 2: Root Locus (cont.) Dr. Benjamas Panomruttanarug Benjamas.pan@kmutt.ac.th Sketching Root Locus (review) 1. Number of branches 2. Symmetry 3. Real axis segment 4. Starting and ending points 5. Behavior at infinity BP INC 342

723 views • 12 slides
James 4:1-6 ...But all these causes for management failure have their root in one common

James 4:1-6 ...But all these causes for management failure have their root in one common

James 4:1-6 ...But all these causes for management failure have their root in one common cause: pride. In the simplest terms, pride is devastating. Im not talking about the pride one has in their work or their accomplishments. Im

484 views • 7 slides
1 Wri<ng and maintaining tests is tedious and error-prone.

1 Wri<ng and maintaining tests is tedious and error-prone.

{HEADSHOT} This lesson introduces a powerful automated tes<ng technique called dynamic symbolic execu<on. This technique is based on hybrid

1.18k views • 72 slides
Failure mode and effect analysis (FMEA) for Paediatric Intubation A technique to evaluate high

Failure mode and effect analysis (FMEA) for Paediatric Intubation A technique to evaluate high

Failure mode and effect analysis (FMEA) for Paediatric Intubation A technique to evaluate high risk procedures Third QPEM Conference 11-13 th of January 2019 Beverley Ludick Disclosure I do not have any relevant financial relationship with

300 views • 13 slides
Failure Sketches: A Be2er Way to Debug Baris Kasikci,

Failure Sketches: A Be2er Way to Debug Baris Kasikci,

Failure Sketches: A Be2er Way to Debug Baris Kasikci, Cris<ano Pereira, Gilles Pokam, Benjamin Schubert, Madan Musuvathi, George Candea Failure and Root

666 views • 37 slides
An Automated Social Graph De-anonymization Technique Kumar Sharad 1 George Danezis 2 1 2

An Automated Social Graph De-anonymization Technique Kumar Sharad 1 George Danezis 2 1 2

An Automated Social Graph De-anonymization Technique Kumar Sharad 1 George Danezis 2 1 2 November 3, 2014 Workshop on Privacy in the Electronic Society, Scottsdale, Arizona, USA This Talk 1 The Art of Data Anonymization 2 The D4D Challenge 3

527 views • 51 slides
Curve Sketching Michael Freeze MAT 151 UNC Wilmington Summer 2013 1 / 10 Section 5.4 :: Curve

Curve Sketching Michael Freeze MAT 151 UNC Wilmington Summer 2013 1 / 10 Section 5.4 :: Curve

Curve Sketching Michael Freeze MAT 151 UNC Wilmington Summer 2013 1 / 10 Section 5.4 :: Curve Sketching 2 / 10 Curve Sketching Make an accurate sketch of the graph the function. Identify any important features such as intercepts,

287 views • 10 slides
Evolving Technique Update Cup positioning to avoid Metalosis Bone preparation to avoid Failure of

Evolving Technique Update Cup positioning to avoid Metalosis Bone preparation to avoid Failure of

OS ET 2017 Bellagio Las Vegas Evolving Technique Update Cup positioning to avoid Metalosis Bone preparation to avoid Failure of Ingrowth Thomas Gross 8:52 AM 33 #1 Preventing Metalosis in Hip Resurfacing using RAIL guidelines and NS IOR

344 views • 33 slides
W O R L D B A N K , 1 8 T H A P R I L 2 0 1 7 THE FAILURE OF THE OLD ECONOMIC DEVELOPMENT

W O R L D B A N K , 1 8 T H A P R I L 2 0 1 7 THE FAILURE OF THE OLD ECONOMIC DEVELOPMENT

W O R L D B A N K , 1 8 T H A P R I L 2 0 1 7 THE FAILURE OF THE OLD ECONOMIC DEVELOPMENT MODELS 2 THE EMERGENCE OF GRASS-ROOT, ENTERPRISE LED MOVEMENT TO DEVELOPMENT 3 THE EXAMPLE OF THE ECONOMY OF COMMUNION 4 EOCS VOLUNTARY SHARING

137 views • 11 slides
Sulfur Recovery Unit Reaction Furnaces Waste Heat Boilers Ferrule Failure Root Cause Analysis

Sulfur Recovery Unit Reaction Furnaces Waste Heat Boilers Ferrule Failure Root Cause Analysis

Sulfur Recovery Unit Reaction Furnaces Waste Heat Boilers Ferrule Failure Root Cause Analysis Presented By: Muhammad Bashir Khan Manager, Technical Page 1 Abu Dhabi Gas Development Company Ltd. CONFIDENTIAL SRU and Ferrule Configuration

349 views • 30 slides
AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 2 ISSTA

AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 2 ISSTA

1 THOMAS WALSH 1 , PHIL MCMINN 1 , GREGORY KAPFHAMMER 2 1 UNIVERSITY OF SHEFFIELD, 2 ALLEGHENY COLLEGE AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 2 ISSTA 2017 A LONG TIME AGO THOMAS WALSH

1k views • 72 slides
Advanced Dvorak Technique: An Automated Approach Derek Wroe Acknowledgements: Tim Olander,

Advanced Dvorak Technique: An Automated Approach Derek Wroe Acknowledgements: Tim Olander,

Advanced Dvorak Technique: An Automated Approach Derek Wroe Acknowledgements: Tim Olander, Cooperative Institute for Hurricane Specialist Meteorological Satellite Studies Central Pacific Hurricane Center Chris Velden, Cooperative Institute

384 views • 10 slides
PRESS ROOT TO PRESS ROOT TO CONTINUE: PRESS ROOT TO PRESS ROOT TO CONTINUE: PRESS ROOT TO

PRESS ROOT TO PRESS ROOT TO CONTINUE: PRESS ROOT TO PRESS ROOT TO CONTINUE: PRESS ROOT TO

Mario Vuksan & Tomislav PericinBlackHat USA 2013, Las Vegas PRESS ROOT TO PRESS ROOT TO CONTINUE: PRESS ROOT TO PRESS ROOT TO CONTINUE: PRESS ROOT TO PRESS ROOT TO PRESS ROOT TO PRESS ROOT TO CONTINUE: CONTINUE: CONTINUE: CONTINUE:

690 views • 45 slides
Certicate Transparency Root Explorer Nikita Korzhitskii Niklas Carlsson Web Public Key

Certicate Transparency Root Explorer Nikita Korzhitskii Niklas Carlsson Web Public Key

Certicate Transparency Root Explorer Nikita Korzhitskii Niklas Carlsson Web Public Key Infrastructure (WebPKI) Root certificates of trusted Certificate Authorities e.g. GlobalSign Root CA, Amazon Root CA, GoDaddy Root CA Root 1 Root 2

347 views • 19 slides
Scaling the Root A study of the impact on the DNS root system of increasing the size and

Scaling the Root A study of the impact on the DNS root system of increasing the size and

Scaling the Root A study of the impact on the DNS root system of increasing the size and volatility of the root zone Jaap Akkerhuis Lyman Chapin Patrik Fltstrm Glenn Kowack Bill Manning Lars-Johan Liman Summary of Findings Root Scaling

558 views • 20 slides
The Aortic Valve and The Aortic Valve and The Aortic Root in The Aortic Root in Type A Aortic

The Aortic Valve and The Aortic Valve and The Aortic Root in The Aortic Root in Type A Aortic

The Aortic Valve and The Aortic Valve and The Aortic Root in The Aortic Root in Type A Aortic Dissections Type A Aortic Dissections Optimal Surgical Surgical Optimal Technique Technique Type A aortic aortic dissection dissection

979 views • 78 slides
Safety evaluation at level crossings with the help of an automated measurement technique

Safety evaluation at level crossings with the help of an automated measurement technique

Accidents at level crossings Safety evaluation at level crossings with the help of an automated measurement technique General What we already know Accidents at level crossings: Funded by FFG and BB seldom, but when they

313 views • 5 slides
On Sketching Quadratic Forms Bo Qin The Hong Kong University of Science and Technology January

On Sketching Quadratic Forms Bo Qin The Hong Kong University of Science and Technology January

On Sketching Quadratic Forms Bo Qin The Hong Kong University of Science and Technology January 16, 2016 Joint with: Alexandr Andoni, Jiecao Chen, Robert Krauthgamer, David Woodruff and Qin Zhang Bo Qin On Sketching Quadratic Forms Outline 1

815 views • 35 slides
Root Cause Analysis 1 Root Cause Analysis Root Cause Analysis is a method that is used to

Root Cause Analysis 1 Root Cause Analysis Root Cause Analysis is a method that is used to

Root Cause Analysis 1 Root Cause Analysis Root Cause Analysis is a method that is used to address a problem or non-conformance, in order to get to the root cause of the problem. It is used so we can correct or eliminate the cause,

389 views • 28 slides

More recommend