Network Failure Mitigation Xin Wu , Daniel Turner, Chao-Chih Chen, - PowerPoint PPT Presentation

NetPilot: Automating Datacenter Network Failure Mitigation Xin Wu , Daniel Turner, Chao-Chih Chen, David A. Maltz, Xiaowei Yang, Lihua Yuan, Ming Zhang

Failures are Common and Harmful • Network failures are common 10,000+ switches 2

Failures are Common and Harmful • Network failures are common • Failures cause long down times 3

Failures are Common and Harmful • Network failures are common Six-month failure logs of production datacenters 25% of failures take 13+ hours to repair • Failures cause long down times Time from detection to repair (minutes) 4

Failures are Common and Harmful • Failures are common due to VERY large datacenters • Failures cause long down times • Long failure duration  large revenue loss 5

Failures are Common and Harmful • Failures are common due to VERY large datacenters • Failures cause long down times • Long failure duration  large revenue loss 6

How to Shorten Failure Recovery Time?

Previous Work • Conventional failure recovery takes 3 steps Detection Diagnosis Repair passive ping active 8

Previous Work • Conventional failure recovery takes 3 steps Detection Diagnosis Repair • Failure localization/diagnosis – [M . K. Aguilera, SOSP’03 ] – [ M. Y. Chen, NSDI’04 ] – [R.R Kompella , NSDI ’05 ] – [P.Bahl , SIGCOMM’07 ] – [S. Kandula , SIGCOMM’09]… 9

Automating Failure Diagnosis is Challenging • Root causes are deep in network stack • Diagnosis involves multiple parties 10

Category Failure types Diagnosis & % Repair Software 21% Link layer loop Find and fix 19% bugs Imbalance  overload 2% Hardware 18% FCS error Replace cable 13% Unstable power Repair power 5% Unknown 23% Switch stops forwarding N/A 9% Imbalance  overload 7% Lost configuration 5% High CPU utilization 2% Configuration Errors on multiple Update 32% 1. Root causes are deep 38% switches configuration in the network stack Errors on one switch 6% • Six -month failure logs from several production DCNs 11

Category Failure types Diagnosis & % Repair Software 21% Link layer loop Find and fix 19% bugs Imbalance  overload 2% Hardware 18% FCS error Replace cable 13% Unstable power Repair power 5% 2. Diagnosis involves Unknown 23% Switch stops forwarding N/A 9% multiple parties Imbalance  overload 7% Lost configuration 5% High CPU utilization 2% Configuration Errors on multiple Update 32% 1. Root causes are deep 38% switches configuration in the network stack Errors on one switch 6% • Six -month failure logs from several production DCNs 12

Category Failure types Diagnosis & % Repair Software 21% Link layer loop Find and fix 19% bugs Imbalance  overload 2% Hardware 18% FCS error Replace cable 13% Unstable power Repair power 5% 2. Diagnosis involves Unknown 23% Switch stops forwarding N/A 9% Failure Diagnosis Requires multiple parties Imbalance  overload 7% Lost configuration 5% Human Intervention ! High CPU utilization 2% Configuration Errors on multiple Update 32% 1. Root causes are deep 38% switches configuration in the network stack Errors on one switch 6% • Six -month failure logs from several production DCNs 13

Can we do something other than failure diagnosis?

NetPilot: Mitigating rather than Diagnosing Failures • Mitigate failure symptoms ASAP, at the cost of reduced capacity Detection Diagnosis Repair 15

16

NetPilot Benefits • Short recovery time • Small network disruption • Low operation cost Automated Detection Diagnosis Repair Mitigation 17

Failure Mitigation is Effective • Most failures can be mitigated by simple actions • Mitigation is feasible due to redundancy 18

Category Failure types Mitigation Repair % Software Link layer loop Deactivate port Find and fix 19% 21% bugs Imbalance- Restart switch triggered overload 2% Hardware FCS error Deactivate port Replace cable 13% 18% Unstable power Deactivate switch Repair power 5% Unknown Switch stops Restart switch N/A 9% 23% forwarding Imbalance- Restart switch 7% triggered overload Lost configuration Restart switch 5% High CPU Restart switch 2% utilization Configurati Errors on multiple n/a Update 32% on 38% switches configuration Errors on single Deactivate switch 6% 19 switch

Category Failure types Mitigation Repair % Software Link layer loop Deactivate port Find and fix 19% 21% bugs Imbalance- Restart switch triggered overload 2% Hardware FCS error Deactivate port Replace cable 13% 18% Unstable power Deactivate switch Repair power 5% Unknown Switch stops Restart switch N/A 9% 23% forwarding Imbalance- Restart switch 7% triggered overload Lost configuration Restart switch 5% High CPU Restart switch 2% utilization Configurati Errors on multiple n/a Update 32% on 38% switches configuration Errors on single Deactivate switch 6% 20 switch

Category Failure types Mitigation Repair % Software Link layer loop Deactivate port Find and fix 19% 21% bugs Imbalance- Restart switch triggered overload 2% Hardware FCS error Deactivate port Replace cable 13% 18% Unstable power Deactivate switch Repair power 5% 68% of failures can be Unknown Switch stops Restart switch N/A 9% 23% forwarding mitigated by simple actions Imbalance- Restart switch 7% triggered overload Lost configuration Restart switch 5% High CPU Restart switch 2% utilization Configurati Errors on multiple n/a Update 32% on 38% switches configuration Errors on single Deactivate switch 6% 21 switch

22

23

Outline • Automating failure diagnosis is challenging • Failure mitigation is effective • How to automate mitigation? • NetPilot evaluations • Conclusion 24

A Strawman NetPilot: Trial-and-error Network failure Localization Roll back if Execute an necessary action No Failure End mitigated? Yes 25

NetPilot: Challenges & Solutions Network failure Localization Localization 1. Blind trial-and-error takes a long time Roll back if necessary Execute an action No Failure End mitigated? Yes 26

NetPilot: Challenges & Solutions Network failure Localization Localization 1. Blind trial-and-error takes a long time Roll back if Failure specific localization necessary Execute an action No Failure End mitigated? Yes 27

NetPilot: Challenges & Solutions Network failure Localization Localization 2. Partition/overload network Estimate impact Impact estimation Roll back if necessary Execute an action No Failure End mitigated? Yes 28

29

30

NetPilot: Challenges & Solutions Network failure Localization Localization Estimate impact 3. Different actions have different side-effects Rank actions Roll back if necessary Rank actions based on impact Execute an action No Failure End mitigated? Yes 31

Failure Specific Localization • Limited # of failure types • Domain knowledge improves accuracy Failure types 1 . Link layer loop 2 . Imbalance-triggered overload 3 . FCS error 4 . Unstable power 5 . Switch stops forwarding 6 . Imbalance-triggered overload 7 . Lost configuration 8 . High CPU utilization 9 . Errors on multiple switches 10 . Errors on single switch 32

Example : Frame Check Sequence (FCS) Errors • 13% of all the failures • Cut-through switching – Forward frames before checksums are verified • Increase application latency 33

Localizing FCS Errors error frames seen on L frames corrupted by L frames corrupted by other links & traverse L • x L : link corruption rate • # of variables = # of equations = # of links • Corrupted links: x L > 0 34

NetPilot Overview Network failure Localization Estimate impact Rank actions Roll back if necessary Execute an action No Failure End mitigated? Yes 35

Impact Metrics • Derived from Service Level Agreement ( SLA ) – Availability: online_server_ratio – Packet loss: total_lost_pkt – latency: max_link_utilization • Small link utilization  small (queuing) delay • Total_lost_pkt & max_link_utilization derived from utilization of individual links 36

Estimating Link Utilization Action Impact Link Traffic Estimator utilization Topology • # of flows >> redundant paths – Traffic evenly distributed under ECMP • Estimate the load contributed by each flow on each link • Sum up the loads to compute utilization 37

Link Utilization Estimation is Highly Accurate • 1-month traffic from a 8000-server network – Log socket events on each server • Ground truth: SNMP counters 38

NetPilot Overview Network failure Localization Estimate impact Choose the action Rank actions Roll back if with the least impact necessary Execute an action No Failure End mitigated? Yes 39

Outline • Automating failure diagnosis is challenging • Failure mitigation is effective • How to automate mitigation? – Localization  impact estimation  ranking • NetPilot evaluations – Mitigating load imbalance – Mitigating FCS errors – Mitigating overload • Conclusion 40

Load Imbalance • Agg a stops receiving traffic • Localize to 4 suspects core a core b Agg b Agg a 41

42

43