Fa Fast and Ca Cautious: Leveraging Multi-path Diversity for Transport Loss Recovery in Data Centers Guo Chen Yuanwei Lu, Yuan Meng, Bojie Li, Kun Tan, Dan Pei, Peng Cheng, Layong (Larry) Luo, Yongqiang Xiong, Xiaoliang Wang, and Youjian Zhao
Motivation n Services care about the tail flow completion time (tail FCT) ¨ Large number of flows generated in each operation ¨ Overall performance governed by the last completed flows App App App App App App App App App App Logic Logic Logic Logic Logic Logic Logic Logic Logic Logic Responding to a Large-scale web application hosted in user request Data Center Network (DCN) 16/6/25 2
Motivation n Services care about the tail flow completion time (tail FCT) ¨ Large number of flows generated in each operation ¨ Overall performance governed by the last completed flows n But packet loss hurts tail FCT ¨ Real case in a Microsoft Azure’s DCN Spine switch 2% random drop rate --> increase of 99 th percentile latency of all users DCN tail latency visualization [Pingmesh (SIGCOMM’15)] 16/6/25 3 (a) Normal (b) Spine failure
Outline n Motivation n Packet Loss in DCN n Impact of Packet Loss n Challenge for Loss Recovery n FUSO Design n Evaluation n Summary 16/6/25 4
Packet Loss in DCN n Loss characteristics ¨ Measured in a Microsoft production DCN during Dec. 1 st -5 th , 2015 78% above ToR Mean loss rate 4% Similar in 5 days Loss rate and location distribution of lossy links (loss rate > 1%) 1) Loss frequently happens (the overall loss rate is low) 2) Most losses happen in the network instead of the edge 16/6/25 5
Packet Loss in DCN n Reasons causing loss Bursty; Transient ¨ Congestion loss Greatly mitigated Uneven load-balance Ø (e.g., 1%->0.01%) Incast Ø [Jupiter Rising SIGCOMM’15] ¨ Failure loss Complex; Hard to detect Common Silent random drop Ø & Huge impact Packet black-hole on performance Ø [Pingmesh SIGCOMM’15] 16/6/25 6
Outline n Motivation n Packet Loss in DCN n Impact of Packet Loss ¨ Why loss hurts the tail? ¨ How hard loss hurts? n Challenge for Loss Recovery n FUSO Design n Evaluation n Summary 16/6/25 7
How TCP Handles Loss? n Fast recovery Sender Receiver ¨ Wait for certain number of DACKs to 1-2 detect the loss and retransmit RTT Ack 1-2 3-6 RTT DupAck 3 Retran 3 8
How TCP Handles Loss? n Fast recovery Sender Receiver ¨ Wait for certain number of DACKs to 1-2 detect the loss and retransmit RTT n Timeout (RTO) Ack 1-2 ¨ If not enough DACKs return, retransmit 3-6 after a timeout RTO >> RTT e.g. RTO=5ms, RTT<100us Timeout [Pingmesh (SIGCOMM’15), DCTCP (SIGCOMM’10)] Retran 3 9
How TCP Handles Loss? n Fast recovery Sender Receiver ¨ Wait for certain number of DACKs to 1-2 detect the loss and retransmit RTT n Timeout (RTO) Ack 1-2 ¨ If not enough DACKs return, retransmit 3-6 after a timeout RTO >> RTT e.g. RTO=5ms, RTT<100us Timeout [Pingmesh (SIGCOMM’15), DCTCP (SIGCOMM’10)] Encountering one RTO à Retran 3 dramatically increase the FCT 10
Loss Incurs Timeout n A little loss causes enough timeout to hurt the tail FCT 10KB(analysis) 100KB(testbed) 10KB(testbed) 100KB(analysis) 3% loss à ~10% timeout 99 th FCT > RTO Timeout probability of flows with different sizes passing a path with different packet loss rate 1. 1% loss à more than 1% flows timeout 2. Larger flows (e.g. 100KB) timeout ratio sharply grows when loss rate > 1% a. 16/6/25 11
Loss Incurs Timeout n A little loss causes enough timeout to hurt the tail FCT 10KB(analysis) 100KB(testbed) 10KB(testbed) 100KB(analysis) 3% loss à ~10% timeout 99 th FCT > RTO Timeout probability of flows with different sizes passing a path with different packet loss rate 1. 1% loss à more than 1% flows timeout To avoid RTO 2. Larger flows (e.g. 100KB) timeout ratio sharply grows when loss rate > 1% a. 16/6/25 12
Outline n Motivation n Packet Loss in DCN n Impact of Packet Loss n Challenge for Loss Recovery n FUSO Design n Evaluation n Summary 16/6/25 13
Challenge for TCP Loss Recovery n Prior works add aggressiveness to congestion control to do loss recovery before timeout (RTO) [SIGCOMM’13, RFC 5827] ¨ Tail Loss Probe (TLP) transmit one prober after 2RTT Ø [SIGCOMM’13, RFC 5827] ¨ Instant Recovery (TCP-IR) generate an FEC packet for every group of packets (up to 16) Ø FEC packets also act as probers, delayed 1/4RTT before sent Ø ¨ Proactive/RepFlow [SIGCOMM’13, INFOCOM’14] Duplicate every packet/flow Ø 16/6/25 14
Challenge for TCP Loss Recovery n How long to wait before sending recovery packets? ¨ For congestion loss Should delay enough in case of worsening congestion Ø Bursty ： Lead to multiple consecutive losses [Incast (WREN’09), DCTCP (SIGCOMM’10)] 16/6/25 15
Challenge for TCP Loss Recovery n How long to wait before sending recovery packets? ¨ For congestion loss Should delay enough in case of worsening congestion Ø ¨ For failure loss such as random drop Should recover as fast as possible, otherwise already increase the FCT Ø • Wait 2RTT is too costly [TLP SIGCOMM’13, RFC 5827] • Accurate & high-precision RTT measurementis challenging 16/6/25 16
Brief Summary n Loss easily incurs timeout to hurt the tail n To prevent timeout, prior works add fixed aggressiveness to recover loss before timeout n Hard to adapt to various loss conditions ¨ Should be fast for failure loss ¨ Should be cautious for congestion loss How to accelerate loss recovery as soon as possible, under various loss conditions without causing congestion? 16/6/25 17
Outline n Motivation n Packet Loss in DCN n Impact of Packet Loss n Challenge for Loss Recovery n FUSO Design n Evaluation n Summary 16/6/25 18
FUSO: F ast M u lti-path Lo s s Rec o very n Utilize the “good” paths to proactively conduct loss recovery for “bad” paths ¨ Leveraging path diversity (multiple paths; a few encounter loss) n Fast and Cautious ¨ Fast Proactive (immediate) recovery for potential packet loss utilizing spare Ø transmission opportunity ¨ Cautious Strictly follow congestion control without adding aggressiveness Ø 16/6/25 19
Multi-path Transport Background Sub-fl Su flows: Implicitly/Explicitly mapping to physical paths ma Sender Receiver SF1 SF1 Da Data Di Distribution CWND 1 SF2 SF2 CWND 2 CWND total SF3 SF3 CWND 3 Mu Multi-pa path h Congestion Co Co Control 16/6/25 20
FUSO Sender Receiver SF1 SF1 CWND 1 SF2 SF2 P5 P4 P3 P2 P1 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 21
FUSO Sender Receiver P1 SF1 SF1 CWND 1 SF2 SF2 P5 P4 P3 P2 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 22
FUSO Sender Receiver P1 SF1 SF1 CWND 1 SF2 SF2 P5 P4 P3 P2 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 23
FUSO Sender Receiver SF1 SF1 CWND 1 SF2 SF2 P1 P5 P4 P3 P2 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 24
FUSO Sender Receiver SF1 SF1 CWND 1 SF2 SF2 P1 P2 P5 P4 P3 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 25
FUSO Sender Receiver SF1 SF1 Lost CWND 1 P2 SF2 SF2 P1 P5 P4 P3 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 26
FUSO Sender Receiver SF1 SF1 Lost CWND 1 P2 SF2 SF2 P1 P5 P4 CWND 2 CWND total SF3 SF3 P3 CWND 3 16/6/25 27
FUSO Sender Receiver SF1 SF1 Lost CWND 1 P2 SF2 SF2 P1 P5 P4 CWND 2 CWND total SF3 SF3 P3 CWND 3 16/6/25 28
FUSO Sender Receiver SF1 SF1 Lost CWND 1 P2 SF2 SF2 P1 P3 P5 P4 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 29
FUSO Sender Receiver SF1 SF1 Lost CWND 1 P2 SF2 SF2 P1 P3 P5 P4 ACK P3 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 30
FUSO Sender Receiver SF1 SF1 Lost CWND 1 P2 SF2 SF2 P1 P3 P5 P4 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 31
FUSO Sender Receiver SF1 SF1 Lost CWND 1 P2 SF2 SF2 P1 P3 CWND 2 CWND total SF3 SF3 P5 P4 CWND 3 16/6/25 32
FUSO Sender Receiver SF1 SF1 Lost CWND 1 P2 SF2 SF2 P1 P3 CWND 2 CWND total SF3 SF3 P5 P4 CWND 3 16/6/25 33
FUSO Sender Receiver SF1 SF1 Lost CWND 1 P2 SF2 SF2 P1 P3 P4 P5 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 34
FUSO Sender Receiver SF1 SF1 Lost CWND 1 P2 ACK P1 SF2 SF2 P1 P3 P4 P5 ACK P4&P5 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 35
FUSO Sender Receiver SF1 SF1 Lost CWND 1 P2 SF2 SF2 P1 P3 P4 P5 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 36
FUSO Sender Receiver SF1 SF1 Lost Spare CWND Sp CWND 1 P2 SF2 SF2 P1 P3 P4 P5 CWND 2 CWND total SF3 SF3 No No new data CWND 3 16/6/25 37
FUSO Proactive loss recovery Sender Receiver SF1 SF1 Lost CWND 1 P2 P2 SF2 SF2 P1 P3 P4 P5 CWND 2 CWND total SF3 SF3 CWND 3 16/6/25 38
FUSO Proactive loss recovery Sender Receiver SF1 SF1 Lost CWND 1 P2 P2 SF2 SF2 P1 P3 P4 P5 CWND 2 CWND total “W “Worst” ” sub-fl flow SF3 SF3 CWND 3 “Best” “B ” sub-fl flow 16/6/25 39
FUSO Proactive loss recovery Sender Receiver SF1 SF1 Lost CWND 1 P2 P2 SF2 SF2 P1 P3 P4 P5 CWND 2 CWND total “W “Worst” ” sub-fl flow P2 SF3 SF3 CWND 3 “B “Best” ” sub-fl flow 16/6/25 40
Recommend
More recommend
