Centers Parmjeet Singh, Myungjin Lee Sagar Kumar, Ramana Rao - PowerPoint PPT Presentation

Enabling Flow-level Latency Measurements across Routers in Data Centers Parmjeet Singh, Myungjin Lee Sagar Kumar, Ramana Rao Kompella

Latency-critical applications in data centers  Guaranteeing low end-to-end latency is important  Web search (e.g., Google’s instant search service)  Retail advertising  Recommendation systems  High-frequency trading in financial data centers  Operators want to troubleshoot latency anomalies  End-host latencies can be monitored locally  Detection, diagnosis and localization through a network: no native support of latency measurements in a router/switch

Prior solutions  Lossy Difference Aggregator (LDA)  Kompella et al. [SIGCOMM ’09]  Aggregate latency statistics  Reference Latency Interpolation (RLI)  Lee et al. [SIGCOMM ’10]  Per-flow latency measurements More suitable due to more fine-grained measurements

Deployment scenario of RLI  Upgrading all switches/routers in a data center network  Pros  Provide finest granularity of latency anomaly localization  Cons  Significant deployment cost  Possible downtime of entire production data centers  In this work, we are considering partial deployment of RLI  Our approach: RLI across Routers (RLIR)

Overview of RLI architecture Router Ingress I  Goal Egress E  Latency statistics on a per-flow basis between interfaces  Problem setting  No storing timestamp for each packet at ingress and egress due to high storage and communication cost  Regular packets do not carry timestamps

Overview of RLI architecture Linear interpolation L 2 1 1 Ingress I Egress E Delay line 1 Reference L Latency R Packet R 2 Estimator Interpolated Injector delay  Premise of RLI: delay locality Time  Approach 1) The injector sends reference packets regularly 2) Reference packet carries ingress timestamp 3) Linear interpolation : compute per-packet latency estimates at the latency estimator 4) Per-flow estimates by aggregating per-packet estimates

Full vs. Partial deployment RLI Sender (Reference Packet Injector) RLI Receiver (Latency Estimator) Switch 1 Switch 3 Switch 5 Switch 2 Switch 4 Switch 6  Full deployment: 16 RLI sender-receiver pairs  Partial deployment: 4 RLI senders + 2 RLI receivers  81.25 % deployment cost reduction

Case 1: Presence of cross traffic RLI Sender (Reference Packet Injector) RLI Receiver (Latency Estimator) Switch 1 Switch 3 Switch 5 Link utilization Bottleneck Cross estimation on Switch 1 Link Traffic Switch 2 Switch 4 Switch 6  Issue: Inaccurate link utilization estimation at the sender leads to high reference packet injection rate  Approach  Not actively addressing the issue  Evaluation shows no much impact on packet loss rate increase  Details in the paper

Case 2: RLI Sender side RLI Sender (Reference Packet Injector) RLI Receiver (Latency Estimator) Switch 1 Switch 3 Switch 5 Switch 2 Switch 4 Switch 6  Issue: Traffic may take different routes at an intermediate switch  Approach: Sender sends reference packets to all receivers

Case 3: RLI Receiver side RLI Sender (Reference Packet Injector) RLI Receiver (Latency Estimator) Switch 1 Switch 3 Switch 5 Switch 2 Switch 4 Switch 6  Issue: Hard to associate reference packets and regular packets that traversed the same path  Approaches  Packet marking: requires native support from routers  Reverse ECMP computation : ‘reverse’ engineer intermediate routes using ECMP hash function  IP prefix matching at limited situation

Deployment example in fat-tree topology RLI Sender (Reference Packet Injector) RLI Receiver (Latency Estimator) Reverse ECMP computation / IP prefix matching IP prefix matching

Evaluation  Simulation setup  Trace: regular traffic (22.4M pkts) + cross traffic (70M pkts)  Simulator 10% / 1% RLI RLI injection rate Sender Regular Receiver Reference Traffic packets Packet Traffic Switch1 Switch2 Divider Trace Cross Traffic Cross Injector Traffic  Results  Accuracy of per-flow latency estimates

Accuracy of per-flow latency estimates Bottleneck link utilization: 93% 67% 10% injection 10% injection 1% injection CDF 1% injection Relative error 1.2% 4.5% 18% 31%

Summary  Low latency applications in data centers  Localization of latency anomaly is important  RLI provides flow-level latency statistics, but full deployment (i.e., all routers/switches) cost is expensive  Proposed a solution enabling partial deployment of RLI  No too much loss in localization granularity (i.e., every other router)

Thank you! Questions?