CROSS-LAYER CROSS-LAYER LATENCY-AWARE AND -PREDICTABLE - PowerPoint PPT Presentation

CROSS-LAYER CROSS-LAYER LATENCY-AWARE AND -PREDICTABLE LATENCY-AWARE AND -PREDICTABLE DATA COMMUNICATION DATA COMMUNICATION Andreas Schmidt Apr 07th, 2020 Anywhere on Earth 復 􏥧  􏞣 Doctoral Colloquium 1

   􏋜  􏡠  􏂆 CYBER-PHYSICAL SYSTEMS CYBER-PHYSICAL SYSTEMS 2 . 1

 􏈴  􏂆  􏂆 􏙢  􏂆   􏂆  􏂆  􏇁 􏡠  DISTRIBUTED CYBER-PHYSICAL SYSTEMS DISTRIBUTED CYBER-PHYSICAL SYSTEMS 2 . 2

A REAL-WORLD A REAL-WORLD DISTRIBUTED CPS DISTRIBUTED CPS  􏡠 Proprietary Solution Enhanced Shock Burst (ESB)  􏡠 Our Open Solution Wi-Fi, IP, PRRT Publication-in-Progress [BSGPH20] 0:00 / 0:41 2 . 3

2008 2008  􏇛 Cyber Physical Systems: Design Challenges [Lee08] "Passing of time is inexorable" Most abstractions insufficiently capture timing Edward A. Lee UC Berkeley 2 . 4

LATENCY-* LATENCY-* AWARENESS AWARENESS PREDICTABILITY PREDICTABILITY  􏁯  􏇦 Know what latency you require Built upon awareness Know what latency you cause Improve confidence Share this with others Share confidence with others 2 . 5

STATE OF AFFAIRS IN NETWORKING STATE OF AFFAIRS IN NETWORKING The Internet most dominant interconnecting communication systems best effort , i.e. no predictable timing (by design) Link: Transmission latencies known, but not exposed  􏃂 IP: No timing at all  􏇡 Transport Layer: 􏑶  TCP:  UDP:  RTP:  Let's go back to proprietary, closed networks? No! Why? 􏃬  2 . 6

Internet Architecture [Cla18] provides INTEROPERABILITY INTEROPERABILITY 2 . 7

RESEARCH QUESTION * RESEARCH QUESTION CAN WE MAINTAIN CAN WE MAINTAIN INTEROPERABILITY INTEROPERABILITY AND AND AT THE SAME TIME AT THE SAME TIME ACHIEVE ACHIEVE LATENCY-AWARENESS AND -PREDICTABILITY? LATENCY-AWARENESS AND -PREDICTABILITY? * as you might assume, this is a question too big for a PhD thesis 2 . 8

TODAY'S RESEARCH QUESTION * TODAY'S RESEARCH QUESTION CAN WE BUILD CAN WE BUILD TRANSPORT LAYERS TRANSPORT LAYERS WITH WITH LATENCY-AWARENESS AND -PREDICTABILITY? LATENCY-AWARENESS AND -PREDICTABILITY? * 2 . 9

OUTLINE OUTLINE 􏃭  􏙄  􏋳   􏕜  􏇦 PRRT PRRT X-LAP X-LAP X-PACE X-PACE TTS TTS OUTLOOK OUTLOOK 2 . 10

 􏃭 PRRT PRRT [SH16a, GGG + 19] 3 . 1

WHAT IS THE PROBLEM? WHAT IS THE PROBLEM? Full reliability infinite time [Sha48, Fei54] ⇒ Most applications... ... have (some) time constraint ... can tolerate (some) missing data  (Today's) transport protocols do not consider that 3 . 2

FLAVOURS OF RELIABILITY FLAVOURS OF RELIABILITY  Full  Partial 3 . 3

PRRT PROTOCOL PRRT PROTOCOL Predictably Reliable Real-time Transport Fundamental work in [Gor12] Redesign for control applications since 2015 3 . 4

PRRT'S PLACE IN THE ISO MODEL PRRT'S PLACE IN THE ISO MODEL Default In General Control/Multimedia App Control/Multimedia App PRRT PRRT UDP Any Layer with IP Process-to-Process Datagram Delivery Any MAC 3 . 5

PRRT CHANNEL MEASUREMENT PRRT CHANNEL MEASUREMENT Parameter Approach Round-trip Time NTP Algorithm [MMBK10] Delivery Rate IETF Dra� [CCYJ17] Loss Rate & Correlation Statistics over success-failure sequences Selective ACKs with feedback Measurements exposed via API 3 . 6

PRRT'S RECEIVE MODES PRRT'S RECEIVE MODES 3 . 7

PRRT'S FUNCTIONS PRRT'S FUNCTIONS Error Control Adaptive Hybrid Error Coding [Gor12] (not covered in this talk) Congestion Control Based on BBR ideas [CCG + 16] Utilization & measurement Latency-avoidance Rate and Flow Control Later: X-Pace  􏋳 3 . 8

EVALUATION EVALUATION [GGG + 19] 3 . 9

USABILITY USABILITY Technology  􏄢 Policy  􏃤 C implementation & API Publicly available at prrt.larn.systems Python API (via Cython ) MIT licensed Rust API (via bindgen ) 3 . 10

 􏙄 X-LAP X-LAP [RSH+17, RSH + 18] 4 . 1

WHAT IS THE PROBLEM? WHAT IS THE PROBLEM?  CPS demand predictable latency and timing Cross-layer, intra-host profiling is required valgrind , wireshark , packetdrill ... insufficient Microsecond regime is increasingly relevant [BMPR17] 4 . 2

STAMPS STAMPS Processor Cycles  􏋜 Wall-Time  􏀘 rdtsc CLOCK_MONOTONIC cheap to capture (≈ 10ns) expensive to capture (≈ 70ns) non-trivial relation to wall-time 4 . 3

WHAT IS X-LAP? WHAT IS X-LAP? X = Cross, Lap = time to do one round X-Lap is a cross-layer, intra-host, timing & latency analysis tool download at xlap.larn.systems 4 . 4

STAMPS IN X-LAP STAMPS IN X-LAP e.g.: ChannelTransmit [ms]  􏀘 ↑ t = 5 ms ↓ c = 7 156 294 [1]  􏋜 4 . 5

X-LAP IN ACTION X-LAP IN ACTION 4 . 6

TRACES DATA FORMAT TRACES DATA FORMAT SN ... ... ChannelTransmit_T ChannelTransmit_C 42 ... 5.012ms ... 7 156 294 43 ... 5.029ms ... 47 961 303 ... ... ... .... ... 4 . 7

RUNTIME TRACING CODE RUNTIME TRACING CODE 1 void send_packet (PrrtPacket* pkt) { 2 pace(); // delays sending of data using cross-layer pacing 3 wait_for_free_congestion_window_space(); 4 compute_next_send_time(); // for cross-layer pacing 5 char [] bytes = serialize_packet_to_bytes(pkt); 6 XlapStamp_Cycle(LinkTransmitStart, pkt->seqno); 7 struct timespec timestamp ; 8 uint64_t cyclestamp; 9 send(sock_fd, bytes, & timestamp, &cyclestamp); 10 XlapStamp_TimeValue(ChannelTransmit, timestamp, pkt->seqno); 11 XlapStamp_CycleValue(ChannelTransmit, cyclestamp, pkt->seqno); 12 track_outstanding_packet(pkt); 13 XlapStamp_Cycle(LinkTransmitEnd, pkt->seqno); 14 } 4 . 8

CYCLE TO TIME CYCLE TO TIME 4 . 9

PACKET TRACE PACKET TRACE 4 . 10

TRACE JITTER TRACE JITTER 4 . 11

MULTISERIES CORRELATION MULTISERIES CORRELATION 4 . 12

 􏋳 X-PACE X-PACE [SRGP + 19] 5 . 1

WHAT IS THE PROBLEM? WHAT IS THE PROBLEM?  Bufferbloat [GN12] Unpaced Paced 5 . 2

PACING PACING PACE PACE intentionally delaying time a transmission P , [ P ] = unit of work 5 . 3

PACED SYSTEMS PACED SYSTEMS CROSS-LAYER PACING CROSS-LAYER PACING every step considers S is paced iff i ∈ [0 : n − 1] and adapts its pace to the P ( i ) ∀ i , j ∈ [0 : n − 1] : bottleneck pace P ( btl ) P ( i ) P ( j ) i < j ⇒ ≥ 5 . 4

PROPAGATE PACES PROPAGATE PACES 5 . 5

REDUCE 無 駄 * REDUCE 無 駄 * muda or waste, originally described by Toyota Production System [Ohn88] Inventory : only a delivered message is useful  􏑩 Waiting : a message loses value over time  􏇛 Overprocessing : as-fast-as-possible causes losses  􏒌 5 . 6

X-PACE IN PRRT X-PACE IN PRRT 5 . 7

CROSS-LAYER API CROSS-LAYER API send_sync() socket.btl_pace Passive Active 5 . 8

EVALUATION APPROACH EVALUATION APPROACH PRRT with X-Pace vs. Optimized TCP Variants (CUBIC, BBR) NODELAY , QUICKACK low_latency SNDBUF & RCVBUF low write() not send() 5 . 9

INTERNET SCENARIO INTERNET SCENARIO traceroute to 79.199.28.123 (79.199.28.123), 30 hops max , 60 byte packets 1 vlan-herfet-neu .nt.uni-saarland.de (134.96.86.1) 0.400 ms 0.358 ms 0.420 ms 2 c65eb36-win .net.uni-saarland.de (134.96.6.54) 0.328 ms 0.314 ms 0.384 ms 3 cr-dui1-te0-5-0-2-4 .x-win.dfn.de (188.1.241.185) 8.687 ms 8.688 ms 8.680 ms 4 cr-fra2-be16 .x-win.dfn.de (188.1.144.178) 9.618 ms 9.732 ms 9.727 ms 5 ffm-b12-link .telia.net (213.248.97.40) 9.226 ms 9.563 ms 9.191 ms 6 ffm-bb3-link .telia.net (62.115.142.46) 10.069 ms 9.761 ms 9.704 ms 7 ffm-b4-link .telia.net (62.115.120.6) 9.802 ms 9.792 ms 10.024 ms 8 dtag-ic-319284-ffm-b4 .c.telia.net (213.248.93.187) 10.374 ms 10.175 ms 10.17 9 91.23.246.213 (91.23.246.213) 13.230 ms 13.252 ms 13.234 ms 10 * * * ... 30 * * * 5 . 10

RESULTS RESULTS 5 . 11

􏕜  TRANSPARENT TRANSMISSION TRANSPARENT TRANSMISSION SEGMENTATION SEGMENTATION [SH16b, SH17b] 6 . 1

WHAT IS A PROBLEM? WHAT IS A PROBLEM? Small receiver buffer flow-limited communication 􏁲  ⇒ 6 . 2

TRANSPARENT TRANSMISSION TRANSPARENT TRANSMISSION SEGMENTATION SEGMENTATION Relay: Transport-Layer Performance-Enhancing-Proxy [ RFC3135 ]  􏃭 6 . 3

TTS IN ACTION TTS IN ACTION 6 . 4

IDEAL RECEIVER BUFFER IDEAL RECEIVER BUFFER derived from a theoretical flow control model RTT 1 B relay = ⋅ B recv RTT 2 R eff , TTS RTT 1 Λ = = + 1 R eff , E 2 E RTT 2 6 . 5

EVALUATION EVALUATION B recv = 4 KiB, RT T 1 = 20 ms, RT T 2 = 5 ms ⇒ Λ = 5, B relay = 4 ⋅ 4KiB 6 . 6

BENEFITS OF TTS BENEFITS OF TTS Lossy Last Mile High Reordering due to Jitter Insufficient Receiver Buffer 6 . 7

 􏇦 OUTLOOK OUTLOOK e.LARN, [SGPH20] 7 . 1

FUTURE WORK FUTURE WORK  􏍷  􏣲  􏇡 Energy- Statistical Multicast Awareness Shaping Support 7 . 2

WRAP-UP WRAP-UP 􏂆  NETWORKED CPS NETWORKED CPS 􏃭  􏙄  􏋳  􏕜  PRRT PRRT X-LAP X-LAP X-PACE X-PACE TTS TTS [SH16a, GGG + 19] [RSH+17, RSH + 18] [SRGP + 19] [SH16b, SH17b] 􏇦  OUTLOOK OUTLOOK e.LARN, [SGPH20] 8 . 1

 􏋾 QUESTIONS QUESTIONS 8 . 2