Bufferbloat mitigation in the Linux WiFi stack – status and ongoing work Toke Høiland-Jørgensen LLC 2019 Lund, Sweden, May 2019 1
Outline Background: Bufferbloat mitigation in Linux Fixing bloat in the WiFi stack Airtime fairness and policy Status of mainline drivers Future / ongoing work - Toke Høiland-Jørgensen < toke@redhat.com > 2
Background: Bufferbloat mitigation in Linux What is bufferbloat? Effects of mitigation in the qdisc layer Pieces of CAKE Byte Queue Limits - Toke Høiland-Jørgensen < toke@redhat.com > 3
What is bufferbloat? - Toke Høiland-Jørgensen < toke@redhat.com > 4
Mitigation in the qdisc layer (wired links) 9.6 9.5 Mean TCP goodput (Mbit/s) 9.4 fq_nocodel 9.3 sfq pfifo_fast fq_codel 9.2 pie codel 9.1 ared 9.0 8.9 8.8 100 80 60 40 20 0 Mean induced latency (ms) - Toke Høiland-Jørgensen < toke@redhat.com > 5
Pieces of CAKE The sch_cake qdisc adds: cake cake_dst cake_src cake_triple fq_codel 2.5 Traffic shaping Host-based fairness queueing 2.0 DiffServ handling Mbits/s 1.5 TCP ACK filtering 1.0 0.5 0.0 A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D - Toke Høiland-Jørgensen < toke@redhat.com > 6
Important feature: Byte Queue Limits (BQL) Queueing needs to happen where it can be managed There’s another queue in the hardware For Ethernet, we have BQL Enabled in most drivers now Check your driver, it’s a pretty small patch - Toke Høiland-Jørgensen < toke@redhat.com > 7
Fixing bloat in WiFi - Toke Høiland-Jørgensen < toke@redhat.com > 8
The qdisc approach doesn’t work for WiFi pie 60 pfifo_fast codel sfq 50 fq_codel Mean TCP goodput (Mbit/s) ared 40 fq_nocodel 30 20 10 fq_codel sfq ared fq_nocodel pie pfifo_fast codel 0 350 300 250 200 150 100 50 Mean induced latency (ms) - Toke Høiland-Jørgensen < toke@redhat.com > 9
Fixing queueing in the WiFi stack Qdisc layer (bypassed) Per HW queue 1000 * Qdisc layer (x4) Assign TID FIFO* TID TID MAC layer Split flows Split flows *Can be replaced with an arbitrary configuration (Global limit) (Global limit) 8192 8192 MAC layer Assign TID FQ- FQ- CoDel CoDel Retries Retries TID TID 123 123 retry_q retry_q ath9k driver buf_q buf_q retry_q retry_q Prio Prio ath9k driver Prio Prio HW queue RR RR (x4) 2 aggr 2 aggr FIFO FIFO To hardware To hardware - Toke Høiland-Jørgensen < toke@redhat.com > 10
Queueing latency impact 1.0 Cumulative probability 0.8 0.6 0.4 0.2 0.0 10 1 10 2 10 3 Latency (ms) With our solution Without our solution - Toke Høiland-Jørgensen < toke@redhat.com > 11
Throughput impact Station 1 Station 2 Station 3 Average 40 30 Mbits/s 20 10 0 FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime - Toke Høiland-Jørgensen < toke@redhat.com > 12
Airtime: fairness and policy How can we control airtime usage of WiFi devices? - Toke Høiland-Jørgensen < toke@redhat.com > 13
802.11 Performance Anomaly Effective transmission time and rate (for station ): T ( i ) R ( i ) i ∈ I 1 ⎧ with fairness ⎪ | I | ⎪ T _ data ( i ) T ( i ) = ⎨ otherwise ∑ _ j ∈ IT _ data ( j ) ⎪ ⎩ ⎪ R ( i ) = T ( i ) R _0( i ) Where is the effective rate of a station transmitting L _ i R _0( i ) = T _ data ( i )+ T _ oh without collisions. - Toke Høiland-Jørgensen < toke@redhat.com > 14
Airtime fairness scheduler Idea: Schedule stations to equalise airtime usage We already have per-station queueing The driver schedules stations anyway So we can simply account airtime using DRR scheduling First prototype: Change the ath9k driver - Toke Høiland-Jørgensen < toke@redhat.com > 15
Airtime fairness - achieved fairness (ath9k) FIFO FQ-CoDel FQ-MAC Airtime fair FQ 1.0 0.8 Airtime share 0.6 0.4 0.2 0.0 Fast 1 Fast 2 Slow Fast 1 Fast 2 Slow Fast 1 Fast 2 Slow Fast 1 Fast 2 Slow - Toke Høiland-Jørgensen < toke@redhat.com > 16
Airtime fairness - throughput benefit (ath9k) Station 1 Station 2 Station 3 Average 40 30 Mbits/s 20 10 0 FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime - Toke Høiland-Jørgensen < toke@redhat.com > 17
Beyond fairness: “Polifi” Enable airtime policy enforcement, supporting: Userspace Policy Prioritising single devices User Station daemon configuration state tracker Assoc / disassoc notifications Use case, e.g.: “My TV needs more airtime” Station queue state Set station weights Kernel Balancing device groups mac80211 subsystem Use case, e.g.: 5G network slicing Station Station Station Weight Weight Weight Deficits Deficits Deficits Limiting groups to a maximum capacity share Use case, e.g.: Guest network Weighted DRR Airtime usage scheduler Device driver WiFi hardware - Toke Høiland-Jørgensen < toke@redhat.com > 18
Policy operating modes Weights set by userspace daemon (hostapd), supporting three modes: Static mode: Specify MAC priority in config Dynamic mode: Specify weight per group (BSS) Limit mode: Like dynamic, but only limit some groups - Toke Høiland-Jørgensen < toke@redhat.com > 19
Policy enforcement effect 1.00 Airtime share 0.75 0.50 0.25 0.00 0 10 20 30 40 Time (s) 1.00 0.75 Airtime share 0.50 0.25 0.00 0 10 20 30 40 Time (s) - Toke Høiland-Jørgensen < toke@redhat.com > 20
Status in the mainline kernel Everything presented here is upstream as of 5.1! - Toke Høiland-Jørgensen < toke@redhat.com > 21
Upstream feature support - kernel versions Feature Version BQL 3.3 sch_fq_codel 3.5 sch_cake 4.19 WiFi queueing 4.10 Airtime fairness (ath9k) 4.11 Airtime fairness + policy 5.1 - Toke Høiland-Jørgensen < toke@redhat.com > 22
Upstream feature support - WiFi drivers (5.1) Driver WiFi-Q Airtime ath9k ✓ ✓ ath10k ( ✓ ) ✓ mt76 ✓ ✕ iwlwifi ✓ ✕ others ✕ ✕ ( ✓ ): See caveat in following slides - Toke Høiland-Jørgensen < toke@redhat.com > 23
Future / ongoing work Airtime Queue Limits Driver support Virtual time-based scheduler - Toke Høiland-Jørgensen < toke@redhat.com > 24
Airtime Queue Limits (AQL) Just like with Ethernet drivers, there can be queueing in WiFi devices E.g., seconds of queueing in ath10k (hence the ( ✓ )). For WiFi we know the packet duration from rate control AQL means limiting queueing based on airtime Prototyped by Google for ath10k with great effect - Toke Høiland-Jørgensen < toke@redhat.com > 25
Driver support Driver support is partial. Efforts to fix this: Porting the full mac80211 TX path to TXQs A method to estimate airtime for devices w/o hw support This may tie into AQL support - Toke Høiland-Jørgensen < toke@redhat.com > 26
Virtual time-based scheduler Current airtime scheduler is DRR-based Requires heavy throttling of stations to achieve fairness A virtual time-based scheduler doesn’t And may also allow MU-MIMO Prototyped in RFC patch - Toke Høiland-Jørgensen < toke@redhat.com > 27
Summary Bufferbloat mitigation in Linux has come a long way! Very close to being “done” for WiFi Everything is upstream as of 5.1 Missing: AQL and full driver support - Toke Høiland-Jørgensen < toke@redhat.com > 28
References Papers with more detail (from my PhD ): Bufferbloat qdisc comparison: The Good, the Bad and the WiFi Description of CAKE: Piece of CAKE WiFi queueing and fairness: Ending the Anomaly Airtime policy: PoliFi - Toke Høiland-Jørgensen < toke@redhat.com > 29
Recommend
More recommend
