Modifying existing applications for 100 Gigabit Ethernet Jelte - PowerPoint PPT Presentation

Modifying existing applications for 100 Gigabit Ethernet Jelte Fennema University of Amsterdam 29th June 2016

Introduction Approach Results Conclusion Introduction ◮ 100 Gigabit Ethernet (100GbE) is becoming common ◮ Measuring the network speed is important ◮ iperf3 is unable to saturate a 100GbE link ◮ Can only reach ~45Gbit/s ◮ CPU core is being maxed out Modifying existing applications for 100 Gigabit Ethernet 2/25 Jelte Fennema

Introduction Approach Results Conclusion DPDK as a possible solution ◮ The Linux networking stack is too slow ◮ Possible solution: Data Plane Development Kit (DPDK) ◮ Developed by Intel for very fast network I/O ◮ Includes special high performance drivers Modifying existing applications for 100 Gigabit Ethernet 3/25 Jelte Fennema

Introduction Approach Results Conclusion Linux networking Application System calls and interupts Linux kernel and network drivers Send/receive data Network card Figure 1: Normal Linux networking Modifying existing applications for 100 Gigabit Ethernet 4/25 Jelte Fennema

Introduction Approach Results Conclusion DPDK networking Application, DPDK and network drivers Send/receive data Network card Figure 2: DPDK networking Modifying existing applications for 100 Gigabit Ethernet 5/25 Jelte Fennema

Introduction Approach Results Conclusion Current DPDK packet generators ◮ Moongen ◮ Achieved 120Gbit/s over multiple 10GbE interfaces ◮ Doesn’t support our Network Interface Card (NIC) ◮ Pktgen ◮ Developed by Intel as official DPDK application ◮ Both have not been tested on 100GbE NICs Modifying existing applications for 100 Gigabit Ethernet 6/25 Jelte Fennema

Introduction Approach Results Conclusion Research questions 1. Can current DPDK packet generators saturate a 100GbE link? 2. What is necessary to modify iperf3 to use DPDK? 3. What throughput improvements can be be achieved by modifying iperf3 to use DPDK? Modifying existing applications for 100 Gigabit Ethernet 7/25 Jelte Fennema

Introduction Approach Results Conclusion Setup 100GbE 100GbE Switch Server 1 Server 2 Figure 3: The test setup Modifying existing applications for 100 Gigabit Ethernet 8/25 Jelte Fennema

Introduction Approach Results Conclusion Accelerated Network Stack ◮ iperf3 uses regular TCP connections ◮ DPDK itself can only be used for sending raw packets Modifying existing applications for 100 Gigabit Ethernet 9/25 Jelte Fennema

Introduction Approach Results Conclusion Accelerated Network Stack ◮ iperf3 uses regular TCP connections ◮ DPDK itself can only be used for sending raw packets ◮ ANS is a FreeBSD networking stack modified for DPDK ◮ Contains support for popular network protocols Modifying existing applications for 100 Gigabit Ethernet 9/25 Jelte Fennema

Introduction Approach Results Conclusion New iperf3 versions Two new iperf3 versions are created: ◮ One modified to use ANS ◮ A Linux version with comparable modifications Modifying existing applications for 100 Gigabit Ethernet 10/25 Jelte Fennema

Introduction Approach Results Conclusion Focus ◮ TCP ◮ Single stream ◮ Single core Modifying existing applications for 100 Gigabit Ethernet 11/25 Jelte Fennema

Introduction Approach Results Conclusion Performance settings ◮ Setting CPU affinity ◮ isolcpus ◮ Disable hyperthreading Modifying existing applications for 100 Gigabit Ethernet 12/25 Jelte Fennema

Introduction Approach Results Conclusion DPDK baseline ◮ Pktgen could reach 86Gbit/s ◮ This is for raw packets Modifying existing applications for 100 Gigabit Ethernet 13/25 Jelte Fennema

Introduction Approach Results Conclusion iperf3 multi process baseline 100 90 80 Total Gbit/s 70 60 50 40 1 2 3 4 Number of iperf3 processes Figure 4: iperf3 speedtest with multiple processes Modifying existing applications for 100 Gigabit Ethernet 14/25 Jelte Fennema

Introduction Approach Results Conclusion Modifications iperf3 ◮ Event loop conversion from select to epoll style ◮ Removal of synchronous network I/O Modifying existing applications for 100 Gigabit Ethernet 15/25 Jelte Fennema

Introduction Approach Results Conclusion Modifications to iperf3 Three iperf3 versions: 1. Regular 2. Epoll 3. ANS Modifying existing applications for 100 Gigabit Ethernet 16/25 Jelte Fennema

Introduction Approach Results Conclusion Initial performance tests 60 50 40 Gbit/s 30 20 10 0 Regular Epoll ANS iperf3 version Figure 5: Initial performance comparison Modifying existing applications for 100 Gigabit Ethernet 17/25 Jelte Fennema

Introduction Approach Results Conclusion Missing performance features ◮ TCP window scaling ◮ Jumbo frames are broken ◮ Offloading to the NIC Modifying existing applications for 100 Gigabit Ethernet 18/25 Jelte Fennema

Introduction Approach Results Conclusion Performance with more streams 60 50 40 Total Gbit/s iperf3 version 30 Regular Epoll 20 ANS 10 0 1 2 3 4 5 Number of streams Figure 6: Performance with multiple TCP streams Modifying existing applications for 100 Gigabit Ethernet 19/25 Jelte Fennema

Introduction Approach Results Conclusion Full impact of missing features 60 50 40 Total Gbit/s iperf3 version 30 Regular Epoll 20 ANS 10 0 1 2 3 4 5 Number of streams Figure 7: Performance comparison without missing ANS features Modifying existing applications for 100 Gigabit Ethernet 20/25 Jelte Fennema

Introduction Approach Results Conclusion Final weird result ◮ Multiple streams improve single stream performance 60 50 40 Total Gbit/s iperf3 version 30 Regular Epoll 20 ANS 10 0 1 2 3 4 5 Number of streams Figure 8: Performance comparison without missing ANS features Modifying existing applications for 100 Gigabit Ethernet 21/25 Jelte Fennema

Introduction Approach Results Conclusion Modified transmit buffer length 14 12 10 8 Gbit/s 6 4 2 0 Regular Epoll ANS New ANS iperf3 version Figure 9: A single TCP stream with performance features disabled Modifying existing applications for 100 Gigabit Ethernet 22/25 Jelte Fennema

Introduction Approach Results Conclusion Modified transmit buffer length 60 50 40 Total Gbit/s iperf3 version Regular 30 Epoll ANS 20 New ANS 10 0 1 2 3 4 5 Number of streams Figure 10: Multiple TCP streams with performance features disabled Modifying existing applications for 100 Gigabit Ethernet 23/25 Jelte Fennema

Introduction Approach Results Conclusion Conclusion ◮ Pktgen was not able to fully fill the 100GbE link ◮ But it was much faster than iperf3 Modifying existing applications for 100 Gigabit Ethernet 24/25 Jelte Fennema

Introduction Approach Results Conclusion Conclusion ◮ Pktgen was not able to fully fill the 100GbE link ◮ But it was much faster than iperf3 ◮ Modifying existing applications for DPDK is relatively easy by using ANS Modifying existing applications for 100 Gigabit Ethernet 24/25 Jelte Fennema

Introduction Approach Results Conclusion Conclusion ◮ Pktgen was not able to fully fill the 100GbE link ◮ But it was much faster than iperf3 ◮ Modifying existing applications for DPDK is relatively easy by using ANS ◮ iperf3 speeds with ANS are currently slower than with Linux Modifying existing applications for 100 Gigabit Ethernet 24/25 Jelte Fennema

Introduction Approach Results Conclusion Conclusion ◮ Pktgen was not able to fully fill the 100GbE link ◮ But it was much faster than iperf3 ◮ Modifying existing applications for DPDK is relatively easy by using ANS ◮ iperf3 speeds with ANS are currently slower than with Linux ◮ When missing ANS features are disabled for Linux ANS is faster Modifying existing applications for 100 Gigabit Ethernet 24/25 Jelte Fennema

Introduction Approach Results Conclusion Conclusion ◮ Pktgen was not able to fully fill the 100GbE link ◮ But it was much faster than iperf3 ◮ Modifying existing applications for DPDK is relatively easy by using ANS ◮ iperf3 speeds with ANS are currently slower than with Linux ◮ When missing ANS features are disabled for Linux ANS is faster ◮ For multiple streams using multiple cores is probably easier Modifying existing applications for 100 Gigabit Ethernet 24/25 Jelte Fennema

Introduction Approach Results Conclusion Future work ◮ Compare iperf3 performance after features have been implemented in ANS ◮ Investigate performance of Moongen on 100GbE Modifying existing applications for 100 Gigabit Ethernet 25/25 Jelte Fennema