Kevin Laatz & Ciara Loftus FOSDEM 2020
Introduction https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2 Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036
Introduction • Userspace Libraries and Drivers. • Accelerate packet processing workloads. • Runs on wide variety of CPU architectures. • Has its own memory management subsystem. • Device specific PMDs (Poll Mode Drivers). https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2 Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036
Introduction AF_XDP • Userspace Libraries and Drivers. • Kernel based address family. • Accelerate packet processing workloads. • Optimized for high performance packet processing. • Runs on wide variety of CPU architectures. • AF_XDP sockets redirect packets to userspace. • Has its own memory management subsystem. • By-passes kernel network stack • Device specific PMDs (Poll Mode Drivers). • “In - kernel fast path”. https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2 Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036
Introduction AF_XDP • Userspace Libraries and Drivers. • Kernel based address family. • Accelerate packet processing workloads. • Optimized for high performance packet processing. • Runs on wide variety of CPU architectures. • AF_XDP sockets redirect packets to userspace. • Has its own memory management subsystem. • By-passes kernel network stack • Device specific PMDs (Poll Mode Drivers). • “In - kernel fast path”. Traditional DPDK Model Userspace DPDK apps i40e ixgbe Other PMD PMD PMDs Kernel space igb_uio vfio https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2 Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036
Introduction AF_XDP • Userspace Libraries and Drivers. • Kernel based address family. • Accelerate packet processing workloads. • Optimized for high performance packet processing. • Runs on wide variety of CPU architectures. • AF_XDP sockets redirect packets to userspace. • Has its own memory management subsystem. • By-passes kernel network stack • Device specific PMDs (Poll Mode Drivers). • “In - kernel fast path”. DPDK with AF_XDP Traditional DPDK Model Userspace Userspace DPDK apps DPDK apps DPDK AF_XDP i40e ixgbe Other PMD PMD PMD PMDs Kernel space Kernel space NIC driver igb_uio vfio https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2 Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036
Problem Statement
Problem Statement The Goal: • All DPDK applications should be able to run out-of-the-box with the AF_XDP PMD. • DPDK app + AF_XDP PMD should run with good performance. •
Problem Statement The Goal: • All DPDK applications should be able to run out-of-the-box with the AF_XDP PMD. • DPDK app + AF_XDP PMD should run with good performance. • The Challenge: • Frameworks like DPDK have their own memory management which come with their own • assumptions and constraints.
Problem Statement The Goal: • All DPDK applications should be able to run out-of-the-box with the AF_XDP PMD. • DPDK app + AF_XDP PMD should run with good performance. • The Challenge: • Frameworks like DPDK have their own memory management which come with their own • assumptions and constraints. Discrepancy between the DPDK and AF_XDP buffer alignment. • Prevents direct mapping of DPDK mempool to AF_XDP UMEM. • Extra complexity/work negatively impacting performance. •
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 Chunk 0 Chunk 1 Page 1 Chunk 2 Chunk 3 Page 2
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 Chunk 0 • Area of memory allocated by the user Chunk 1 for packet data Page 1 Chunk 2 Chunk 3 Page 2
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 Chunk 0 • Area of memory allocated by the user Chunk 1 for packet data Page 1 Chunk 2 • Split up into equal Chunk 3 sized-chunks Page 2
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 Chunk 0 • Area of memory allocated by the user Chunk 1 for packet data Page 1 Chunk 2 • Split up into equal Chunk 3 sized-chunks Page 2 • RX Path: Kernel places packet data in a chunk for userspace (DPDK) to retrieve
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 Chunk 0 • Area of memory allocated by the user Chunk 1 for packet data Page 1 Chunk 2 • Split up into equal Chunk 3 sized-chunks Page 2 • RX Path: Kernel places packet data in a chunk for userspace (DPDK) to retrieve • TX Path: Userspace places packet data in chunk for kernel NIC driver to transmit
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 Chunk 0 Chunk 1 Page 1 Chunk 2 Chunk 3 Page 2 *Prior to Kernel 5.4
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 PAGE_SIZE = 4K Chunk 0 Start address must be Chunk 1 PAGE_SIZE aligned Page 1 Chunk 2 Chunk 3 Page 2 *Prior to Kernel 5.4
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 PAGE_SIZE = 4K Chunk 0 Start address must be Chunk 1 PAGE_SIZE aligned Page 1 Chunk 2 Chunks must be ^2 Chunk 3 sized Page 2 *Prior to Kernel 5.4
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 PAGE_SIZE = 4K Chunk 0 Start address must be Chunk 1 PAGE_SIZE aligned Page 1 Chunk 2 Chunks must be ^2 Chunk 3 sized Page 2 Chunks may not cross page boundaries *Prior to Kernel 5.4
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 PAGE_SIZE = 4K Chunk 0 DPDK MBUF 0 Start address must be Chunk 1 PAGE_SIZE aligned Page 1 DPDK Chunk 2 MBUF 1 Chunks must be ^2 Chunk 3 sized DPDK Page 2 MBUF 2 Chunks may not cross DPDK page boundaries MBUF 3 Page 3 *Prior to Kernel 5.4
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 PAGE_SIZE = 4K Chunk 0 DPDK MBUF 0 Start address must be Mbufs can be any Chunk 1 PAGE_SIZE aligned size Page 1 DPDK Chunk 2 MBUF 1 Chunks must be ^2 Chunk 3 sized DPDK Page 2 MBUF 2 Chunks may not cross DPDK page boundaries MBUF 3 Page 3 *Prior to Kernel 5.4
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 PAGE_SIZE = 4K Chunk 0 DPDK MBUF 0 Start address must be Mbufs can be any Chunk 1 PAGE_SIZE aligned size Page 1 DPDK Chunk 2 MBUF 1 Chunks must be ^2 Mbufs can have Chunk 3 sized arbitrary DPDK Page 2 MBUF 2 alignment Chunks may not cross DPDK page boundaries MBUF 3 Page 3 *Prior to Kernel 5.4
AF_XDP UMEM* vs DPDK Mbuf Pool Page 0 PAGE_SIZE = 4K Chunk 0 DPDK MBUF 0 Start address must be Mbufs can be any Chunk 1 PAGE_SIZE aligned size Page 1 DPDK Chunk 2 MBUF 1 Chunks must be ^2 Mbufs can have Chunk 3 sized arbitrary DPDK Page 2 MBUF 2 alignment Chunks may not cross DPDK page boundaries MBUF 3 Page 3 *Prior to To get the highest performing integration of AF_XDP and DPDK, the DPDK Kernel 5.4 mbuf pool must be mapped into the UMEM for a zero copy datapath.
DPDK Solutions
DPDK Solutions Copy Mode UMEM CHUNK DPDK MBUF pkt memcpy pkt • Memcpy packets between UMEM & mbufpool • Cycle-heavy memcpy • DPDK 19.05
DPDK Solutions Copy Mode Alignment API UMEM UMEM DPDK CHUNK CHUNK MBUF DPDK MBUF pkt pkt pkt memcpy pkt • Memcpy packets between • API to change mbuf pool UMEM & mbufpool alignment, then 1:1 map • Cycle-heavy memcpy • Invasive change • DPDK 19.05 • No DPDK release
DPDK Solutions Copy Mode Alignment API External Mbuf UMEM UMEM UMEM DPDK CHUNK CHUNK MBUF CHUNK DPDK DPDK MBUF MBUF pkt pkt pkt pkt * memcpy pkt *pkt • Memcpy packets between • API to change mbuf pool • Mbuf points to packet UMEM & mbufpool alignment, then 1:1 map data in UMEM chunk • Cycle-heavy memcpy • Invasive change • Additional complexity • DPDK 19.05 • No DPDK release • DPDK 19.08
New Solution: Kernel Arbitrary Alignment
New Solution: Kernel Arbitrary Alignment Page 0 PAGE_SIZE = 4K Chunk 0 Chunk 1 Page 1 Chunk 2 Main benefits of the new solution: Enables arbitrary chunk alignment. • Chunk 3 Page 2 Page 3
Recommend
More recommend
