Rekindling Network Protocol Innovation with User-Level Stacks Felip - PowerPoint PPT Presentation

Rekindling Network Protocol Innovation with User-Level Stacks Felip Felipe e Huici ici (NE (NEC Europ rope) e) Michio Honda (NetApp), Joao Taveira Araujo (UCL), Luigi Rizzo (Pisa University), Costin Raiciu (Universitea Buchalest) https://github.com/cnplab/multistack 1

Motivation • Extending layer 4 functionality could address a lot of problems – Increased performance • MPTCP , WindowScale, FastOpen, TLP , PRR – Ubiquitous encryption • TcpCrypt 2

Motivation • Extending layer 4 functionality could address a lot of problems – Increased performance • MPTCP , WindowScale, FastOpen, TLP , PRR – Ubiquitous encryption • TcpCrypt Is it really possible to deploy layer 4 extensions? 3

Motivation • Extending layer 4 functionality could address a lot of problems – Increased performance • MPTCP , WindowScale, FastOpen, TLP , PRR – Ubiquitous encryption • TcpCrypt Is it really possible to deploy layer 4 extensions? • Networks still accommodate TCP extensions – 86 % of the paths are usable for well-designed TCP extensions despite middleboxes 4

Protocol Stacks in End Systems: � The Theory • OSes implement stacks – High performance – Isolation between applications – Socket APIs • New OS versions adopt new protocols/extensions 5

Extending Protocol Stacks: � The Reality • OSes’ release cycle is slow • Support in the newest OS version does not imply deployment – Stakeholders are reluctant to upgrade their OS – Often new features, even if available, must be explicitly enabled 6

How Long does Deployment Take? 1.00 Option SACK 0.75 Ratio of flows Timestamp Windowscale 0.50 Direction Inbound 0.25 Outbound * Traffic trace from single 0.00 transit link in Japan (MAWI) 2007 2008 2009 2010 2011 2012 Date – Wind ndows : SACK is default since Windows 2000. WS and TS implemented in Windows 2000 but enabled as default since Windows Vista (2009) – Li Linux nux : SACK/TS on by default since 1999, WS since 2004 7

• To ease upgrade, we need to move protocol stacks up into user-space 8

• To ease upgrade, we need to move protocol stacks up into user-space • Problem: no practical way to do this, we need: – Isolation between applications – Support for legacy applications and the OS’s stack – High performance 9

MultiStack: Operating System Support for User-space Stacks (TCP port 22) (TCP port 80) (UDP port 53) User legacy apps . . . App 1 App N Kernel Socket API Stack 1 Stack N OS's stack Netmap API . . . Virtual ports Multiplex / Demultiplex packets (3-tuple) MultiStack NIC 10

MultiStack: Operating System Support for User-space Stacks (TCP port 22) (TCP port 80) (UDP port 53) User legacy apps . . . App 1 App N Kernel Socket API Stack 1 Stack N OS's stack Netmap API . . . Virtual ports Multiplex / Demultiplex packets (3-tuple) MultiStack NIC Apps/stacks register desired 3-tuple with the MulBStack kernel module 11

MultiStack: operating system support for user-space stacks (TCP port 22) (TCP port 80) (UDP port 53) User legacy apps . . . App 1 App N Kernel Socket API Stack 1 Stack N OS's stack Netmap API . . . Virtual ports Multiplex / Demultiplex packets (3-tuple) MultiStack NIC • Support for multiple stacks (including OS’s stack) • Namespace isolation based on traditional 3-tuple • Very high performance • Runs on FreeBSD and Linux (and it’s open source!) 12

MultiStack: operating system support for user-space stacks (TCP port 22) (TCP port 80) (UDP port 53) User legacy apps . . . App 1 App N Kernel Socket API Stack 1 Stack N OS's stack Netmap API . . . Virtual ports Multiplex / Demultiplex packets (3-tuple) MultiStack NIC • Support for multiple stacks (including OS’s stack) • Namespace isolation based on traditional 3-tuple • Very high performance • Runs on FreeBSD and Linux (and it’s open source!) 13

MultiStack: operating system support for user-space stacks (TCP port 22) (TCP port 80) (UDP port 53) User legacy apps . . . App 1 App N Kernel Socket API Stack 1 Stack N OS's stack Netmap API . . . Virtual ports Multiplex / Demultiplex packets (3-tuple) MultiStack NIC • Support for multiple stacks (including OS’s stack) • Namespace isolation based on traditional 3-tuple • Very high performance • Runs on FreeBSD and Linux (and it’s open source!) 14

MultiStack: operating system support for user-space stacks (TCP port 22) (TCP port 80) (UDP port 53) User legacy apps . . . App 1 App N Kernel Socket API Stack 1 Stack N OS's stack Netmap API . . . Virtual ports Multiplex / Demultiplex packets (3-tuple) MultiStack NIC • Support for multiple stacks (including OS’s stack) • Namespace isolation based on traditional 3-tuple • Very high performance • Runs on FreeBSD and Linux (and it’s open source!) 15

Multistack Base Performance (Tx) pktgen stack . . . . . . . . . core 0…n 3-tuple filter . . . NIC 16

Multistack Base Performance (Tx) pktgen stack . . . . . . . . . core 0…n 3-tuple filter . . . NIC • App creates every packet from scratch, and sends it to the kernel • Multistack validates the source 3-tuple of every packet, and copies the packet to the NIC’s TX buffer 17

Multistack Base Performance (Tx) pktgen stack . . . Throughput (Gbps) . . . 10 . . . core 0…n 8 1 core 6 2 cores 3-tuple filter 4 4 cores . . . 2 Line rate NIC 0 64 128 256 512 • App creates every packet from Packet size (bytes) scratch, and sends it to the kernel • Multistack validates the source 3-tuple of every packet, and copies the packet to the NIC’s TX buffer 18

Multistack Base Performance (Rx) pktrx stack . . . . . . . . . core 0…n 3-tuple mux . . . NIC 19

Multistack Base Performance (Rx) pktrx stack . . . . . . . . . core 0…n 3-tuple mux . . . NIC • Multistack receives a packet • It identifies destination 3-tuple of the packet • It delivers the packet to the corresponding app/stack 20

Multistack Base Performance (Rx) pktrx stack Throughput (Gbps) . . . 10 . . . 8 . . . core 0…n 1 core 6 2 cores 4 4 cores 3-tuple mux 2 Line rate . . . 0 NIC 64 128 256 512 Packet size (bytes) • Multistack receives a packet • It identifies destination 3-tuple of the packet • It delivers the packet to the corresponding app/stack 21

Many Apps/Stacks (Tx) pktgen pktgen pktgen stack stack stack … . . . . . . . . . 3-tuple filter . . . NIC 22

Many Apps/Stacks (Tx) pktgen pktgen pktgen stack stack stack … . . . . . . . . . 3-tuple filter . . . NIC • App creates every packet from scratch, and sends it to the kernel • Multistack validates the source 3-tuple of every packet, and copies the packet to the NIC’s TX buffer 23

Many Apps/Stacks (Tx) pktgen pktgen pktgen stack stack stack … Throughput (Gbps) 10 . . . . . . . . . 8 8 ports 6 16 ports 4 3-tuple filter 64 ports . . . 2 Line rate 0 NIC 64 128 256 512 Packet size (bytes) • App creates every packet from scratch, and sends it to the kernel • Multistack validates the source 3-tuple of every packet, and copies the packet to the NIC’s TX buffer 24

Many Apps/Stacks (Rx) pktrx pktrx pktrx stack stack stack … . . . . . . . . . 3-tuple mux . . . NIC 25

Many Apps/Stacks (Rx) pktrx pktrx pktrx stack stack stack … . . . . . . . . . 3-tuple mux . . . NIC • Multistack receives a packet • It identifies destination 3-tuple of the packet • It delivers the packet to the corresponding app/stack 26

Many Apps/Stacks (Rx) pktrx pktrx pktrx stack stack stack … Throughput (Gbps) 10 . . . . . . . . . 8 8 ports 6 16 ports 4 3-tuple mux 64 ports 2 . . . Line rate 0 NIC 64 128 256 512 Packet size (bytes) • Multistack receives a packet • It identifies destination 3-tuple of the packet • It delivers the packet to the corresponding app/stack 27

Many Apps/Stacks (Rx) pktrx pktrx pktrx stack stack stack … Throughput (Gbps) 10 . . . . . . . . . 8 8 ports 6 16 ports 4 3-tuple mux 64 ports 2 . . . Line rate 0 NIC 64 128 256 512 Packet size (bytes) • Multistack receives a packet • It identifies destination 3-tuple • A bit lower performance on many of the packet ports is due to the reduced number of packets taken in a • It delivers the packet to the single systemcall corresponding app/stack 28

Multistack Performance Summary • 4 Gbps for 64 byte packets with a single CPU core • 10 Gbps for 64 byte packets with two CPU cores • 10 Gbps for 256 byte packets with a single CPU core 29

Performance with User-Level Stacks • A simple HTTP server on top of our work-in-progress user-space TCP (UTCP) • The same app running on top of OS’s TCP 10 Client establishes a nginx-TSO OSTCP-TSO TCP connection, and 8 OSTCP UTCP sends HTTP GET 6 Server replies with Gbps HTTP OK (1- 32KB) 4 Single TCP connection is used for a single 2 HTTP transaction 0 1 8 16 32 Fetch size (KB) 30

Conclusion • Multi-stack: OS support for user-space stacks to rekindle widespread, timely deployment of new protocols/extensions Try it out! h/ps://github.com/cnplab/mul:stack Rekindling Network Protocol Innova4on with User-Level Stacks. SIGCOMM CCR 31