Networking in Biscuit Sebastian Voit, Paul Emmerich January 18, - PowerPoint PPT Presentation

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Networking in Biscuit Sebastian Voit, Paul Emmerich January 18, 2019 Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Biscuit • Biscuit is an operating system kernel written in Go • Evaluate the impact of high-level languages on low-level systems • We took a look at Biscuit’s network driver (ixgbe) Sebastian Voit, Paul Emmerich — Networking in Biscuit 2

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Overview 1. Golang summary 2. Kernel network drivers 3. Introduction to PCIe device programming 4. Biscuit’s ixgbe driver Sebastian Voit, Paul Emmerich — Networking in Biscuit 3

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Golang summary Yet another programming language: Go(lang) • Go is a compiled (statically linked binaries) programming language • Runtime with garbage collection, memory safe, concurrency • Chosen for biscuit for asm support, compilation to machine code, ... → look at preventable C bugs that cannot occur in memory safe languages Sebastian Voit, Paul Emmerich — Networking in Biscuit 4

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Golang summary Yet another programming language: Go(lang) • Go is a compiled (statically linked binaries) programming language • Runtime with garbage collection, memory safe, concurrency • Chosen for biscuit for asm support, compilation to machine code, ... → look at preventable C bugs that cannot occur in memory safe languages • Syntactically similar to C: package main import "fmt" import "time" func main() { for i := 0; i < 10; i++ { go func (num int){ fmt.Printf("counting up, currently at %v\n", num) }(i) } time.Sleep(1 * time.Second) } Sebastian Voit, Paul Emmerich — Networking in Biscuit 4

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Kernel network drivers • Enable communication between NIC and the rest of the system: • Hand received packets to network stack and • Expose sending functionality Sebastian Voit, Paul Emmerich — Networking in Biscuit 5

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Kernel network drivers • Enable communication between NIC and the rest of the system: • Hand received packets to network stack and • Expose sending functionality • NICs are PCIe devices Sebastian Voit, Paul Emmerich — Networking in Biscuit 5

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Kernel network drivers • Enable communication between NIC and the rest of the system: • Hand received packets to network stack and • Expose sending functionality • NICs are PCIe devices • How do we communicate with PCIe devices? Sebastian Voit, Paul Emmerich — Networking in Biscuit 5

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Introduction to PCIe device programming 1. Allocate shared memory Sebastian Voit, Paul Emmerich — Networking in Biscuit 6

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Introduction to PCIe device programming 1. Allocate shared memory 2. Write driver Sebastian Voit, Paul Emmerich — Networking in Biscuit 6

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Introduction to PCIe device programming 1. Allocate shared memory 2. Write driver 3. ??? Sebastian Voit, Paul Emmerich — Networking in Biscuit 6

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Introduction to PCIe device programming 1. Allocate shared memory 2. Write driver 3. ??? 4. Profit Sebastian Voit, Paul Emmerich — Networking in Biscuit 6

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Introduction to PCIe device programming How it’s actually done: Sebastian Voit, Paul Emmerich — Networking in Biscuit 7

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Introduction to PCIe device programming How it’s actually done: • Communicate with PCIe devices via MMIO (Memory Mapped I/O): • Allocate DMA memory for BAR (Base Address Register) → Reading from/writing to memory area reads from/writes to the device • Datasheet describes registers, offsets, values, etc. Sebastian Voit, Paul Emmerich — Networking in Biscuit 7

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Introduction to PCIe device programming How it’s actually done: • Communicate with PCIe devices via MMIO (Memory Mapped I/O): • Allocate DMA memory for BAR (Base Address Register) → Reading from/writing to memory area reads from/writes to the device • Datasheet describes registers, offsets, values, etc. • In user space we would use mmap() • In kernel space the memory is controlled directly • Not the driver’s responsibility though Sebastian Voit, Paul Emmerich — Networking in Biscuit 7

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Introduction to PCIe device programming How to communicate with the NIC: • Use the shared memory • Datasheet tells us how: 1. Figure out what to do 2. Look up register offset with control values up in the datesheet 3. Read+set registers accordingly • Example: Blinking LED Sebastian Voit, Paul Emmerich — Networking in Biscuit 8

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Example: LEDs const ( LEDCTL = 0x00200 LED0_BLINK_OFFS = 7 ) leds := atomic.LoadUint32(&bar0[LEDCTL]) //compiler barrier atomic.StoreUint32(&bar0[LEDCTL], leds | (1 << LED0_BLINK_OFFS)) Sebastian Voit, Paul Emmerich — Networking in Biscuit 11

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Example: LEDs const ( LEDCTL = 0x00200 LED0_BLINK_OFFS = 7 ) leds := atomic.LoadUint32(&bar0[LEDCTL]) //compiler barrier atomic.StoreUint32(&bar0[LEDCTL], leds | (1 << LED0_BLINK_OFFS)) • Define constants according to datasheet • Register write and store: compiler barrier! (volatile in C, atomic.Load/Store in go) • Change value in the correct register to turn LEDs on/off Sebastian Voit, Paul Emmerich — Networking in Biscuit 11

Chair of Network Architectures and Services Department of Informatics Technical University of Munich More info In case you want more info on PCIe programming/NIC drivers/biscuit/etc. • Watch our talk at 35c3: Safe and Secure Drivers in High-Level Languages 1 • Visit our ixy-languages repository (many more implementations then just go) 2 • Read the seminar paper • Look at biscuit’s ixgbe driver 3 1 https://media.ccc.de/v/35c3-9670-safe_and_secure_drivers_in_high-level_languages 2 https://github.com/ixy-languages/ixy-languages 3 https://github.com/mit-pdos/biscuit/blob/master/biscuit/src/ixgbe/ixgbe.go Sebastian Voit, Paul Emmerich — Networking in Biscuit 12

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Biscuit’s ixgbe driver First of all, what is an ”ixgbe”? Sebastian Voit, Paul Emmerich — Networking in Biscuit 13

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Biscuit’s ixgbe driver First of all, what is an ”ixgbe”? • Intel ixgbe family: 82599ES (aka X520), X540, X550, Xeon D embedded NIC • Very common with good datasheet (as seen before) • Contrary to other NICs: • No message exchange, direct access • Hardware features directly exposed instead of black-box NIC Sebastian Voit, Paul Emmerich — Networking in Biscuit 13

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Biscuit’s ixgbe driver Now how does biscuit’s driver work? 1. Initialize and set up the device Sebastian Voit, Paul Emmerich — Networking in Biscuit 14

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Biscuit’s ixgbe driver Now how does biscuit’s driver work? 1. Initialize and set up the device 2. Forever: • Hand received packets over to network stack (interrupt on incoming packets) • Instruct NIC to send packets (exposed sending methods) Sebastian Voit, Paul Emmerich — Networking in Biscuit 14