NOVA Microhypervisor on ARMv8-A FOSDEM 2020 Udo Steinberg BedRock - PowerPoint PPT Presentation

NOVA Microhypervisor on ARMv8-A FOSDEM 2020 Udo Steinberg BedRock Systems, Inc. February 2, 2020 Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 1 / 17

Outline NOVA Microhypervisor 1 ARMv8-A Virtualization 2 Current Status, Demo, Roadmap 3 Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 2 / 17

NOVA: System Architecture – x86 Guest VM Guest VM Guest VM Guest VM Guest VM Ring3 (G) App App App App App App App App App App OS Kernel OS Kernel OS Kernel OS Kernel OS Kernel Ring0 (G) VMM VMM VMM VMM VMM Ring3 (H) App Partition Manager App Host Ring0 (H) NOVA Microhypervisor x86 Hardware The microhypervisor is the only privileged component 1 Every virtual machine has its own VMM instance 1Ignoring SMM and Firmware, which are beyond our control Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 3 / 17

NOVA: System Architecture – ARMv8-A non-secure secure Guest VM Guest VM TEE App VMM VMM EL0 Partition Manager App App App App Host OS Kernel OS Kernel EL1 NOVA Microhypervisor EL2 Monitor EL3 ARM Hardware The microhypervisor is the only privileged component 1 Every virtual machine has its own VMM instance 1Ignoring TF-A, Monitor and TEE, which are beyond our control Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 4 / 17

NOVA: Capability-Based Access Control Protection Domain A Protection Domain B Selector Selector 2 PD Capability 6 0 0 PT 1 1 PT Capability 2 2 PT Capability PT 3 3 EC Capability 4 4 SC Capability 5 5 EC PT Capability 6 6 PD Capability PT Capability 7 7 Object Space Object Space SC PD A PD B Capability is pointer to KObject or PFrame + permissions Protection Domain has Object Space, Memory Space, ... Hypercall ctrl pd with take/grant semantics replaces MDB Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 5 / 17

NOVA: Basic Abstractions PD A PD B PD A PD B SC SC EC caller EC callee EC caller EC callee UTCB UTCB UTCB UTCB ipc call (P, MTD); ipc reply (MTD); P NOVA Microhypervisor NOVA Microhypervisor Protection Domains, Execution+Scheduling Contexts, Portals Semaphores for Synchronization and Interrupt Delivery Hypercall interface uses capabilities for all operations Synchronous IPC with timeslice donation ⇒ priority inheritance MTD defines number of words to copy UTCB caller ⇄ UTCB callee Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 6 / 17

NOVA: Handling VM Exits / Exceptions VM VMM VM VMM SC SC vCPU EC handler vCPU EC handler UTCB UTCB VMEXIT/EXCEPTION VMRESUME/ERET ipc reply (MTDARCH); P call (P, MTDARCH); VMCB VMCB NOVA Microhypervisor NOVA Microhypervisor vCPU state saved to / restored from VMCB Microhypervisor synthesizes IPC call on behalf of vCPU Destination portal selected based on type of event IPC reply from VMM provides updated architectural state MTD ARCH defines state to copy VMCB ⇄ UTCB handler Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 7 / 17

CPU Virtualization: Architectural State ARMv8-A Message Transfer Descriptor (MTD ARCH ) A32 DACR IFSR EL2 ELR SPSR EL1 ELR SPSR EL2 ESR FAR EL1 ESR FAR EL2 HPFAR EL1 SCTLR EL1 VBAR EL1 MAIR EL1 TTBR EL1 AFSR A32 SPSR EL2 HCR EL2 IDR EL1 TCR EL1 IDR EL0 IDR POISON EL1 SP EL0 SP GIC TMR FPR GPR 31 30 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 8 7 4 3 2 1 0 x86 Message Transfer Descriptor (MTD ARCH ) GPR 8 − 15 CS/SS FS/GS DS/ES FLAGS GPR 4 − 7 GPR 0 − 3 CTRL QUAL IDTR GDTR LDTR FPU TSC STA INJ LBR DR CR TR IP 31 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 For every bit set to 1, the corresponding architectural state is transmitted from the vCPU to the VMM handler or vice versa. Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 8 / 17

ARM: FPU Virtualization Hypervisor context-switches FPU state (32x128bit SIMD registers) between ECs lazily FPU Access Disabling/Enabling Switch away from FPU owner ⇒ disable FPU Switch back to FPU owner ⇒ enable FPU FPU switch moved out of critical IPC path using hazard tricks CPU Hazard Bit EC Hazard Bit FPU is disabled (0) EC is not FPU owner (0) FPU is enabled (1) EC is FPU owner (1) Slow path taken only if CPU Hazard ⊕ EC Hazard is 1 FPU use must be explicitly declared during EC creation Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 9 / 17

ARM: Interrupt Virtualization SPI Distributor GICD PPI Redistributor Redistributor Redistributor Redistributor GICR GICH Hyp IF VMM inject deactivate pINTID vINTID → pINTID vINTID GICC GICV pCPU IF vCPU IF Guest VM program pIRQ EOI vIRQ EOI Processing Element Unified interrupt injection interface for GICv2/GICv3 Hypercall assign int for configuring and routing SPIs Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 10 / 17

ARM: Timer Virtualization System Counter System Time Distribution managed by Microhypervisor – CNTVOFF CNTPCT CNTVCT Physical Timer Virtual Timer Real system counter System counter - offset Can be trapped Cannot be trapped ⇒ Trap & emulate timer ⇒ Context-switch timer pTimer interrupt emulated vTimer interrupt temporarily with semaphore timeouts belongs to current VM ⇒ Asynchronous delivery ⇒ Synchronous via Portal Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 11 / 17

ARM: System MMU Configuration Translation SMG VMID Miss CTX 0x860 PTAB TTBR SID Fill SMG TLB 0x404 VMID Miss CTX VMID:VA ⇒ PA:PERM PTAB TTBR SMG VMID:VA ⇒ PA:PERM Fill VA 0x405 VMID:VA ⇒ PA:PERM PA Stream Mapping Groups Translation Contexts Translation Lookaside Buffer DMA Page Tables System MMU protects against rogue DMA Limited number of stream mapping groups and translation contexts managed by partition manager Hypercall assign dev for configuring SID/SMG/CTX and binding a device to a protection domain Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 12 / 17

Currently Supported ARM Platforms Avnet Xilinx Ultra 96 NXP i.MX 8MQuad 4x Cortex-A53 4x Cortex-A53 GICv2 GICv3 SMMUv2 Renesas R-Car M3/H3 Raspberry Pi 4B 4x Cortex-A53 4x Cortex-A72 4x Cortex-A57 GICv2 GICv2 QEMU Virt Platform Cortex-A GICv2/GICv3 Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 13 / 17

Current Status: Demo VM VM VM Guest vCPU vCPU vCPU vCPU vUART vGIC vETH vUART vGIC vETH vUART vGIC vETH Host VMM VMM VMM UART Driver UART Multiplexer Virtual Ethernet Switch Partition Manager NOVA Microhypervisor UART Display SD USB WiFi Core Core Core Core Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 14 / 17

Roadmap Architecture Unification Merge significant portions of the x86 and ARMv8 source code { src/x86 64, src/aarch64 } ⇒ src { inc/x86 64, inc/aarch64 } ⇒ inc Support for newer ARM features (ARMv8.1 – ARMv8.6) Additional NOVA functionality Relocatable microhypervisor binary VM introspection support Improved kernel resource management Useful external features and bug fixes Performance Optimizations Formal Verification of the NOVA microhypervisor ... and of components running on top of it Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 15 / 17

Formal Verification /* * \arg{v1} "x" (Vint v1) * \arg{v2} "y" (Vint v2) * \pre empSP * \post{}[Vint (trim 32 (v1+v2))] empSP */ auto add_func (uint32 x, uint32 y) { return x + y; } C++ Semantics Source Code clang Plugin AST Coq Representation foo.cpp cpp2v foo cpp.v Machine-Checked Proof coqc foo cpp proof.vo Code Specifications foo cpp spec.v Proof Automation Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 16 / 17

Source code available under GPLv2 license at: https://github.com/bedrocksystems/NOVA https://github.com/udosteinberg/NOVA Checkout the ” arm ” branch. Further information (papers, links) at: https://bedrocksystems.com http://hypervisor.org Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 17 / 17