Nested Virtualization on ARM NEVE: Nested Virtualization Extensions Jin Tack Lim Christoffer Dall Shih-Wei Li Jason Nieh Marc Zyngier LEADING jitack@cs.columbia.edu christo ff er.dall@linaro.org shih-wei@cs.columbia.edu, COLLABORATION nieh@cs.columbia.edu marc.zyngier@arm.com IN THE ARM ECOSYSTEM connect.linaro.org
Nested Virtualization VM VM VM App App App App VM Kernel Kernel App App Hypervisor Kernel Hypervisor Hardware
Terminology VM Nested VM Nested VM App App App App VM Kernel Kernel App App Guest Hypervisor Kernel Host Hypervisor Hardware
Use Cases 1. Run guest operating systems with built-in virtualization. 2. IaaS hosting private clouds 3. Test your hypervisor in a VM 4. Debug your hypervisor in a VM 5. Develop hypervisors using a cloud
ARM Virtualization Extensions VM VM User Space User Space EL0 Kernel Kernel EL1 Hypervisor EL2
ARM Nested Virtualization VM VM User Space User Space EL0 Kernel Kernel EL1 Virtual Guest Hypervisor Guest Hypervisor EL2 Host Hypervisor EL2
ARM Nested Virtualization VM VM User Space User Space EL0 Kernel Kernel EL1 Guest Hypervisor EL ?? Guest Hypervisor Host Hypervisor EL2
ARMv8.0 Nested Virtualization VM VM User Space User Space EL0 Kernel Kernel EL1 Guest Hypervisor EL0 Guest Hypervisor Trap-and-emulate Host Hypervisor EL2
ARMv8.0 Nested Virtualization VM VM User Space User Space EL0 Kernel Kernel EL1 Guest Hypervisor EL1 Guest Hypervisor ?? -and-emulate Host Hypervisor EL2
ARMv8.3 Nested Virtualization • Gives you software emulation of VM vEL2 in EL1 User Space EL0 • HCR_EL2.NV: Kernel EL1 • Traps EL2 operations executed in EL1 to EL2 Guest Hypervisor EL1 • Traps eret to EL2 Trap -and-emulate Host Hypervisor EL2 • CurrentEL reports EL2 even in EL1
KVM/ARM Nested Virtualization Implementation • EL2 Emulation • Stage 2 MMU Virtualization • Hyp Timer Virtualization • Nested Virtual Interrupts
Nested CPU Virtualization struct kvm_cpu_context { u64 sys_regs[NR_SYS_REGS]; + u64 el2_regs[NR_EL2_REGS]; } struct kvm_vcpu_arch { … struct kvm_cpu_context ctxt; }
Hypervisor-VM Switch Host VM App App App App EL0 Kernel EL1 Save EL1 sys_regs Linux KVM EL2 Restore EL1 sys_regs
Hypervisor-Hypervisor Switch Host VM App App EL0 Kernel EL1 Guest Hypervisor Save/restore EL1 sys_regs Linux KVM EL2 Save/restore el2_regs
Emulating EL2 in EL1 • Define mapping of EL2 registers to EL1 registers • Example: TTBR0_EL2 to TTBR0_EL1 • Example: SCTLR_EL2 adapted to SCTLR_EL1 • Shadow EL1 registers
Nested CPU Virtualization struct kvm_cpu_context { u64 sys_regs[NR_SYS_REGS]; + u64 el2_regs[NR_EL2_REGS]; + u64 shaow_sys_regs[NR_SYS_REGS]; } struct kvm_vcpu_arch { … struct kvm_cpu_context ctxt; }
Shadow Registers &sys_regs &shadow_sys_regs PSTATE.mode == EL0/1 PSTATE.mode == EL2 u64 *vcpu->ctxtx.hw_regs
Virtual Exceptions VM EL0 User Space • Trap to virtual EL2 EL1 Kernel • “Forward” exceptions vEL2 Guest KVM • Emulate virtual exceptions Host KVM EL2
Virtual Exceptions VM EL0 User Space • Returning from virtual EL2 EL1 Kernel • Trap eret to EL2 (ARMv8.3) vEL2 Guest KVM • Emulate virtual exception return Host KVM EL2
KVM/ARM Nested Virtualization Implementation • EL2 Emulation • Stage 2 MMU Virtualization • Hyp Timer Virtualization • Nested Virtual Interrupts
Memory Virtualization EL0 User Space Stage 1: VA -> IPA Kernel EL1
Memory Virtualization VM EL0 User Space Stage 1: VA -> IPA Kernel EL1 Stage 2: IPA -> PA Host Hypervisor EL2
Memory Virtualization VM Nested VM EL0 User Space Stage 1: VA -> IPA Kernel EL1 ???? Guest Hypervisor Stage 2: IPA -> PA Host Hypervisor EL2
Memory Virtualization VM Nested VM EL0 User Space Stage 1: VA -> IPA Kernel EL1 Virtual stage 2 Guest Hypervisor Shadow Stage 2: IPA -> PA Host Hypervisor EL2
KVM/ARM Nested Virtualization Implementation • EL2 Emulation • Stage 2 MMU Virtualization • Hyp Timer Virtualization • Nested Virtual Interrupts
Nested Timer Virtualization • ARM provides a virtual and physical timer in EL1 • EL2 provides a separate EL2 “hyp” timer • Nested KVM/ARM supports a virtual CPU with EL2 and the hyp timer
KVM/ARM Nested Virtualization Implementation • EL2 Emulation • Stage 2 MMU Virtualization • Hyp Timer Virtualization • Nested Virtual Interrupts
ARM Generic Interrupt Controller (GIC) IRQ CPU Interface Device Interrupt Dist. CPU ACK/EOI Lines CPU Interface GIC
ARM Generic Interrupt Controller (GIC) List Registers (LRs) IRQ CPU Interface VIRQ Virtual CPU Interface Dist. CPU ACK/EOI CPU Interface ACK/EOI Virtual CPU Interface GIC List Registers (LRs)
Nested Interrupt Virtualization VM Nested VM User Space • Deliver virtual interrupts Virtual CPU Kernel from the host to the VM Interface Guest VMM LRs Host VMM
Nested Interrupt Virtualization VM • Deliver virtual interrupts Nested VM from the guest hypervisor User to the nested VM Space Virtual CPU • Shadow list registers Kernel Interface • The nested VM can ACK Guest VMM and EOI virtual interrupts LRs without trapping Host VMM
Performance Evaluation • Problem: No ARMv8.3 hardware available. • Solution: Use ARMv8.0 hardware with the software modification
Emulating v8.3 on v8.0 VM Nested VM App App App EL0 OS Kernel EL1 Guest Hypervisor HVCHVC HVC Paravirtualization HVC HVC Host Hypervisor EL2 ARMv8.0 Hardware
Hypercall MicroBenchmark VM Nested VM VM App App App EL0 App App App EL0 OS Kernel EL1 OS Kernel EL1 Hypercall Return Guest Hypervisor Hypercall Return Hypervisor EL2 Host Hypervisor EL2
Hypercall MicroBenchmark ARMv8.3 VM Nested VM 2,729 422,720 Cycle counts 1 155x Ratio to VM
Application Benchmarks Normalized overhead (lower is better) 50 ARMv8.3 VM 40 ARMv8.3 Nested 30 20 10 0 5 4 3 2 1 0 Kernbench Hackbench SPECjvm2008 TCP RR TCP STREAM TCP MAERTS Apache Nginx Memcached MySQL
Nested VM Exit/Entry on ARM VM Nested VM App App App EL0 OS Kernel EL1 VM Exit VM Entry EL1 Registers Guest Hypervisor … Host Hypervisor EL2 Registers EL2 > 120 traps
NEVE: NEsted Virtualization Extensions for ARM • Supports unmodified guest hypervisors and OSes • Improves performance by providing register redirection
Register Classification • VM registers: EL1 registers only a ff ecting the nested VM’s execution • Hypervisor registers: EL2 registers a ff ecting the hypervisor’s execution
VM Registers VM Nested VM App App App EL0 OS Kernel EL1 VM Exit VM Entry EL1 Registers Guest Hypervisor … Host Hypervisor EL2 This is when VM register states are used
VM Registers: Logging to Memory VM Register Trap! Without NEVE msr x0, TTBR0_EL1 Memory
VM Registers: Logging to Memory VM Register msr x0, TTBR0_EL1 Memory With NEVE TTBR0_EL1
Hypervisor control registers • Can’t apply the technique for VM registers • They have an immediate impact (EL2 system registers) • Traps are handled by redirecting to EL1 registers in software Guest Hypervisor EL1 Registers EL1 Host Hypervisor EL2 EL2 Registers
Hypervisor control registers • Can’t apply the technique for VM registers • They have an immediate impact (EL2 system registers) • Traps are handled by redirecting to EL1 registers in software • Redirect in hardware instead! Guest Hypervisor EL1 Registers EL1 Host Hypervisor EL2 EL2 Registers
Hypercall MicroBenchmark ARMv8.3 NEVE VM Nested VM Nested VM 92,385 2,729 422,720 Cycle counts 34x 155x Ratio to VM 15 1 126 Trap counts
Application Workloads Application Description Application Description Netperf TCP_RR Kernbench Kernel compile Network performance Hackbench Scheduler stress Netperf TCP STREAM Network performance SPECjvm2008 Java Runtime Netperf TCP MAERTS Network performance MySQL Database management Apache Web server stress Memcached Key-Value store Nginx Web server stress
Experimental Setup • Software • ARM Hardware • Native/VM/Nested VM • APM X-Gene (ARMv8.0) • KVM on KVM • 4-way SMP • 8-way SMP • 64 GB RAM • v4.10 • 12 GB RAM • x86 Hardware • Virt I/O • Intel E5-2630 v3 (VM/nested VM) • VMCS Shadowing • 10 Gb Ethernet • 8-way SMP • 128 GB RAM
Application Benchmarks Normalized overhead (lower is better) 50 ARMv8.3 VM 40 ARMv8.3 Nested NEVE Nested 30 20 10 0 5 4 3 2 1 0 Kernbench Hackbench SPECjvm2008 TCP RR TCP STREAM TCP MAERTS Apache Nginx Memcached MySQL
Application Benchmarks Normalized overhead (lower is better) 50 ARMv8.3 VM 40 ARMv8.3 Nested NEVE Nested 30 x86 Nested VM 20 10 0 5 4 3 2 1 0 Kernbench Hackbench SPECjvm2008 TCP RR TCP STREAM TCP MAERTS Apache Nginx Memcached MySQL
Recommend
More recommend
