Deduplication in VM Environments Frank Bellosa < bellosa@kit.edu > Konrad Miller < miller@kit.edu > Marc Rittinghaus < rittinghaus@kit.edu > K ARLSRUHE I NSTITUTE OF T ECHNOLOGY (KIT) - S YSTEM A RCHITECTURE G ROUP www.kit.edu KIT – Universit¨ at des Landes Baden-W¨ urttemberg und nationales Forschungszentrum in der Helmholtz-Gemeinschaft
Memory Sharing in VM Environments Operating Systems do a fairly good job in sharing, but there is still duplicate content in memory How much ? How long ? Wherefrom ? State of the art in deduplication KSM++: introducing hints for memory scanners Evaluation Challenge: NUMA and SCM 2/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Memory Sharing in VM Environments Operating Systems do a fairly good job in sharing, but there is still duplicate content in memory How much ? How long ? Wherefrom ? State of the art in deduplication KSM++: introducing hints for memory scanners Evaluation Challenge: NUMA and SCM 2/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Virtualization for Server Consolidation Past : One physical machine for each service Present : Multiple isolated virtual machines on a single physical host Improved hardware utilization Increased flexibility (placement/migration) Smaller hardware footprint Energy efficiency 3/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Virtualization for Server Consolidation Past : One physical machine for each service Present : Multiple isolated virtual machines on a single physical host Improved hardware utilization Increased flexibility (placement/migration) Smaller hardware footprint Energy efficiency 3/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Memory Duplication in VM Environments Main memory is the primary bottleneck when consolidating machines Different VMs often contain pages with equal content System Configuration Equal pages VMware ESX (VMware@OSDI’02) 10 VMs, SPEC95 65 % Difference Engine (UCSD@OSDI’08) 3 VMs, XP/Linux, RUBiS/LAMP 40 % – 85 % Satori (Cambridge@USENIX ATC’09) 2 VMs, Apache 66 % Satori (Cambridge@USENIX ATC’09) 2 VMs, Kernel build 11 % Chang et al (Taiwan Univ@ISPA’11) Hadoop, HOMP (MPI), LAMP 11 % – 86 % Barker et al (UMASS@USENIX ATC’12) offline comparison of 15 % desktop/server snapshots Goal: Merge pages, free memory for additional VMs 4/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Memory Deduplication for VMs Physical Hypervisor VM 1 Physical Hypervisor VM 1 Memory Memory libA.so libA.so Free Free kernel kernel Free Free Data A Data A ? Free ? Free VM 2 VM 2 ? ? ? kernel ? kernel ? libA.so ? libA.so ? Data B ? Data B Without deduplication : Every guest page maps to a different host page With deduplication : Pages with identical content are merged and shared between VMs through copy-on-write (COW) How can pages with equal content be identified? 5/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Memory Deduplication for VMs Physical Hypervisor VM 1 Physical Hypervisor VM 1 Memory Memory libA.so libA.so Free Free kernel kernel Free Free Data A Data A ? Free ? Free VM 2 VM 2 ? ? ? kernel ? kernel ? libA.so ? libA.so ? Data B ? Data B Without deduplication : Every guest page maps to a different host page With deduplication : Pages with identical content are merged and shared between VMs through copy-on-write (COW) How can pages with equal content be identified? 5/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Memory Deduplication for VMs Physical Hypervisor VM 1 Physical Hypervisor VM 1 Memory Memory libA.so libA.so Free Free kernel kernel Free Free Data A Data A ? Free ? Free VM 2 VM 2 ? ? ? kernel ? kernel ? libA.so ? libA.so ? Data B ? Data B Without deduplication : Every guest page maps to a different host page With deduplication : Pages with identical content are merged and shared between VMs through copy-on-write (COW) How can pages with equal content be identified? 5a/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Semantic Gap Traditional Sharing Mechanisms : Physical Hypervisor VM 1 Memory Based on source object, not on content libA.so Free fork() : parent process kernel Free Semantic Gap mmap() : equal inode Data A ? ? Virtualization introduces semantic gap VM 2 ? between guest and host ? kernel Source objects unknown to the host ? libA.so No semantic information about guest ? Data B pages Traditional sharing mechanisms cannot be used for deduplicating VMs 6/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Getting Around the Semantic Gap Memory scanners directly address page content Continuously catalog page content Physical Hypervisor Memory Random order (VMware ESX, OSDI’02) Scan Linear order (Linux’ KSM, Linux Symposium’09) pages Classify pages based on their modification ? Memory frequency ? Scanner ? “Has the page’s content changed since last visit?” ? Build index of infrequently modified pages ? ? Merge/mark COW equal pages that have been found through the index 7a/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Memory Scanners Pay memory density with CPU/memory bandwidth overhead Scan Rate Scan Time CPU Overhead 1000 pages 5 minutes ∼ 28 % Default second gigabyte 5000 pages minute ∼ 70 % Aggressive 1 second gigabyte Initial benchmarks: more than 70 % of mergable pages modified. . . . . . within a single scan round → not caught by scanner . . . late enough to amortize the merge cost Scan Interval Sharing Opportunity Time [m] 0 5 10 15 8b/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Closing the Semantic Gap Paravirtualization /Introspection closes the semantic gap Physical Hypervisor VM 1 Assumption: Memory Many deduplication candidates. . . libA.so Interface Free kernel . . . stem from Virtual Disk Image (VDI) Free Data A (programs, libraries, data) kernel . . . are copies from other data in the system libA.so VM 2 Data A kernel kernel Interface Transport information about duplication Data B libA.so from guests to host libB.so Data B Modify guests’ VDI driver (Satori, USENIX’09) Hook guests’ syscalls (Disco, SOSP’97) 9/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
State of the Art Memory scanners: Deduplicate sharing opportunities of any source Can catch sharing opportunities if they live long enough ( > 5 – 30 min) Paravirtualization based approaches: Deduplicate short and long-lived opportunities that stem from disk Process all I/O → Bottleneck for I/O-intensive workloads Take-away message: Memory scanners exploit sharing opportunities from all sources Deduplication schemes can be improved through semantic information Guests’ I/O pages are prime deduplication candidates 10/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Temporal Memory Duplication Characteristics 3 VMs: Ubuntu + Firefox + { LibreOffice, Gimp, Eclipse } in Simics 60,0% 54,5% E[t Visit ] 50,0% Equal Pages Cumulative Sharing E[t Visit ] 40,0% Deduplication Improvement E[t Visit ] 30,0% Potential E[t Visit ] 20,0% 15,0% t 13,0% 9,8% KSM 1 10,0% KSM++ 1 2 1 2 1 0,0% ≥ 1 sec ≥ 30 sec ≥ 5 min ≥ 30 min Sharing opportunities live. . . . . . extremely short → not worth sharing . . . between 1 sec – 30 sec → not caught by memory scanners . . . long → already caught by memory scanners Visiting sharing opportunities earlier leads to more deduplicated pages 11/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Semantic Memory Duplication Characteristics 3 VMs: Ubuntu + Firefox + { LibreOffice, Gimp, Eclipse } in Simics Reserved Other Memory Prop. of Slab Category Sharing Cache File 73.7 % Anonymous Heap 9.2 % Heap Anonymous 6.3 % Slab Cache 5.8 % Reserved 1 3.8 % Other 1.3 % File 1 Non-free pages not explicitly tracked by OS introspection (e.g., driver private pages) Barker et al.: 50 % Heap, 43 % File Kloster et al.: 64 % – 94 % File 12/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
KSM++: Hints for Memory Scanners Best of both worlds: Integrate I/O-based dedup into memory scanner Host/Hypervisor does I/O on behalf of guest VMs I/O-operations target guests’ buffer caches and mmap areas Record Host-VFS target memory areas in a “Hints Buffer” Visit I/O-pages earlier in memory scanner No paravirtualization required guest-agnostic App also works for native apps App Guest VFS Read (e.g., Zero Install) Guest OS Guest OS VDMA Native Read App Hypervisor Host VFS Read Hint KSM++ Host OS Real DMA Read Physical VDI File VDI File Disk 13/11 K. Miller, M. Rittinghaus, F . Bellosa – Deduplication in VM Environments
Recommend
More recommend
