Providing Hybrid Block Storage for Virtual Machines using - PowerPoint PPT Presentation

Providing Hybrid Block Storage for Virtual Machines using Object-based Storage Sixiang Ma*, Haopeng Chen*, Heng Lu*, Bin Wei § , Pujiang He § *Shanghai Jiao Tong University Email: { masixiang, chen-hp, lu007heng } @sjtu.edu.cn § Intel Asia-Pacific R&D Ltd. Email: { bin.wei, pujiang.he } @intel.com

Trends: Virtualization REliable, INtelligent & Scalable Systems l Virtualization Ø Key technology to increase resource sharing Ø 70% x86 server are virtualized l Virtual Block Devices Ø Network-based storage Ø Amazon EBS, Ceph RBD, Sheepdog, GlusterFS, etc. Ø Higher scalability, availability, manageability than direct-attached disks 2

Trends: SSDs and Hybrid Storage REliable, INtelligent & Scalable Systems l SSDs play a critical role in storage landscape Ø Superior random I/O performance than HDDs Ø VMs demanding high storage performance benefit Ø Higher unit capacity cost than HDDs l Hybrid storage systems provide eclectic solutions Ø Cost saving by HDDs Ø Performance improvement by SSDs 3

Issues: Hybrid Storage System for VMs REliable, INtelligent & Scalable Systems l Virtualized workload Ø Virtual Machine Disk Images, VMDIs Ø Most I/Os accessing the unstructured data l High Availability Guarantee Ø Service Level Agreements, SLAs Ø Offline methods are unfeasible l Data migration hurts scalability Ø Ideally cloud service is expected to expand unlimitedly Ø Suffer from resource bottleneck by data migration 4

Address Issues using Object-based storage REliable, INtelligent & Scalable Systems l Object-based Storage Ø Objects are logical storage entities with file-like access Ø Object Storage Devices provide higher-level interface than block storage Ø Direct data accessing from clients to OSDs -> high performance Ø Data (e.g., VMDIs) are stripped and randomly stored among OSDs for load balancing and parallelism Ø No meta-data nodes like file systems -> higher scalability 5

What our research focuses REliable, INtelligent & Scalable Systems Hybrid VBDs using Object- Object- based Storage Based Storage Virtual Hybrid Block Storage System Device 6

Background: I/O Virtualization REliable, INtelligent & Scalable Systems Hyperviosr Virtual Machine Operating System App App Network File Local File System 1a. Network-based Systtem file system Block Layer Controller Driver Network based solutions Emulated I/O Controller Disk 2c. Block-based Local- Virtual Block Image 2e is what our 2d. File-based Attached Device File Block Device 2e. Object-based work focuses Local solutions 2b. Files on local file Underlying Storage systems System 2a. Direct-Attached Drives Drives Storage (DAS) 7

System Architecture of MOBBS REliable, INtelligent & Scalable Systems Extent Table VM extent id value Block I/O Request 0 HDD POOL Hypervisor Emulated I/O Controller 1 SSD POOL Extent N SSD POOL Mapper Analyzer Table Block I/O Request Migration Command Object Client Client Migrater Component Clients Object I/O Request Sub-Migration Command Failure Object I/O detection Request Object OSD OSD Migrater Interface Object Client MOBBS OSD Component Monitors OSDs File System Object-based Storage 8

The Hybrid Pool REliable, INtelligent & Scalable Systems l Static object placement in current object-based systems Ø One disk image, One Storage pool Ø Can not take advantage of I/O locality Extent l MOBBS stripes a VMDI into VMDI Extents (multiple of objects) and Object Object Object Object Object stores into different pools Hybrid Pool Ø Reorganize Extents between hdd pool ssd pool different pools dynamically Ø Monitor real-time workloads Object HDD OSDs SSD OSDs 9

Placement Model: SSDs vs. HDDs REliable, INtelligent & Scalable Systems 140 ssd-pool-seq 120 ssd-pool-ran hdd-pool-seq 100 Banwidth (MB/s hdd-pool-ran read 80 60 40 SSDs excel: 20 0 ü Small I/Os 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M 4M Request Size (B) ü Random I/Os 600 ssd-pool-seq ssd-pool-ran 500 hdd-pool-seq hdd-pool-ran Banwidth (MB/s 400 write 300 200 100 0 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M 4M Request Size (B) 10

Placement Model: Pool Identification REliable, INtelligent & Scalable Systems ü Maximize the rate of small and random I/Os on SSDs ü Calculate beneficial score (BS) of each I/O ü Calculate beneficial rate (BR) of each extent ü The higher BR is, more beneficial to be stored with SSDs 11

Migration Distribution REliable, INtelligent & Scalable Systems l Stripping Extent Migration into Object Migration l OSD where object is stored takes the responsibility of real data migration ü Read locally instead of network I/O, only one write operation ü Concurrent object migration ü Little burden for VMs, absorb data migration across the OSD cluster Extent Migration Object Object Object Object Migrati on Migrati on Migrati on Migrati on Control I/O Data I/O OSD OSD OSD OSD 12

Implementation Issues REliable, INtelligent & Scalable Systems l Ceph 0.72 ü 2,500 lines among the librbd module ü No modification required among the OSD module ü OSD Migraters are user-level daemons l KVM-QEMU ü Avoidance of large changes ü Only 12 lines are modified 13

Evaluation: Methodology REliable, INtelligent & Scalable Systems Ceph SSD Pool VM Filebench Fio Ceph CREATE ATTCH Ext4 Hybrid Pool VBD MOBBS Client Pool 3 Pools Evaluate VBDs OSDs = 6 SSDs + 6 HDDs 14

Evaluation: Block I/O Workloads REliable, INtelligent & Scalable Systems Increasing skewness of random 4k writes 1.2 8 ü MOBBS provides higher 7 throughput than Hybrid 1 Ceph 6 0.8 ceph-ssd-vbd Throughput (MB/s) 5 ceph-hybrid-vbd ü MOBBS Close to SSD SSD Ratio (%) mobbs-vbd 0.6 4 Ceph with workload ssd ratio becomes skewer 3 0.4 2 ü SSD usage drops 0.2 1 0 0 0 1.25 1.5 1.75 2 2.25 Zipf Distribution 15

Evaluation: Block I/O Workloads REliable, INtelligent & Scalable Systems Different I/O size of Zipf1.5 random writes ü Throughput of Hybrid 1.2 6 Ceph increase when I/O size becoming larger 1 5 ü MOBBS outperforms 0.8 4 Throughput (MB/s) Hybrid Ceph with small SSD Ratio (%) I/Os and equalizes with 0.6 3 large I/Os ceph-ssd-vbd 0.4 2 ceph-hybrid-vbd ü SSD usage drops with mobbs-vbd I/O size increasing, 0.2 1 ssd ratio while both Hybrid Ceph and MOBBS approaching 0 0 16KB 32KB 64KB 128KB 256KB 512KB SSD Ceph I/O Requst Size 16

Evaluation: File System Ext4 REliable, INtelligent & Scalable Systems ü IOPS of four applications: fileserver, varmail, webserver, viderserver ü No ssd usage for videoserver, equivalent performance 3000 ceph-ssd-vbd mobbs-vbd 2500 ceph-hybrid-vbd 2000 IOPS (op/s) 1500 1000 500 0 fileserver varmail webserver videoserver 41% 22% 28% 0% Applications 17

Evaluation: File System Ext4 REliable, INtelligent & Scalable Systems Average latencies of 4 applications 450 ceph-ssd-vbd 400 mobbs-vbd 350 ceph-hybrid-vbd 300 Average Latency (ms) 250 200 150 100 50 0 fileserver varmail webserver videoserver Applications 18

Evaluation: File System XFS REliable, INtelligent & Scalable Systems ü IOPS of four applications: fileserver, varmail, webserver, viderserver ü No ssd usage for videoserver, equivalent performance 3000 ceph-ssd-vbd mobbs-vbd 2500 ceph-hybrid-vbd 2000 IOPS (op/s) 1500 1000 500 0 fileserver varmail webserver videoserver Applications 37% 15% 25% 0% 19

Evaluation: File System XFS REliable, INtelligent & Scalable Systems Average latencies of 4 applications 1400 ceph-ssd-vbd 1200 mobbs-vbd ceph-hybrid-vbd 1000 Average Latency (ms) 800 600 400 200 0 fileserver varmail webserver videoserver Applications 20

Thank You!Q/A