Duy Le (Dan) - The College of William and Mary Hai Huang - IBM T. J. Watson Research Center Haining Wang - The College of William and Mary
Virtualization Games Games Web server Videos Programming Database server Web File server Mail server Performance Implications Disks Disks Disks of [Jujjuri-LinuxSym’10] VirtFS - File system pass- File Systems through [Boutcher-Hotstorage’09] [Tang-ATC’11] Different I/O scheduler combinations File Systems File Systems File Systems [Tang-ATC’11] Storage space allocation and tracking dirty blocks functionality optimization Storage Storage 2
“Selected file systems are based on workloads” ◦ Only true in physical systems File systems for guest virtual machine ◦ Workloads ◦ Deployed file systems (at host level) Investigation needed! Guest File Systems Ext2, Ext3, Ext4, ReiserFS, XFS, and JFS Host File Systems Ext2, Ext3, Ext4, ReiserFS, XFS, and JFS 3
For the best performance? Best and worst Guest/Host File System combinations? Guest and Host File System Dependency ◦ Varied I/Os and interaction ◦ File disk images and physical disks 4
� Experimentations � Macro level � Throughout analysis � Micro level � Findings and Advice 5
� Experimentations � Macro level � Throughout analysis � Micro level � Findings and Advice 6
Qemu 0.9.1; 512MB RAM 9 x 106 blocks Linux 2.6.32 Guest File Systems vdc1 vdc2 vdc3 Ext2 Ext3 Ext4 ReiserFS vdc4 vdc5 vdc6 XFS JFS VirtIO File Systems sdb1 Ext2 sdb2 sdb3 Ext3 Ext4 ReiserFS XFS sdb4 sdb5 sdb6 sdb7 JFS BD Host Pentium D 3.4 GHz, 2GB Ram Raw partition as 60 x 106 blocks Linux 2.6.32 + Qemu-KVM 0.12.3 block device ( BD ) Sparse disk image 1 TB, SATA 6Gb/s, 64MB Cache 60 x 106 blocks 7
Filebench ◦ File server, web server, database server, and mail server. Throughput Latency I/O Performance ◦ Different abstraction consideration Via block device (BD) Via nested file systems ◦ Relative performance variation BD as baseline 8
Relative performance to baseline of guest file systems Host file Baseline systems 9
ReiserFS ReiserFS guest file system guest file system 10
Throughput (MB/s) Throughput (MB/s) 30 30 120 Percentage (%) 25 25 100 R 20 20 80 15 15 60 10 10 40 5 5 20 0 0 0 BD BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Ext4 ReiserFS guest file system guest file system 11
Guest file system Host file systems ◦ Varied performance Host file system Guest file systems ◦ Impacted differently Right and wrong combinations ◦ Bidirectional dependency I/Os behave differently ◦ Writes is more critical than Read (mail server) 12
Throughput (MB/s) Throughput (MB/s) 8 8 100 Percentage (%) 90 7 7 80 R 6 6 70 5 5 60 50 4 4 40 3 3 30 2 2 20 1 1 10 0 0 0 Ext2 Ext3 Ext4 ReiserFS XFS JFS BD BD Ext2 Ext3 guest file system guest file system 13
Guest file system Host file systems ◦ Varied performance Host file system Guest file systems ◦ Impacted differently Right and wrong combinations ◦ Bidirectional dependency (mail server) I/Os behave differently ◦ Writes is more critical than Read (mail server) 14
80 Throughput (MB/s) Throughput (MB/s) 2.5 2.5 Percentage (%) 70 2 2 R 60 50 1.5 1.5 40 1 1 30 20 0.5 0.5 10 0 0 0 Ext2 Ext3 Ext4 ReiserFS XFS JFS BD BD Ext2 guest file system 15
Guest file system Host file systems ◦ Varied performance Host file system Guest file systems ◦ Impacted differently Right and wrong combinations ◦ Bidirectional dependency I/Os behave differently ◦ Writes is more critical than Reads 16
Lower than 100% 80 Throughput (MB/s) Throughput (MB/s) 2.5 2.5 Percentage (%) 70 2 2 R 60 50 1.5 1.5 40 1 1 30 20 0.5 0.5 10 0 0 0 Ext2 Ext3 Ext4 ReiserFS XFS JFS BD BD More WRITES 17
Guest file system Host file systems ◦ Varied performance Host file system Guest file systems ◦ Impacted differently Right and wrong combinations ◦ Bidirectional dependency I/Os behave differently ◦ WRITES are more critical than READS 18
Guest file system Host file systems ◦ Varied performance Host file system Guest file systems ◦ Impacted differently Right and wrong combinations ◦ Bidirectional dependency I/Os behave differently ◦ WRITES are more critical than READS Latency is sensitive to nested file systems 19
� Experimentations � Macro level � Throughout analysis � Micro level � Findings and Advice 20
Same testbed Primitive I/Os ◦ Reads or Writes ◦ Random or Sequential FIO benchmark Description Parameters T otal I/O size 5 GB I/O parallelism 255 Block size 8 KB I/O pattern Random/Sequential I/O mode Native async I/O 21
Throughput (MB/s) Throughput (MB/s) 2 2 160 Percentage (%) 140 R 1.6 1.6 120 100 1.2 1.2 Random 80 0.8 0.8 60 40 0.4 0.4 20 0 0 0 BD BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Throughput (MB/s) Throughput (MB/s) 100 100 160 Percentage (%) 140 R 80 80 120 100 60 60 Sequential 80 40 40 60 40 20 20 20 0 0 0 BD BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Ext3 guest file system Ext3 guest file system 22
Read dominated workloads ◦ Unaffected performance by nested file systems Write dominated workloads ◦ Heavily affected performance by nested file systems 23
Throughput (MB/s) Throughput (MB/s) 4 4 160 Percentage (%) 140 R 3 3 120 100 Random 2 2 80 60 1 1 40 20 0 0 0 BD BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Throughput (MB/s) Throughput (MB/s) 100 100 160 Percentage (%) 140 R 80 80 120 100 60 60 Sequential 80 40 40 60 40 20 20 20 0 0 0 BD BD Ext2 Ext3 Ext4 ReiserFS XFS JFS 24
Read dominated workloads ◦ Unaffected performance by nested file systems Write dominated workloads ◦ Heavily affected performance by nested file systems 25
Throughput (MB/s) Throughput (MB/s) 100 100 160 Throughput (MB/s) Throughput (MB/s) 100 100 160 Read dominated workloads Percentage (%) Ext3 guest file system 140 Percentage (%) 140 R 80 80 R JFS guest file system JFS guest file system 80 80 120 120 ◦ Unaffected performance by nested file systems 100 60 60 100 60 60 80 80 Write dominated workloads Ext3 guest file system Ext3 guest file system 40 40 60 40 40 60 40 ◦ Heavily affected performance by nested file systems 40 20 20 20 20 20 20 0 0 0 0 0 0 BD BD Ext2 Ext3 Ext4 ReiserFS XFS JFS BD BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Sequential Reads: Ext3/JFS vs. Ext3/BD Sequential Reads: Ext3/JFS vs. Ext3/BD Sequential Writes: ◦ Ext3/ReiserFS vs. JFS/ReiserFS (same host file systems) ◦ JFS/ReiserFS vs. JFS/XFS (same guest file systems) I/O analysis using blktrace 26
Findings: Readahead at the hypervisor when nesting FS Long idle times for queuing 27
Different guests (Ext3, JFS) same host (ReiserFS) ◦ I/O scheduler and Block allocation scheme 28
Long waiting in the queue Well merged Low I/Os � Ext3 causes multiple for journaling back merges � JFS coalescences multiple log entries 29
Different guests (Ext3, JFS) same host (ReiserFS) ◦ I/O scheduler and Block allocation scheme ◦ Findings I/O schedulers are NOT effective for ALL nested file systems I/O scheduler’s effectiveness on block allocation scheme 30
Different guests (Ext3, JFS) same host (ReiserFS) Same guest (JFS) different hosts (ReiserFS, XFS) ◦ Block allocation schemes 31
Longer waiting in queue Fairly similar CDF of disk I/Os 1 Repeated 0.8 logging Long distance 0.6 seeks 0.4 ReiserFS 0.2 XFS 0 0 0.2 0.4 0.6 0.8 1 Normalized seek distance 32
Different guests (Ext3, JFS) same host (ReiserFS) Same guest (JFS) different hosts (ReiserFS, XFS) ◦ Block allocation schemes ◦ Findings: Effectiveness of guest file system’s block allocation is NOT guaranteed Journal logging on disk images lowers the performance 33
� Experimentations � Macro level � Throughout analysis � Micro level � Findings and Advice 34
Advice 1 – Read-dominated workloads ◦ Minimum impact on I/O throughput ◦ Sequential reads: even improve the performance Advice 2 – Write-dominated workloads ◦ Nested file system should be avoided One more pass-through layer Extra metadata operations ◦ Journaling degrades performance 35
Advice 3 – I/O sensitive workloads ◦ I/O latency increased by 10-30% Advice 4 – Data allocation scheme ◦ Data and Metadata I/Os of nested file systems are not differentiated at host ◦ Pass-through host file system is even better! Advice 5 – Tuning file system parameters ◦ “Non-smart” disk ◦ Noatime and nodiratime 36
37
Devices Blocks (x10 6 ) Speed (MB/s) Type sdb2 60.00 127.64 Ext2 sdb3 60.00 127.71 Ext3 sdb4 60.00 126.16 Ext4 sdb5 60.00 125.86 ReiserFS sdb6 60.00 123.47 XFS sdb7 60.00 122.23 JFS sdb8 60.00 121.35 Block Device 38
Recommend
More recommend
