OpenSolaris NFS TM Performance Alain van Hoof February 3, 2010 - PowerPoint PPT Presentation

OpenSolaris NFS TM Performance Alain van Hoof February 3, 2010

Research Question \begin{quotation} How can the performance bottlenecks be monitored and identified on an OpenSolaris OS NFS server. What are realistic load simulations and create a base-line. \end{quotation} 2

NFS layer and ...... Client Linux NFS Client TCP/IP TCP/IP NFS Server VFS Layer ZFS FileSystem ZIL L2ARC Physical Devices Server OpenSolaris 3

Inspecting the layers Client Linux NFS Client nfsstat -c TCP/IP netstat tcpdump/wireshark mib:ip:ip_ TCP/IP netstat nfsv3:nfssrv:rfs3_ NFS Server nfsstat -s fsinfo:genunix:fop_ VFS Layer fsstat zilstat fbt:zfs:arc_ ZFS FileSystem ZIL L2ARC arcstat fbt:zfs:zil_lwb_write zpool iostat io:genunix::start iostat Physical Devices Dtrace Probes Server OpenSolaris 4

Read Write Dtrace Probes TCP/IP Ard Awr mib:ip:ip_xmit:ipIfStatsHCOutTransmits mib:ip:ip_input:ipIfStatsHCInReceives NFSv3 Server Brd Bwr nfsv3:nfssrv:rfs3_read:op-read-start nfsv3:nfssrv:rfs3_write:op-write-start VFS Layer Cwr Crd fsinfo:genunix:fop_write:write fsinfo:genunix:fop_read:read ZFS FileSystem Drd Dwr fbt:zfs:zfs_read:entry fbt:zfs:zfs_write:entry ARC ZIL Frd fbt:zfs:arc_read:entry E fbt:zfs:zil_lwb_write_start:entry Fwr fbt:zfs:arc_write:entry Physical Disk G io:genunix::start 5

Benchmarking Marcro benchmarks real-world workload Trace benchmarks replay recorded real-world scenario Micro benchmarks one or two operations 6

Filebench Macro benchmark TM Supported by SUN fileserver: SPECsfs workload IOzone Micro benchmark A benchmark UvA-IC is familiar with Random Read/Write 7

Auto-pilot Automate the running of benchmarks Avoid user errors (typo’s) Automatic mount and unmount filesystems Calculate confidence level and run more test to be more confident 8

Test Setup VMware 01 VMware 02 TestClient LoadClient LoadClient LoadClient LoadClient Test Results 01 02 03 04 auto-pilot autopilot autopilot autopilot autopilot filebench filebench filebench filebench filebench IOzone NFS Server Server Metrics Log 9

Benchmark & loadgeneration Run benchmark on TestClient while load is generated on LoadClients Real-world benchmark is also real-world load generator 10

Meanwhile on the NFS server 12 Dtrace probes are logged during testing Output: Every 10 seconds Probes triggered/second 11

Preliminary Test A simple test to learn about behavior of the Dtrace probe “Accidentally” Identified NFS bottle-neck Changed number of NFS daemons Positive result 12

Dtrace write probes 1000 Awr/32 Bwr Cwr 800 Dwr Before After E 600 G/10 400 200 0 Time 13

Results of Test Filebench and IOzone show performance loss when load increases Filebench more close to “user experience” IOzone tests are not within the confidence interval (95%) after 30 runs 14

filebench 40 35 30 25 MB/s 20 15 10 5 0 0 1 2 3 4 8 16 Load 15

IOzone Random Write 40 35 30 25 MB/s 20 15 10 5 0 0 1 2 3 4 8 16 Load 16

Awr/32 1400 Bwr Cwr 1200 Dwr Probes / Second E 1000 G/10 800 600 400 200 0 Time ZIL write probe 17

Time ZIL write probe 18

ZFS and ZIL Synchronous Write Asynchronous Write All NFS writes are ZFS ZIL Synchronous and pass the ZIL 19

Improve performance Place ZIL on fast device Solid State Disk (SSD) Well known practice 20

Results - ZIL on SSD Filebench and IOzone show performance increase When the load increases performance decreases IOzone tests are not within the confidence interval (95%) after 30 runs 21

filebench ZIL on SSD 60 50 filebench normal ZIL 40 MB/s 30 20 10 0 0 1 2 3 4 8 16 Load 22

Dtrace write Probes Awr/32 1400 Bwr Cwr 1200 Dwr Probes / Second E 1000 G/10 800 600 400 200 0 Time ZIL write probe 23

Conclusion Using Dtrace probes, performance bottlenecks on the server can be identified, and a baseline can be created. A realistic load can be generated using the filebench - fileserver personality Filebench test created a confident baseline (unlike IOzone) 24

Questions? Thanks to: Jeroen Roodhart Auke Folkerts 25