Supporting Transactions for Bulk NFSv4 Compounds 13 th ACM International Systems and Storage Conference (SYSTOR 2020) Wei Su 1 , Akshay Aurora 1 , Ming Chen 2 , Erez Zadok 1 1 Stony Brook University; 2 Google October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 1
Background: Vectorized NFS ● Ideal utilization of compounding: Writing multiple files in one compound request NFS Server Application NFS Client SEQUENCE; PUTROOTFH; LOOKUP “etc” ; GETFH; GETATTR; Vectorized SAVEFH; OPEN “passwd” ; WRITE 1800 47 ; CLOSE ; GETFH; File-system GETATTR; RESTOREFH; OPEN “group” ; WRITE 878 11 ; API CLOSE ; GETFH; GETATTR; RESTOREFH; OPEN “shadow” ; WRITE 1170 124 ; CLOSE ; GETFH; GETATTR; RESTOREFH; vec_write( 1 [‘/etc/passwd’, ‘/etc/group’, Status codes, file handles, and file attributes ‘/etc/shadow’ ], 5 ... ) October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 2
Background: Vectorized NFS ● Performance evaluation: Metadata intensive workload Recursive listing, symlink, and removal ◆ 259 ⨉ 16~259 ⨉ 7~106 ⨉ 2.5~12 ⨉ 2.5 ⨉ LAN WAN October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 3
Motivation ● NFSv4 introduces “Compound” procedure ◆ Clients can pack multiple NFS operations in one “compound” ◆ This amortizes network latency and improves I/O throughput Compounding speeds up NFS I/O by up to 2 orders of magnitude ◆ ● Challenge to client’s error handling ◆ If an operation in a compound fails ▪ NFS server only reports the error, but does not rollback ◆ If the server crashed when executing a compound ▪ Nothing will be done when it restarts ◆ Difficulty for applications to handle errors ▪ Any operation may fail, and crash may occur anytime ▪ Hard to restore to initial state for a failed large compound October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 4
Design Overview Vectorized NFSv4 Client ① Compound request reaches TCNFS server PUTFH NFH1 ; WRITE ; PUTFH NFH2 ; WRITE ; <LFH1,NFH1> … ... <LFH3,NFH2> ② TCNFS writes the compound request into … ① <NFH1,LFH1> metadata database as a Recovery Record Compound <NFH1, Path1> TCNFS Server Request #23 <NFH2,LFH3> (RR) <NFH2, Path2> … ⑥ ② <RR#23,...> Transaction Layer ③ TCNFS backs up data blocks of files that Metadata Database Write LFH1; will be changed by the compound request LFH3; ⋯⋯ ④ ④ TCNFS executes the operations Clone ⑤ TCNFS removes backup data ③ ⑤ ⑥ TCNFS removes the recovery record Files in backup directory#23 Original files October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 5
Design: Error Handling Vectorized NFSv4 Client ● In case of an error… ◆ TCNFS reverses previously PUTFH NFH1 ; WRITE ; PUTFH NFH2 ; WRITE ; <LFH1,NFH1> executed operations … ... <LFH3,NFH2> … ● In case of a server crash… ① <NFH1,LFH1> Compound <NFH1, Path1> TCNFS Server Request #23 <NFH2,LFH3> ◆ The recovery record will be <NFH2, Path2> … ⑥ ② <RR#23,...> present in the metadata database Transaction Layer ◆ TCNFS will parse the recovery Metadata Database Write LFH1; LFH3; ⋯⋯ ④ record to retrieve the failed compound request Clone ◆ TCNFS reverses the partially done ③ ⑤ compound request Files in backup directory#23 Original files October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 6
Prototype Architecture (1) NFS Ganesha ● Lock Manager Transaction RPC + Protocol Layer (NFS v4) Logger ◆ Coordinates multi-client File System Abstraction Layer conflicting access MDCACHE: Metadata Cache ● Backup Manager TC-NFS Transaction Layer Metadata Lock Manager ◆ Database Creates and cleans up backups ● Undo Executor Backup Undo Metadata Manager Executor Translator Offline ◆ Reverts partially executed Undo Executor Vectorized NFSv4 API VFS: File System Wrapper compounds due to failure ● Metadata Translator System Call/ioctl User Kernel ◆ Mappings between NFS file Virtual File System handle and local file handle Networking CoW-enabled File System: XFS, btrfs (TCP/IP) SSD with Power-loss Protection October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 7
Prototype Architecture (2) NFS Ganesha ● Transaction Logger Transaction RPC + Protocol Layer (NFS v4) Logger ◆ Creates and cleans up the File System Abstraction Layer Recovery Records MDCACHE: Metadata Cache ● Offline Undo Executor TC-NFS Transaction Layer Metadata Lock Manager ◆ Database Reverts partially executed compounds due to server crash Backup Undo Metadata Manager Executor Translator ● CoW-enabled File System Offline Undo Executor Vectorized NFSv4 API ◆ VFS: File System Wrapper Use CoW to create backups to reduce I/O overhead System Call/ioctl User ● SSD with Power Protection Kernel Virtual File System ◆ Ensures endurance and reduces Networking CoW-enabled File System: XFS, btrfs (TCP/IP) the latency of fsync() SSD with Power-loss Protection October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 8
Experimental Setup ● 3 identical machines, 1 Server + 2 Clients ◆ Each client machine runs 4 KVM virtual machines ◆ Each VM runs Ubuntu 18.04 and one vNFS/NFSv4 client ● CPU: Intel Xeon X5650 ● RAM: 64GB ● Storage ◆ 147GB hard drive for system disk (ext4) ◆ 200GB Intel DC-S3700 SSD for server’s TC-NFS backend storage (XFS) ● Network ◆ 10GbE NIC connected via 10GbE switch ◆ average RTT = 0.2ms ● OS: Ubuntu 18.04 with Linux Kernel v4.15 October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 9
Micro-Benchmark: Writefiles ● Writefiles (Multi-client) ◆ Write 1,000 fixed-size files from 1K to 16M in parallel ◆ 1~8 clients, 0.2ms network latency Large files ( ⩾ 256K) 4.1~20% Overhead Small files ( ⩽ 128K) 36%~26 ⨉ Overhead October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 10
Explore the Bottleneck ● Local “Writefiles” Workload Simulation ◆ Concurrently writes 1,000 equal-size files locally to the SSD using 1~8 threads repeatedly for 30s ◆ fsync() is called after writing each file to simulate the behavior of the NFSv4 server (NFS-Ganesha) ◆ Two types of workload ▪ Interleaving-backup: Create backup for the target file before each write() operation using Copy-on-Write cloning ▪ No-backup: Only do write() and fsync() , no backups October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 11
Exploring the Bottleneck Solid lines: No-backup; Dashed lines: Interleaving-backup ● ● No-backup (NB) workload scales well with number of threads ● Interleaving-backup (IB) workload does not scale or become worse ● This reproduces the bad scalability of TC-NFS Speedup Ratio at 8Th NB: 2.0~4.6 ⨉ Speedup Ratio at 8Th IB: 0.36~0.95 ⨉ October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 12
Macro-Benchmark: Coreutils ● Test target: Linux kernel 4.20.7 source tree ◆ 62,447 regular files (Average size: 14.9 KB) 4,148 directories (Average 15 children per directory ◆ ● Single-client, varied network latency between 0.2ms to 30.2ms ● Baseline: vNFS Client + Vanilla NFSv4 Server; Measured total runtime Symlink: 8.8~50% Overhead 35~50% Removal: 7.9~42% Overhead Copy: 7.9~35% Overhead ⩽ 10% Listing: 1.1~18% Overhead October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 13
Conclusions ● Provides transaction support for NFSv4 compounds ◆ Makes error handling easier for applications ◆ Currently supports the following operations: OPEN, CREATE, WRITE, LINK, REMOVE and simple RENAME (non-directories) ● Introduces modest overhead to single-client workloads and real-world applications ◆ Considering the improvement vNFS provides, vNFS Client + TC-NFS is still much faster than traditional NFSv4 system ● Higher overhead in multi-client workloads ◆ This is because CoW cloning + synced writes are slow on XFS Will be resolved once the CoW feature is optimized ◆ October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 14
Supporting Transactions for Bulk NFSv4 Compounds 13 th ACM International Systems and Storage Conference (SYSTOR 2020) Wei Su 1 , Akshay Aurora 1 , Ming Chen 2 , Erez Zadok 1 1 Stony Brook University; 2 Google Paper: https://www.fsl.cs.sunysb.edu/docs/nfs4perf/tcnfs-systor2020.pdf Project Source Code: https://github.com/sbu-fsl/fsl-tc-server Q&A October 14, 2020 Supporting Transactions for Bulk NFSv4 Compounds (ACM SYSTOR 2020) 15
Recommend
More recommend
