Remote File System Suite Softwarepraktikum f ur Fortgeschrittene - PowerPoint PPT Presentation

Introduction Evaluation Conclusion rfsd librfs grfs Remote File System Suite Softwarepraktikum f¨ ur Fortgeschrittene Michael Kuhn Parallele und Verteilte Systeme Institut f¨ ur Informatik Ruprecht-Karls-Universit¨ at Heidelberg 2009-07-07 1 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Introduction 1 Introduction Remote File System Daemon 2 Remote File System Library 3 Global Remote File System 4 Evaluation 5 Conclusion 6 2 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Introduction FUSE Goal was to implement a global network file system Needed to implement the underlying network file system first Should be implemented as a FUSE file system Runs in user space Relatively easy to implement Relatively easy to maintain High performance Microscope delivers 1 GB/s Transparent Use existing file systems as storage No striping of files 3 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Introduction Overview rfsd – Remote File System Daemon Low-level network file system librfs – Remote File System Library Abstracts protocol implementation rfsc – Remote File System Client Basically a simple throughput and metadata benchmark grfs – Global Remote File System High-level global network file system 4 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Introduction 1 Remote File System Daemon 2 Motivation Overview Features Remote File System Library 3 Global Remote File System 4 Evaluation 5 Conclusion 6 5 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Motivation A separate protocol was designed Existing protocols did not meet the requirements SSH Does not support unencrypted data channels Data encryption makes transfers too slow Not possible to deactivate the encryption FTP Only possible to write a complete file or append data to it File listings are hard to parse, because their format is not well-defined 6 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Overview Implement our own protocol Separate control and data channels No encryption Control channel can be encrypted via SSH forwarding Should be as fast as possible Microscope pumps out 1 GB/s 6 · 2 servers ⇒ 100-200 MB/s Should be as transparent as possible Use underlying local file system Do not stripe files across servers Should be as safe as possible Support replication 7 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Features Basically provide remote access to the local file system Protocol very similar to POSIX pread() , pwrite() , . . . Plus some fancy features, of course :-) Fully multi-threaded Each connection handled in its own thread Long-running operations do not block other connections Replication Master-slave concept One master, multiple slaves All operations are replicated in a background thread Pushed into the thread when the operation begins Check whether the thread finished when the operation ends 8 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Features Logging All operations are logged when a slave is offline Kept in memory and written to log file Replayed when slave comes online chroot -like restrictions All accesses can be restricted to a sub-tree of the file system 9 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Introduction 1 Remote File System Daemon 2 Remote File System Library 3 Motivation Global Remote File System 4 Evaluation 5 Conclusion 6 10 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Motivation Hide all the “ugly” implementation details :-) Good error reporting via GError Part of GLib Some operations require multiple steps For example: rfs read() (“ open ”), rfs read do() (“ pread ”), rfs read end() (“ close ”) 11 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Introduction 1 Remote File System Daemon 2 Remote File System Library 3 Global Remote File System 4 Overview Features Evaluation 5 Conclusion 6 12 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Overview Merge multiple file systems into one global namespace Example: serv1 has directory /foo , serv2 has directory /bar $ grfs serv1:6666 serv2:6666 /grfs $ ls /grfs > foo bar 13 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Features High Availability Continues to work when servers go offline Provides a partial view of the file system setuid -like functionality Supports FUSE’s allow other option Fast reads and writes Keeps the state of read and write operations Accesses to the same file are handled with very little overhead 14 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Introduction 1 Remote File System Daemon 2 Remote File System Library 3 Global Remote File System 4 Evaluation 5 Configurations Evaluation Conclusion 6 15 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Configurations 12 Daemons & 12 Clients One daemon and one client on each machine 6 Daemons & 6 Clients One daemon on each of the first six machines One client on each of the last six machines 6 Daemons & 12 Clients One daemon on each of the first six machines Two clients on each of the last six machines All numbers are per-client 16 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Evaluation Remote File System Performance 120 Σ=640 100 Σ=1106 Σ=1067 Σ=968 80 MB/s 60 40 Σ=437 Σ=216 20 0 write read Configurations 12 Daemons & 12 12 Daemons & 12 6 Daemons & 6 Clients Clients (without Clients Replication) Replication: Significant performance drop 17 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Evaluation Remote File System Metadata Performance 6000 5000 4000 Operations/s 3000 2000 1000 0 create stat access chmod chown unlink Configurations 12 Daemons & 12 12 Daemons & 12 6 Daemons & 6 Clients Clients (without Clients Replication) 18 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Evaluation Global Remote File System Performance 80 70 60 50 MB/s 40 30 Σ=114 Σ=114 Σ=108 20 Σ=48 10 0 6 Servers/Clients 6 Servers/Clients (Direct I/O) 1 Server/Client 1 Server/Client (Direct I/O) Configurations Write Read 1 GBit/s maximum throughput – Read: 1.7 GB/s (cache) 19 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Evaluation Global Remote File System Metadata Performance 2000 1800 1600 1400 Operations/s 1200 1000 800 600 400 200 0 6 Servers/Clients 6 Servers/Clients (Direct I/O) 1 Server/Client 1 Server/Client (Direct I/O) Configurations create unlink 20 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Introduction 1 Remote File System Daemon 2 Remote File System Library 3 Global Remote File System 4 Evaluation 5 Conclusion 6 Conclusion 21 / 22

Introduction Evaluation Conclusion rfsd librfs grfs Conclusion Global Remote File System performance is limited by FUSE Read and write buffers are at most 128 KB in size FUSE 2.8.0 pre-release supports up to 512 KB 22 / 22