Introduction Container Library Container Tools FUSE Container File System Evaluation Container Library and FUSE Container File System Softwarepraktikum f¨ ur Fortgeschrittene Michael Kuhn Parallele und Verteilte Systeme Institut f¨ ur Informatik Ruprecht-Karls-Universit¨ at Heidelberg 2008-02-29 1 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Introduction 1 Introduction Requirements Container Library 2 Container Tools 3 FUSE Container File System 4 Evaluation 5 2 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Introduction An attempt to decrease the metadata overhead is to maintain a reduced set of metadata files This usually has to be done manually Available file systems do not permit the user to change which metadata is stored One approach is to pack them together in one file – a container The file system only manages metadata for one file Within this container, the files and their corresponding metadata can be managed arbitrarily 3 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Requirements The container format should enable random access to provide access times independent of the position of a file within the container Existing formats are insufficient The tar format does not provide random access The iso format stores too much metadata Therefore a new container format was designed and implemented by Hendrik Heinrich This existing implementation was used as a basis for all work presented here 4 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Introduction 1 Container Library 2 License General Overhaul New Features Container Tools 3 FUSE Container File System 4 Evaluation 5 5 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation License The container library was not licensed in any way This made future development and usage difficult It has been licensed under a 2-clause BSD license in agreement with the original author 6 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation General Overhaul The library was completely overhauled It now provides consistently named functions and data types The comments within the code were modified to allow the automatic generation of an API documentation with Doxygen 7 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation New Features Architecture Independence The original implementation did not honor the different sizes of data types on 32 and 64 bit architectures This made it impossible to use a container created with a 32 bit version of the library with a 64 bit version and vice versa By using datatypes of a fixed size the containers can now be used on both 32 and 64 bit architectures without problems The size of the metadata structures used in the library is now independent of the architecture C pads structures for better memory alignment 8 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation New Features Thread-Safety The library was not thread-safe in its original form It uses a shared file pointer for all files in a container The functions read and write were used in combination with lseek The library was modified to use the pread and pwrite functions that do not modify the shared file pointer It can now be used safely in multi-threaded applications 9 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation New Features Write Support The original version of the container library lacked an easy-to-use method to create containers A convenient interface to add new files was added to the library Either from memory with ct file create buffer Or from an existing file with ct file create path There are currently also “fast” versions of these functions 10 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation New Features File Hashing File data stored in the container is protected from silent corruption Currently a SHA-1 hash of it is stored along with its metadata This also needs to be done for the metadata 11 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Introduction 1 Container Library 2 Container Tools 3 Overview Tools FUSE Container File System 4 Evaluation 5 12 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Overview Five tools to work with containers on the command line ctcat ctcp ctls ctmk ctpc 13 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Tools ctcat Print the contents of a file in a container to stdout ctcp Copy a file from a container to a local file system ctls Print the names of all files in a container Optionally also print their hashes or sizes ctmk Create a new container with all files in a given directory ctpc Copy a file from a local file system to a container 14 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Introduction 1 Container Library 2 Container Tools 3 FUSE Container File System 4 Motivation Realization Evaluation 5 15 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Motivation There are legacy applications that use the POSIX API and can not be ported It may be too much work to port them Or the source code is not available at all Provide POSIX access to containers The easiest way to do this is to write a FUSE file system 16 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Realization Provide an overlay file system The whole underlying file system is accessible Directories are handled normally – as directories Files are handled as containers – that is, also as directories For example ctfs is mounted at /ctfs There is a container available as /storage/stuff.ct All files within this container would be available in the directory /ctfs/storage/stuff There is more room for optimization Does two stat() calls for each container file One can be eliminated easily 17 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Introduction 1 Container Library 2 Container Tools 3 FUSE Container File System 4 Evaluation 5 Evaluation 18 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Evaluation Without Metadata (First Execution) 45 40 35 Time (in seconds) 30 10,000 25 100,000 1,000,000 20 15 10 5 0 POSIX ctfs ct 19 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Evaluation Without Metadata 1 0,9 0,8 0,7 Time (in seconds) 0,6 10,000 100,000 0,5 1,000,000 0,4 0,3 0,2 0,1 0 POSIX ctfs ct 20 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Evaluation With Metadata (First Execution) 700 600 500 Time (in seconds) 10,000 400 100,000 1,000,000 300 200 100 0 POSIX ctfs ct 21 / 22
Introduction Container Library Container Tools FUSE Container File System Evaluation Evaluation With Metadata 180 160 140 Time (in seconds) 120 10,000 100 100,000 1,000,000 80 60 40 20 0 POSIX ctfs ct 22 / 22
Recommend
More recommend
