Chelonia a lightweight self-healing distributed storage Zsombor Nagy ( zsombor@niif.hu ) Salman Toor ( salman.toor@it.uu.se ) Jon Kerr Nilsen ( j.k.nilsen@fys.uio.no )
Motivation • How to easily... • Create a storage resource from available disk space on any computer • Connect storage resources through the Internet • Provide an easy-to-understand way for users to upload and share their files • Use these files in a Grid environment
Design • The system should be... • self-healing • easy to deploy • easy to maintain • lightweight • no single point of failure
Decisions • Files • Replication • Berkeley DB HA • HTTP(S) file transfer • SOAP messaging • X509 certificates
Chelonia NORDUGRID Grid Solution for Wide Area Grid Solution for Wide Area Computing and Data Handling Computing and Data Handling • Developed by the KnowARC project • Will be maintained by the NorduGrid Collaboration • Running in NorduGrid ARC’s web service container • Written in Python • Running on Linux and Mac OS X (and Windows soon)
The Chelonia Cloud • Storage cloud for anyone • Can be created by anyone • Can be shared with anyone • Can be used by anyone
Global namespace • Global hierarchical namespace • Files are organized in collections • All users see the same tree • Can use logical names • Paths in Chelonia namespace • Similar to paths in a regular local filesystem
Replication • Files are replicated • User specifies how many replicas are needed • Chelonia itself takes care of the replication • Easy to create a new storage node
Access Control • Access policies to files and collections • Can grant access to • Individual users • If in a Grid environment, to entire virtual organizations
Client tools • Several ways to access Chelonia • Command line interface • FUSE module • In an ARC enabled Grid environment • Through job specification file • With ARC client tools
FUSE module • We have a FUSE module to mount Chelonia as a local filesystem • You can upload and download files, create collections (directories) with your OS’s CLI and GUI tools (e.g. drag and drop) • Linux and Mac OS X
Components of Chelonia • Consists of four services • The Bartender • The Shepherd • The Librarian • The A-Hash • Can have multiple instances of all services
The Shepherd • Manages the storage node with the actual file data • Replication requires more storage nodes, each managed by a Shepherd • Adding a Shepherd, replica will automatically be created (if needed)
The Librarian and A-Hash • The Librarian manages the metadata • Uses the A-Hash to store the data • Both metadata and Librarians can be replicated • More fault tolerant • Better load balance
The Bartender • Serves the users by negotiating with • the Librarian to query files and collections • the Shepherd to upload and download files • More than one Bartender to eliminate single point of failure • Users can contact any Bartender and get the same result
Downloading A-H 1 . g e t F i l e ( L N ) B A-H A-H 2. traverse(LN) 6. TURL ) D A-H a I t a e d a c t e n m e s ' e r l e fi . f 3 e r ( t e 5. TURL g . 4 7. downloading the file L S USER S � E � S
Uploading 2. traverse(LN) A-H 3. parent's data A-H A-H a ) t B a d a e t m ( w e n 4 . A-H ) L N e ( i l t F p u . 1 5. file's GUID 6. add entry to parent 12. TURL L e l 11. TURL fi e h ) . USER t . 8. list of Shepherds . g ( n t u i d p a o . 9 l p u S 1 0 . r . e g 3 i s t e r n 1 e w r e p l i c a 14. report state change S � E � S
Depth test
Width test
Multiple users
Number of replicas Five Shepherds were used for this test ALIVE OFFLINE THIRDWHEEL CREATING User Side 10 files of size 100MB have 3 replicas and now user: • Increase the number replicas to 5. • Decrease the number of replicas to 2 50 Number of files in the system 50 40 40 30 30 20 20 10 10 0 0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 15 30 45 60 75 90 105 120 135 105 165 180 195 210 225 240 255 Time in seconds taken by the system to achieve the needed replicas System Side Number of files in the system 50 40 10 files with 4 replicas are in the system. 30 • One Shepherd went OFFLINE for sometime 20 • Became ONLINE 10 0 15 30 45 60 75 90 105 120 135 150 165 Time in seconds taken by the system to achieve the needed replicas
Memory usage 8 days running with lots of file uploads and downloads
Future? • Better clients? • SRM? WebDAV? • Encryption? • AAI? • Transfer protocols? • Transfer security? • Versioning? • Resuming uploads? • SOAP over HTTPS? • Space reservation?
ClusterGrid • Maintained by NIIF, Hungary • Since 2002 connecting a few hundred PCs located at several universities in Hungary • Previously using NIIF’s own grid middleware • Soon it will use ARC and Chelonia
Chelonia and ClusterGrid • Soon there will be a purchase of some storage solutions which will be deployed at chosen Hungarian universities • We will deploy Chelonia storage nodes on these • The ClusterGrid users will be able to • upload and download (input/output) files to the Chelonia cloud • and refer to these files in the job descriptions
Questions / Demo
Thank You! web : http://www.knowarc.eu/chelonia/ paper : http://arxiv.org/abs/1002.0712 video : http://www.youtube.com/watch?v=NEUWzGHHGhc Zsombor Nagy, NIIF (Hungary), zsombor@niif.hu
More recommend
