decentralized deduplication in san cluster file systems
play

Decentralized Deduplication in SAN Cluster File Systems Austin T. - PowerPoint PPT Presentation

Apr 04, 2023 •102 likes •1.28k views

Decentralized Deduplication in SAN Cluster File Systems Austin T. Clements Irfan Ahmad Murali Vilayannur Jinyuan Li VMware, Inc. MIT CSAIL Decentralized Deduplication in SAN Cluster File Systems Storage Area Networks Decentralized

  1. The Index 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable Decentralized Deduplication in SAN Cluster File Systems

  2. The Index 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable Decentralized Deduplication in SAN Cluster File Systems

  3. The Index 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks Decentralized Deduplication in SAN Cluster File Systems

  4. The Index 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks Decentralized Deduplication in SAN Cluster File Systems

  5. The Index 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks Decentralized Deduplication in SAN Cluster File Systems

  6. Indexing and Duplicate Elimination 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  7. Indexing and Duplicate Elimination 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  8. Indexing and Duplicate Elimination 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  9. Indexing and Duplicate Elimination 6cd412.. 12067c.. c277d6.. 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  10. Indexing and Duplicate Elimination 12067c.. 0e7a26.. Unique 6cd412.. 15ba2b.. Shared c277d6.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  11. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. 15ba2b.. Shared 6cd412.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  12. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 6cd412.. 15ba2b.. c277d6.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  13. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 6cd412.. 15ba2b.. c277d6.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  14. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 6cd412.. 15ba2b.. c277d6.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  15. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  16. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  17. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  18. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  19. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. ? f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  20. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  21. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  22. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  23. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  24. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  25. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  26. Phew. Decentralized Deduplication in SAN Cluster File Systems

  27. Evaluation Decentralized Deduplication in SAN Cluster File Systems

  28. Evaluation How much space does DeDe save? Decentralized Deduplication in SAN Cluster File Systems

  29. Evaluation How much space does DeDe save? How much overhead does DeDe introduce? Decentralized Deduplication in SAN Cluster File Systems

  30. Evaluation How much space does DeDe save? How much overhead does DeDe introduce? How fast can DeDe deduplicate? Decentralized Deduplication in SAN Cluster File Systems

  31. Space Savings: VDI Cluster Corporate Virtual Desktop Infrastructure cluster Desktop XP VM’s 6–12 months of active use ... Originally cloned from small number of base images Decentralized Deduplication in SAN Cluster File Systems

  32. Space Savings: VDI Cluster Corporate Virtual Desktop 113 VM’s Infrastructure cluster                            Desktop XP VM’s 6–12 months of active use ... Originally cloned from small number of base images   1.9 TB  Decentralized Deduplication in SAN Cluster File Systems

  33. Space Savings: VDI Cluster Corporate Virtual Desktop 113 VM’s Infrastructure cluster                            Desktop XP VM’s 6–12 months of active use ... Originally cloned from small number of base images   1.9 TB  1.3 TB Decentralized Deduplication in SAN Cluster File Systems

  34. Space Savings: VDI Cluster 1.3 TB Decentralized Deduplication in SAN Cluster File Systems

  35. Space Savings: VDI Cluster 237 GB 1.3 TB Decentralized Deduplication in SAN Cluster File Systems

  36. Space Savings: VDI Cluster 237 GB 1.3 TB 173 GB 61 GB unique shared Decentralized Deduplication in SAN Cluster File Systems

  37. Space Savings: VDI Cluster 237 GB 1.3 TB 173 GB 61 GB 2.7 GB unique shared Decentralized Deduplication in SAN Cluster File Systems

  38. Space Savings: VDI Cluster 237 GB 1.3 TB 173 GB 61 GB 2.7 GB unique shared 1.3 GB 194 MB 1.1 GB index file v. arena FS metadata Decentralized Deduplication in SAN Cluster File Systems

  39. Runtime Effects Write monitoring Disk array caching Decentralized Deduplication in SAN Cluster File Systems

  40. Runtime Effects Write monitoring Disk array caching Decentralized Deduplication in SAN Cluster File Systems

  41. Runtime Effects Write monitoring Disk array caching EMC CLARiiON CX3-40 Decentralized Deduplication in SAN Cluster File Systems

  42. Runtime Overhead: Write Monitoring Worst-case benchmark: 100% sequential write IO, No computation Decentralized Deduplication in SAN Cluster File Systems

  43. Runtime Overhead: Write Monitoring Worst-case benchmark: 100% sequential write IO, No computation Baseline Write Monitor CPU 33% 220% Decentralized Deduplication in SAN Cluster File Systems

Recommend

Decentralized Deduplication in SAN Cluster File Systems Austin T. Clements Irfan Ahmad Murali

Decentralized Deduplication in SAN Cluster File Systems Austin T. Clements Irfan Ahmad Murali

Decentralized Deduplication in SAN Cluster File Systems Austin T. Clements Irfan Ahmad Murali Vilayannur Jinyuan Li MIT CSAIL VMware, Inc. Abstract SANs provide multiple hosts with direct SCSI access to shared storage volumes. Regular

677 views • 14 slides
On the Effjciency of Decentralized File Storage for Personal Information Management Systems

On the Effjciency of Decentralized File Storage for Personal Information Management Systems

Gabriele DAngelo 2 1 Universidad Politcnica de Madrid 2 University of Bologna 3 University of Urbino Carlo Bo On the Effjciency of Decentralized File Storage for Personal Information Management Systems Mirko Zichichi 1 , 2 , Stefano

469 views • 22 slides
Profiling and diagnosing large-scale decentralized systems David Oppenheimer ROC Retreat

Profiling and diagnosing large-scale decentralized systems David Oppenheimer ROC Retreat

Profiling and diagnosing large-scale decentralized systems David Oppenheimer ROC Retreat Thursday, June 5, 2003 1 Why focus on P2P systems? There are a few real ones file trading, backup, IM Look a lot like other decentralized

685 views • 22 slides
Divide and conquer Shared disk cluster file systems shipped with the Linux kernel Udo Seidel

Divide and conquer Shared disk cluster file systems shipped with the Linux kernel Udo Seidel

Divide and conquer Shared disk cluster file systems shipped with the Linux kernel Udo Seidel Shared file systems Multiple server access same data Different approaches Network based, e.g. NFS, CIFS Clustered Shared disk,

768 views • 40 slides
Lustre at GSI - Evaluation of a cluster file system Walter Schn, GSI Walter Schn, GSI Topic

Lustre at GSI - Evaluation of a cluster file system Walter Schn, GSI Walter Schn, GSI Topic

Lustre at GSI - Evaluation of a cluster file system Walter Schn, GSI Walter Schn, GSI Topic Introduction motivation lustre test cluster Performance server controller, RAID level file systems parallel I/O

689 views • 22 slides
Evaluating selected cluster file systems with Parabench Internship report Authors: Marcel Krause;

Evaluating selected cluster file systems with Parabench Internship report Authors: Marcel Krause;

Evaluating selected cluster file systems with Parabench Internship report Authors: Marcel Krause; Jens Schlager, B.A. Tutors: Olga Mordvinova, Julian M. Kunkel Winter term 20092010 Overview Test Scenarios Test Patterns

542 views • 13 slides
Chapter 6: File Systems File systems n Files n Directories & naming n File system

Chapter 6: File Systems File systems n Files n Directories & naming n File system

Chapter 6: File Systems File systems n Files n Directories & naming n File system implementation n Example file systems Chapter 6 CS 1550, cs.pitt.edu 2 (originaly modified by Ethan Long-term information storage n Must store large amounts

1.01k views • 66 slides
Chapter 6: File Systems File systems Files Directories & naming File system

Chapter 6: File Systems File systems Files Directories & naming File system

Chapter 6: File Systems File systems Files Directories & naming File system implementation Example file systems Chapter 6 CMPS 111, UC Santa Cruz 2 Long-term information storage Must store large amounts of data

829 views • 57 slides
Efficient Locally Trackable from seed Deduplication in Replicated Systems Joo Barreto and

Efficient Locally Trackable from seed Deduplication in Replicated Systems Joo Barreto and

technology Efficient Locally Trackable from seed Deduplication in Replicated Systems Joo Barreto and Paulo Ferreira Distributed Systems Group INESC-ID/Technical University Lisbon, Portugal www.gsd.inesc-id.pt Instituto de Engenharia de

491 views • 24 slides
File Systems: All data structures are cached for better performance Create a new file

File Systems: All data structures are cached for better performance Create a new file

File Systems: Consistency Issues What about Multiple Updates? File systems maintain many data structures Several file system operations update multiple data structures Free list/bit vector Directories Examples: File headers and inode

257 views • 3 slides
an Object-Based File System for Large-Scale Federated IT Infrastructures Jan Stender, Zuse

an Object-Based File System for Large-Scale Federated IT Infrastructures Jan Stender, Zuse

an Object-Based File System for Large-Scale Federated IT Infrastructures Jan Stender, Zuse Institute Berlin HPC File Systems: From Cluster To Grid October 3-4, 2007 In this talk ... Introduction: Object-based File Systems Target Environment

525 views • 31 slides
Distributed File Systems Distributed File Systems A distributed file system (DFS) is a

Distributed File Systems Distributed File Systems A distributed file system (DFS) is a

Distributed File Systems Distributed File Systems A distributed file system (DFS) is a distributed Definitions implementation of a classical time-sharing model of a file Distributed system - A number of loosely coupled system. machines

997 views • 8 slides
File Systems: Semantics & Structure What is a File a file is a named collection of

File Systems: Semantics & Structure What is a File a file is a named collection of

5/16/2018 File Systems: Semantics & Structure What is a File a file is a named collection of information 11A. File Semantics primary roles of file system: 11B. Namespace Semantics to store and retrieve data 11C. File

373 views • 13 slides
File Systems: Semantics & Structure What is a File a file is a named collection of

File Systems: Semantics & Structure What is a File a file is a named collection of

5/15/2017 File Systems: Semantics & Structure What is a File a file is a named collection of information 11A. File Semantics primary roles of file system: 11B. Namespace Semantics to store and retrieve data 11C. File

352 views • 16 slides
Steganographic File Systems Steganographic File Systems 1 Conventional Protection Mechanisms in

Steganographic File Systems Steganographic File Systems 1 Conventional Protection Mechanisms in

Steganographic File Systems Steganographic File Systems 1 Conventional Protection Mechanisms in File S Systems t User Access Control The operating system is fully trusted to enforce the security policy. Is it good enough? Is it

888 views • 44 slides
CSN08101 Digital Forensics Lecture 8: File Systems Lecture 8: File Systems Module Leader: Dr

CSN08101 Digital Forensics Lecture 8: File Systems Lecture 8: File Systems Module Leader: Dr

CSN08101 Digital Forensics Lecture 8: File Systems Lecture 8: File Systems Module Leader: Dr Gordon Russell Lecturers: Robert Ludwiniak Objectives Investigative Process Analysis Framework last week File Systems File

795 views • 50 slides
Week 10: File Management What is a file? Elements of file management File

Week 10: File Management What is a file? Elements of file management File

CPSC 410 / 611 : Operating Systems Week 10: File Management What is a file? Elements of file management File organization Directories File allocation UNIX file system Reading: Silberschatz, Chapter 10, 11

498 views • 13 slides
CPSC 410/611: Final Week -- File Systems File Systems over a Networks: Sun NFS Aspects

CPSC 410/611: Final Week -- File Systems File Systems over a Networks: Sun NFS Aspects

CPSC 410 / 611 : Operating Systems CPSC 410/611: Final Week -- File Systems File Systems over a Networks: Sun NFS Aspects of Distributed File Systems Communication Processes Naming Synchronization Fault Tolerance Caching and

315 views • 9 slides
File Systems Chapter 11, 13 OSPP What is a File? What is a Directory? Goals of File System

File Systems Chapter 11, 13 OSPP What is a File? What is a Directory? Goals of File System

File Systems Chapter 11, 13 OSPP What is a File? What is a Directory? Goals of File System Performance Controlled Sharing Convenience: naming Reliability File System Workload File sizes Are most files small or large?

996 views • 62 slides
Deduplication CSCI 333 Spring 2019 Logistics Lab 2a/b Final Project Final Exam

Deduplication CSCI 333 Spring 2019 Logistics Lab 2a/b Final Project Final Exam

Deduplication CSCI 333 Spring 2019 Logistics Lab 2a/b Final Project Final Exam Grades 2 Last Class BetrFS [FAST 15] Linux file system using B e -trees Metadata B e -tree: path -> struct stat Data in B e

540 views • 29 slides
NFS Heterogeneous systems must be supported Different HW, OS, underlying file system

NFS Heterogeneous systems must be supported Different HW, OS, underlying file system

Distributed File Systems Distributed Systems Case Studies NFS AFS CODA DFS SMB CIFS Distributed File Systems Dfs WebDAV GFS Gmail -FS ? xFS Paul Krzyzanowski pxk@cs.rutgers.edu Except as otherwise noted,

430 views • 19 slides
File Systems: Consistency Issues 1 File Systems: Consistency Issues File systems maintain many

File Systems: Consistency Issues 1 File Systems: Consistency Issues File systems maintain many

File Systems: Consistency Issues 1 File Systems: Consistency Issues File systems maintain many data structures Free list/bit vector Directories File headers and inode structures Data blocks All data structures are cached for better

357 views • 14 slides
CPSC 410/611: File Management What is a file? Elements of file management File

CPSC 410/611: File Management What is a file? Elements of file management File

CPSC 410 / 611 : Operating Systems CPSC 410/611: File Management What is a file? Elements of file management File organization Directories File allocation UNIX file system Reading: Silberschatz, Chapter 10, 11

186 views • 14 slides
A Decentralized and Distributed E-voting Scheme Based on Cryptographic Shuffles Decentralized

A Decentralized and Distributed E-voting Scheme Based on Cryptographic Shuffles Decentralized

A Decentralized and Distributed E-voting Scheme Based on Cryptographic Shuffles Decentralized and Distributed Systems Laboratory Andy Caforio Responsible: Prof. Bryan Ford Supervision: Linus Gasser, Philipp Jovanovic 1 Way back when

445 views • 20 slides

More recommend