The Google File System Presented by: Alexa Leal Architecture the - PowerPoint PPT Presentation

The Google File System Presented by: Alexa Leal

Architecture – the basic idea Question: 1. GFS does not cache file data. Why does this design choice not lead to performance loss?

Single Master • Clients never read or write to the master • metadata is kept in memory • It has an Operation Log • Communicates with chunkservers in HeartBeat messages Question: 1. What’s the benefit of having only a single master? What’s its potential performance risk? How does GFS minimize such a risk? 2. Why is GFS’s master able to keep the metadata in memory?

Chunks & Chunkservers • Chunks are 64MB • Chunkservers communicate with client • Chunkservers keep track of their chunks and present to them to master (HeartBeat) • Allocation of new chunks uses Lazy space allocation method Questions: 1. How does GFS collaborate with chunkserver’s local file system to store file chunks? What’s lazy space allocation and what’s its benefit ? 2. How does chunkserver communication help improve the system’s performance?

Chunk Leases & Mutations • Mutation is changing of contents like a write or an append • Leases maintain a consistent mutation order across chunks for 60 seconds *example of a write

Atomic Record Appends • Client only specifies data & GFS chooses offset when appending data to file then returns that offset to the client • Appending cannot exceed chunk size • If it fails, the client will have to retry the operation

Snapshot • Instantaneously makes a copy • Master will duplicate its metadata • The snapshot will point to the same chunk as source files • Used to make branch copies

Chunk creation, re-replication, rebalancing • Chunk replicas are used for these three things • Creation – master chooses where to place the initially empty replica • Master re-replicates a chunk if available replicas fall under a specified goal • Master rebalances periodically Questions: 1. What are criteria for choosing where to place the initially empty replicas? 2. When a new chunkserver is added into the system, the master mostly uses chunk rebalancing rather than using writing new chunks to fill up it. Why?

Garbage Collection Any replica not known to master is garbage Master will remove hidden files if they have existed for 3 days Question:. How are files and chunks are deleted? What’s the advantages of the delayed space reclamation (garbage collection), rather than eager deletion?

Stale Replica Detection • A chunk replica will become stale if chunkserver fails or misses mutations • Master will remove this stale replica during garbage collection when version chunk numbers do not match

Fault tolerance & Diagnosis • Fast recovery and replication • Data integrity by checksum

Conclusion • Optimized for huge files – appending is the norm and then read sequentially • Component failures are treated as the norm • Fault tolerance by constant monitoring, replication, and recovery

Questions?