The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun- - PowerPoint PPT Presentation

The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun- Tak Leung SOSP 2003 Presented by Wenhao Xu University of British Columbia

Outline • Background • Assumption and Google Workloads • Architecture • System Interaction • Master • Relaxed Consistency Model • Experiment • Conclusion • Discussion

Background • In 2003, – 50+ GFS clusters – Each with thousands of storage nodes – Managing petabytes of data – GFS is under lots of applications and other systems in Google, such as bigtable, etc. • Built on top of commodity machine and disks Tailored for Google workloads

Assumption & Google Workloads High availability: • Failures occur everywhere Replicas, Heartbeat, Fast recovery, – Disks, Network, Machines checksums • Store a modest number of large files Large data chunk & No – A few million files; Each typically 100MB or Client data cache larger in size Throughput is more • Read: Large streaming reads and small important random reads – Typically read hundreds of KBs of an individual Make appending fast operation, more commonly 1MB or more Relaxed consistency – Small reads typically read a few KBs at arbitrary model & Atomicity offsets, which are usually batched and sorted to Record Append avoid seeking back and forth Customizable API & • Mutation: Mostly appending new data Don’t need to conform to POSIX semantic! • Concurrent clients • Design flexibility: Co-designing the application and the file system API

Architecture Single Master A Chunk • Single point of Failures?  Shadow masters, operation log • Size: 64MB • Performance bottlenecks?  Less • 64 bit handler involvment • Simplicity: concurrency control, • Stored as a Linux file, metadata management, etc. lazy space allocation • Global knowledge: make sophisticated • 3+ replicas placement and replication decisions.

System interaction – Mutation Data flow • Leases – Primary replica • Master is only involved in 1 & 2 • Separated data flow and control flow – Employs network topology to transfer data – Easily to control the operation order

System Interaction - Heartbeat • What is a heartbeat message like? – “hey, I am still healthy. And I have chunks a, b, c …. Their versions are v1, v3, v9, …” • What is heartbeat for? – Chunkserver status – Chunk placement – Garbage collection – Stale Replica deletion – Lease extension

Master - Metadata Management • Three major types – File and chunk namespaces – (File, chunk) mapping – Location of each chunk’s replicas • All metadata is kept in the memory – Possibility: (<64B/chunk, 2 26 (64M) * 64 = 4G, a few millions of files) – Fast access & scanning • Operation log – Store First two types of metadata – Any changes to the first two types of metadata are first written to the operation log – “It is replicated on multiple remote machines and respond to a client operation only after flushing the corresponding log record to disk both locally and remotely” – Checkpoints are made to avoid larger log and long starting up time for masters.

Master – Namespace Management & locking • Namespace is organized into a lookup table – (file name, metadata) mapping – Does not support aliases for the same file or directory • Mutation lock – Read lock of all parents directory – Write lock of mutation directory – E.g: /d1/d2/…/dn/leaf creation • Read lock: /d1, /d1/d2, …, /d1/d2/…/dn • Write lock: /d1/d2/…/dn/leaf

Master - Replica Creation and Placement • Consideration – Balance disk space utilization – Performance: Limit the number of “recent” creations (hot pot) on each chunkserver – Availability: different racks; as far as possible – Maximize network utilization: as less switches as possible; exploit the aggregate bandwidth of multiple racks; – Multiple racks: • Tradeoff: read bandwidth & write traffic • Re-replication & rebalancing

Relaxed Consistency model • Two state definitions: – Consistent: All the clients will always see the same data, regardless of which replicas they read from – Defined: Consistent & Clients will see what the mutation writes in its entirety • The state after mutations depends on – whether the ops succeeds or fails – Whether there are concurrent mutations • Move work to applications/clients – Error code  check consistency – Application checksum  check if it is defined – Identifier for record  figure out duplicates – Retry • File namespace mutation are atomic – Handled exclusively by the master

Atomic Record Appends • Used heavily by Google applications • Clients specify data – Not specify offset in files • GFS appends it to the files – At least once atomically – Successfully written regions are defined – Does not guarantee that all replicas are identical • Intervening regions are inconsistent

Experiments

Conclusion • Gfs is designed for Google applications • It is simple, but good enough for Google applications • It is important for Google to store data for other applications.

Discussion (1/5) • It is designed specifically to serve search needs. For what other applications this filesystem is suitable for. And for what applications, it is not optimal? • Reply – Depends on: • Major file size (small or large) • Major file reads (streaming or not) • Major writing modes (appending or not)

Discussion (2/5) • When the Master decides on which replicas should a particular chunk goes in, shouldn't it consider the potential applications that is going to access that chunk ? – Reply: Not really. More replicas if it is hot spot. • We don't want to see unbalanced disk usages. In the paper it mentioned that it "rebalances" for load balancing but a way of doing it is not mentioned. – Reply : Master knows the disk usages of all chunkservers • In case of a shadow master is operating, it does not provide any mutation operations. But the processes that got lease from the primary before primary got crashed will continue on writing to the disks right ? So how can we keep track of those writes that is taken place while the shadow master is in operation ? – Reply : It can run until finished.

Discussion (3/5) • What happens in GFS if the primary master is down? Will one of the “shadow” masters become a new primary master? If so, how to pick one from several “shadow” ones? – Reply : Maybe. • Will the centralized master-server architecture degrade the GFS's scalability? – Reply : Seems not under Google’s workload. • GFS uses the "closest" client for replication. However, the term "closest" is never explained - is it based on network latency, or does it factor in network load on the closest chunkserver, does it factor in the system latency? How does one chunkserver figure out the closest server on the fly? – Reply : Deduced from IP addresss

Discussion (4/5) • The concept of 'records' appears frequently in the paper. GFS can guarantee that a record is appended atomically, and can associate an application defined checksum with each record. This seems to imply that GFS keeps metadata for each record. How is this done? – Reply : The checksum is in the record data, not in the metadata. • Clients cache chunk locations, so if one chunk server cannot contact the master it may unwittingly serve stale data to clients. Comments? – Reply : The master includes the chunk version number when it informs clients.

Discussion (5/5) • What do you think about the performance gain of “Concurrent appends to the same file operation” and the “relaxed” consistency model under normal workload? (I believe this is only suits well to google). Since appending is key to GFS write performance in a multi-writer environment, it might be that GFS would give up much of its performance advantage even in large serial writes. What do you think about this? • Reply: – It is a tradeoff between complexity between GFS and its clients. – I think it is good for performance. – It is tailored for google’s workloads (appending).

Thanks for your attention!