CSE 6350 File and Storage System Infrastructure in Data centers Supporting Internet-wide Services MapReduce: Simplified Data Processing on Large Clusters Presenter: Uday Sagar Adusumalli
Outline Introduction • MapReduce – Overview • Programming Model • Implementation • Backup Tasks •
Introduction Google have implemented number of special-purpose computations to process • large amount of raw data. The input data is large and the computations have to be distributed across thousands • of machines in order to finish in reasonable amount of time. In dealing with huge data and complex code, we faces issues to parallelize the • computation, distribute the data and handling failures. To reduce this complexity, a new design that helps to express simple computations • that hides messy details of parallelization, fault-tolerance, data distribution and load balancing.
MapReduce - Overview MapReduce is a programming model and a associated implementation for • processing and generating large data sets. MapReduce contains two user defined functions Map and Reduce. • Both Map and Reduce functions changes based on environment and • implementation. Map function processes key/value pairs to generate intermediate set of key/value • pairs. Reduce function processes the intermediate key/value pair set to merge all the • intermediate values associated with the same intermediate key. MapReduce automatically parallelizes and executed on a large cluster of • commodity machines.
Q(1) Compared with traditional parallel programming models, such as multithreading and MPI, what are major advantages of MapReduce? Scalability • Flexibility • Cost Effective • Simple Model of Programming • Security and Authentication • Each worker has their own data set (No sharing) • No deadlocks • Large datasets •
Programming Model User of MapReduce library expresses the computations as two functions: • Map and Reduce User defined Map function, takes input key/Value pairs and produces a set • of intermediate Key/Value pairs. The MapReduce Library takes the intermediate Values associated with the • same intermediate key I and groups them accordingly and sends to Reduce function. User defined Reduce function, takes the intermediate key I and all the • values associated with key I and merges these values to form a relatively small set of values. Generally zero or one output value is produced per Reduce Invocation. • map (k1,v1) → list(k2,v2) reduce (k2,list(v2)) → list(v2)
Example – Word Count • Input consists of (url, contents) pairs • map(key=url, val=contents): – For each word w in contents, emit(w,”1”) • reduce(key=word, values= uniq_counts): – Sum all “1”s in values list – Emit result “(word, sum)” See 1 Bob 1 Bob 1 Throw 1 Throw 1 See 2 See Bob Throw See 1 Spot 1 See Spot Run Spot 1 Run 1 Run 1
Example – Word Count In the above example we are counting the occurrences. of each word from an article. • First the master assigns input article to each mapper machine. Here master assigns an • input article containing input text as “See Bob Throw See Spot Run”. The mapper machine reads each word from the input article and treats each word as a “key”. It then assigns values to each word(key). Here mapper assigns value “1” to each key • word. These key /value pairs are known as intermediate key/value pairs. Then these intermediate key values are stored in memory buffer. Master sends • locations of these intermediate key/value pairs to reducer machines based on their hash values. The reducer then sorts and groups these intermediate key/value pairs such that the • values for the same intermediate keys are grouped together and there are added. Like we have 2 occurrences of key word “See” in our input article. So that our output file from reducer contains value as “2” for the kay “See”. The remaining key’s values will be “1” since they are not repeated. And these output files are saved on GFS.
Implementation
Q(2) Use Figure 1 to explain a MR program’s execution. 1. MapReduce Library splits input files into M pieces(16 to 64 MB each). It makes copies of program on a cluster of machines(Master and Workers) 2. Total M map tasks and R reduce tasks. Master picks idle workers and assign them map or reduce tasks. 3. Worker with map task reads input split of key/value pairs and passes it the user’s map function. Intermediate key/value pairs are generated and are buffered in the memory. 4. These buffered pairs are written to local disk and partitioned into R regions by partitioner. Master receives locations of these regions and forwards these locations to reduce workers. 5. Reducer worker receives locations from Master and then it reads buffered data from the local disks. Reducer worker sorts all immediate keys so that all the values with same key are grouped together. 6. The reducer worker passes the intermediate keys and corresponding set of values to the Reducer function. The output of the Reduce function is appended to a final output file for this reduce partition. 7. When all map tasks and reduce tasks have been completed, the master wakes up the user program.
Q(3) Describe how MR handles worker and master failures. Worker Failure Master pings each worker periodically. If no response in certain amount of time, it marks • worker as failed. Once a Mapper or Reducer finishes task they are marked as idle. • In-progress failed workers are mapped as idle for future use. • Completed map tasks are re-executed on a failure because their output is stored on the • failed machine and it is inaccessible. Completed reduce task is not re-executed because its output is stored on global file system • Master Failure Master write periodic checkpoints of the master data structures(idle, in-progress , • completed). There is only one Master, its failure is unlikely. • If Master fails, the current implementation aborts the MapReduce computation. • Client checks the condition and retries the MapReduce operation. •
Q(4) The implementation of MapReduce enforces a barrier between the Map and Reduce phases, i.e., no reducers can proceed until all mappers have completed their assigned workload. For higher efficiency, is it possible for a reducer to start its execution earlier, and why? (clue: think of availability of inputs to reducers) No, it is not possible for a reducer to start its execution until all mappers have • completed their assigned workload. Reducer’s input is grouped by the intermediate key that is produced by the last • Mapper. So without receiving the input key(intermediate key) from the Mapper, no reducer • can start execution.
Backup Tasks Straggler : Any machine that takes long time to complete a task • Many reasons : Bad disk, Network bandwidth, CPU competition • Solution: Master schedules backup executions for in-progress tasks • Task is completed when either primary or backup execution completes •
Thank You
Recommend
More recommend
