CSC 369: Distributed Computing Alex Dekhtyar May 6 Day 14: Java - PowerPoint PPT Presentation

CSC 369: Distributed Computing Alex Dekhtyar May 6 Day 14: Java Hadoop API

CSC 369: Distributed Computing Alex Dekhtyar May 6 Day 14: Java Hadoop API HAPPY EQUATOR DAY!

Housekeeping Lab 4 (mini-project): due Sunday night Lab 5: due tonight (grace period tomorrow) Lab 6: full lab coming out Friday Grading: slowly happening…

Hadoop Java API

Hadoop API Current Version is 3.2.1. hadoop Command-line tools hdfs We limit ourselves to hadoop jar yarn

Hadoop Java API org.apache.hadoop Let’s concentrate on things we absolutely need

Hadoop Java API org.apache.hadoop Core MapReduce classes org.apache.hadoop.mapreduce Inuput/Output org.apache.hadoop.mapreduce.lib.input parsing org.apache.hadoop.mapreduce.lib.output atomic type wrappers org.apache.hadoop.io Job configuration org.apache.hadoop.conf File system classes org.apache.hadoop.fs

org.apache.hadoop.mapreduce MapReduce Job org.apache.hadoop.mapreduce.Job org.apache.hadoop.mapreduce.Mapper Extensible Mapper org.apache.hadoop.mapreduce.Reducer Extensible Reducer Parent class for org.apache.hadoop.mapreduce.Partitioner Partitioning tasks org.apache.hadoop.mapreduce.InputFormat Parent classes for org.apache.hadoop.mapreduce.OutputFormat Input/Output Formats Parent class for org.apache.hadoop.mapreduce.InputSplit Input Split

How it works Input File

How it works InputSplit InputSplit InputSplit Input File

How it works Job InputSplit Mapper Combiner (Reducer) InputSplit Reducer InputSplit Input File

How it works Compute Node1 Job InputSplit Mapper Combiner (Reducer) Compute Node2 InputSplit Reducer InputSplit Compute Node3 Input File

How it works Compute Node1 InputSplit Combiner (Reducer) Mapper Compute Node2 InputSplit Combiner (Reducer) Mapper InputSplit Compute Node3 Combiner (Reducer) Mapper Mapper Input File

Compute Node1 InputSplit Mapper Combiner (Reducer)

time MAP STAGE Reduce STAGE Compute Node1 Compute Node1 Mapper Reducer Combiner Compute Node2 Compute Node2 Mapper Reducer Combiner Compute Node3 Compute Node3 Reducer Mapper Combiner

time MAP STAGE Reduce STAGE Compute Node1 Compute Node1 Partitioner Mapper Reducer Combiner Compute Node2 Compute Node2 Partitioner Mapper Reducer Combiner Compute Node3 Compute Node3 Partitioner Reducer Mapper Combiner

time MAP STAGE Reduce STAGE Compute Node1 Compute Node1 Partitioner Mapper Reducer Combiner Compute Node2 Compute Node2 Partitioner Mapper Reducer Combiner Compute Node3 Compute Node3 Partitioner Reducer Mapper Shuffle STAGE Combiner

Mapper in a nutshell protected void setup(org.apache.hadoop.mapreduce.Mapper.Context context) protected void map(KEYIN key, VALUEIN value, org.apache.hadoop.mapreduce.Mapper.Context context) protected void cleanup(org.apache.hadoop.mapreduce.Mapper.Context context) void run(org.apache.hadoop.mapreduce.Mapper.Cont ext context)

run(InputSplit s, Context c): Run setup() once setup(s,c); for each record in s do: Run map() for each record map(record, c); end for; cleanup(s,c) Run cleaunp() once

Reducer in a nutshell protected void setup(org.apache.hadoop.mapreduce.Mapper.Context context) protected void reduce(KEYIN key, Iterable<VALUEIN> value, org.apache.hadoop.mapreduce.Mapper.Context context) protected void cleanup(org.apache.hadoop.mapreduce.Mapper.Context context) void run(org.apache.hadoop.mapreduce.Mapper.Cont ext context)

Shuffle Sort SecondarySort run(InputSplit s, Context c): Run setup() once setup(s,c); Run map() for each record for each record in s do: map(record, c); end for; Run cleaunp() once cleanup(s,c)

Hadoop Java API org.apache.hadoop Core MapReduce classes org.apache.hadoop.mapreduce Inuput/Output org.apache.hadoop.mapreduce.lib.input parsing org.apache.hadoop.mapreduce.lib.output atomic type wrappers org.apache.hadoop.io Job configuration org.apache.hadoop.conf File system classes org.apache.hadoop.fs

org.apache.hadoop.mapreduce.lib.input Single File Input Format Generic Input File format (others extend it) FileInputFormat Text Input TextInputFormat User-defined Key-Value Pairs KeyValueInputFormat Fixed Length Records in input FixedLengthInputFormat NLineInputFormat Controls the size of split (in terms of #lines)

org.apache.hadoop.mapreduce.lib.input Single File Input Format Generic Input File format (others extend it) FileInputFormat Text Input TextInputFormat User-defined Key-Value Pairs KeyValueInputFormat Fixed Length Records in input FixedLengthInputFormat Controls the size of split (in terms of #lines) NLineInputFormat Other Important Classes Multiple Files as inputs to a single Mapper MultipleInputs File Partitions FileSplits

CSC 369: Distributed Computing Alex Dekhtyar May 6 Day 14: Java - PowerPoint PPT Presentation

CSC 369: Distributed Computing Alex Dekhtyar May 6 Day 14: Java Hadoop API CSC 369: Distributed Computing Alex Dekhtyar May 6 Day 14: Java Hadoop API HAPPY EQUATOR DAY! Housekeeping Lab 4 (mini-project): due Sunday night Lab 5: due

Advanced Architectures Goals of Distributed Computing better services 15A. Distributed

Distributed Computing Environments 1 Distributed computing environment consists of entities

61A Lecture 35 Characteristics of distributed computing: Computers are independent they do

Advanced Architectures Goals of Distributed Computing better services 15A. Distributed

Algorithms in Nature Distributed computing Example Distributed systems Internet ATM

On safety in distributed computing Srivatsan Ravi On safety in distributed computing Safety in

Collaborative Human Computing Zack Zhu March 31, 2010 Seminar for Distributed Computing 1

CSC 369: Distributed Computing Alex Dekhtyar Day 1: Welcome Syllabus Teaching and

Distributed Computing Programs that cooperate and communicate over a network E-mail

Historical perspective From sequential computing to distributed computing and ...

Coordinating distributed systems Marko Vukoli Distributed Systems and Cloud Computing Previous

Self-Stabilization in Distributed Systems Course: Distributed Computing Faculty: Dr. Rajendra

DISTRIBUTED SYSTEMS Department of Computing Science Umea University Distributed Systems - D N

Distributed Quantum Computing Using the most precise description of our world to solve some of

ATLAS Worldwide Distributed Computing ATLAS Worldwide Distributed Computing Zhongliang Ren 03

Distributed Algorithms Tutorial Roger Wattenhofer ETH Zurich Distributed Computing

A Web-Based Platform for Publication and Distributed Execution of Computing Applications Oleg

Consensus in Distributed Systems Course: Distributed Computing Faculty: Dr. Rajendra Prasath

Distributed Mutual Exclusion Algorithms Course: Distributed Computing Faculty: Dr. Rajendra

Computing While Charging Building a Distributed Computing Infrastructure Using Smartphones

LHCb Computing Computing LHCb Nick Brook Organisation LHCb software Distributed

Distributed Computing: Distributed Objects/Services Manish Parashar parashar@ece.rutgers.edu

Selected Topics in Cloud Computing Marko Vukoli Distributed Systems and Cloud Computing This

Bisimulation and Modal Logic in Distributed Computing Tuomo Lempi ainen Distributed Algorithms