Cutting MapReduce Cost with Spot Market Huan Liu Accenture Technology Labs
Why spot market? 2
Challenge with Spot Market 3
Cloud MapReduce � Hadoop � Our prior work MapReduce App MapReduce App Cloud MapReduce Hadoop Cloud OS Amazon 4
Cloud MapReduce architecture Output queue Master reduce queue Reducer SimpleDB ….. Reduce queues Input queue Mapper 5
A map node in Spot CMR Input queue Spot CMR This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd input KV pairs Dfji adjfi adf pjdoaf Akdjiji ad jijifad inform This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd Dfji adjfi adf pjdoaf Akdjiji ad jijifad This is a script Asfjialsjdf User-defined Map Function Sfjiasdfji asjfdif Sdji sdi alkjdi fd Dfji adjfi adf pjdoaf Akdjiji ad jijifad This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd Dfji adjfi adf pjdoaf Akdjiji ad jijifad temp buffer staging buffer Shutdown This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd request Dfji adjfi adf pjdoaf One KV Previous KV Akdjiji ad jijifad Shutdown output outputs script Hypervisor Flush ….. 6 Reduce queues
Summary � First-ever MapReduce implementation that works in a spot market environment � Uses on-demand checkpointing � Dramatic cost savings 7
Recommend
More recommend
