an introduction to dryadlinq
play

An Introduction to DryadLINQ Christophe Poulain Microsoft Research - PowerPoint PPT Presentation

Feb 05, 2023 •706 likes •1.27k views

An Introduction to DryadLINQ Christophe Poulain Microsoft Research Microsoft Research Virtual School of Computational Science and Engineering Big Data For Science Course, July 28, 2010 The Fourth Paradigm: Data The Fourth Paradigm: Data-

  1. An Introduction to DryadLINQ Christophe Poulain Microsoft Research Microsoft Research Virtual School of Computational Science and Engineering Big Data For Science Course, July 28, 2010

  2. The Fourth Paradigm: Data The Fourth Paradigm: Data- -Intensive Science Intensive Science http://research.microsoft.com/fourthparadigm Scientific discovery is increasingly driven by exploration of large amounts of data from many sources. Scientific breakthrough will be powered by advanced computing capabilities that help researchers manipulate and explore massive datasets. 2

  3. Data Data- -intensive computing is increasingly prevalent intensive computing is increasingly prevalent Powered by powerful multi-core workstations, readily available commodity clusters and cloud services platforms 112 containers x 2000 servers = 224000 servers Programming data analyses that scale from desktop to a large number of compute nodes remains challenging 3

  4. Dryad and DryadLINQ Dryad and DryadLINQ Research programming models for writing distributed data-parallel applications that scale from a small cluster to a large data-center. A DryadLINQ programmer can use thousands of machines, each of them with multiple processors or cores, without prior knowledge in parallel programming. 4

  5. Availability Availability Dryad/DryadLINQ on Windows HPC 2008 (SP1) is available as a free download from: http://research.microsoft.com/collaboration/tools/dryad.aspx – DryadLINQ (in source) & Dryad (in binary) – With tutorials, programming guides, sample codes, libraries, and a community site: http://connect.microsoft.com/dryad – Windows HPC Server licenses freely available through your department’s subscription to MSDN Academic Alliance 5

  6. Outline Outline • DryadLINQ programming model • Dryad and DryadLINQ overview • Applications

  7. Dryad DryadLINQ LINQ Experience Experience Use a cluster as if it were a single computer • Sequential, single machine programming abstraction • Same program runs on single-core, multi-core, or cluster • Familiar programming languages • C#, VB, F#, IronPython… • C#, VB, F#, IronPython… • Familiar development environment • .NET, Visual Studio or other IDE

  8. LINQ LINQ • Microsoft’s Language INtegrated Query – Released with .NET Framework 3.5, Visual Studio optional • A set of operators to manipulate datasets in .NET – Support traditional relational operators • Select, Join, GroupBy, Aggregate, etc. – Integrated into .NET programming languages • Programs can call operators • Programs can call operators • Operators can invoke arbitrary .NET functions • Data model – Data elements are strongly typed .NET objects – Much more expressive than SQL tables • Extremely extensible – Add new custom operators – Add new execution providers

  9. Example of a Example of a LINQ Query LINQ Query IEnumerable<string> logs = GetLogLines(); var logentries = Go through logs and keep only lines from line in logs that are not comments. Parse each where !line.StartsWith("#") line into a LogEntry object. select new LogEntry(line); var user = Go through logentries and keep from access in logentries where access.user.EndsWith(@"\ulfar") only entries that are accesses by select access; select access; ulfar. ulfar. var accesses = from access in user group access by access.page into pages select new UserPageCount("ulfar", pages.Key, pages.Count()); var htmAccesses = Group ulfar’s accesses according to from access in accesses what page they correspond to. For where access.page.EndsWith(".htm") each page, count the occurrences. orderby access.count descending select access; Sort the pages ulfar has accessed according to access frequency. 9

  10. DryadLINQ Data Model DryadLINQ Data Model .Net objects Partition PartitionedTable<T> PartitionedTable<T> implements IQueryable<T> and IEnumerable<T> PartitionedTable exposes metadata information: • type, partition, compression scheme, etc. 10

  11. A complete DryadLINQ program public class LogEntry { PartitionedTable<string> logs = PartitionedTable.Get<string>( @”file:\\MSR-SCR-DRYAD01\DryadData\cpoulain\logfile.pt” public string user; public string ip; ); public string page; var logentries = from line in logs public LogEntry(string line) { where !line.StartsWith("#") string[] fields = line.Split(' '); select new LogEntry(line); this.user = fields[8]; this.ip = fields[9]; var user = this.page = fields[5]; from access in logentries where access.user.EndsWith(@"\ulfar") } select access; select access; } var accesses = public class UserPageCount { from access in user public string user; group access by access.page into pages select new UserPageCount("ulfar", pages.Key, pages.Count()); public string page; var htmAccesses = public int count; from access in accesses public UserPageCount( string user, string page, int where access.page.EndsWith(".htm") count) { orderby access.count descending select access; this.user = user; htmAccesses.ToPartitionedTable( this.page = page; @”file:\\MSR-SCR-DRYAD01\DryadData\cpoulain\results.pt” this.count = count; } ); }

  12. ���� Executing the log query DryadLINQ for Dryad on Windows Server 2008 HPC Cluster DryadLINQ for Dryad on Windows Server 2008 HPC Cluster � 12 �

  13. MapReduce MapReduce in DryadLINQ in DryadLINQ MapReduce(source, // sequence of Ts mapper, // T -> Ms keySelector, // M -> K reducer) // (K, Ms) -> Rs { { var map = source. SelectMany (mapper); var group = map. GroupBy (keySelector); var result = group. SelectMany (reducer); return result; // sequence of Rs } 13

  14. Outline Outline • DryadLINQ programming model • Dryad and DryadLINQ overview • Applications

  15. Software Stack Software Stack … Machine Image Graph Data Learning Processing Analysis Mining Applications Other Applications DryadLINQ DryadLINQ Other Languages Other Languages Dryad CIFS/NTFS SQL Servers Azure Storage Cosmos DFS Cluster Services (Azure, HPC, or Cosmos) Windows Windows Windows Windows Server Server Server Server 15

  16. Dryad Dryad • Provides a general, flexible execution layer – Dataflow graph as the computation model – Higher language layer supplies graph, vertex code, channel types, hints for data locality, … • Automatically handles execution • Automatically handles execution – Distributes code, routes data – Schedules processes on machines near data – Masks failures in cluster and network – Fair scheduling of concurrent jobs

  17. Dryad Job Structure Dryad Job Structure Channels Input files Stage Output files sort grep awk sed perl sort sort grep grep awk sed grep sort Vertices (processes) Channel is a finite streams of items Channel is a finite streams of items • NTFS files (temporary) • NTFS files (temporary) • TCP pipes (inter-machine) • TCP pipes (inter-machine) • Memory FIFOs (intra-machine) • Memory FIFOs (intra-machine)

  18. Dryad System Architecture Dryad System Architecture data plane job manager Files, TCP, FIFO Job 1 V V V PD PD PD PD PD PD control plane Job 1 : v 11 , v 12 , … Job 2 : v 21 , v 22 , … New jobs Job 3 : … cluster scheduler

  19. ���� Fault tolerance DryadLINQ for Dryad on Windows Server 2008 HPC Cluster DryadLINQ for Dryad on Windows Server 2008 HPC Cluster � 19 �

  20. Fault Tolerance Fault Tolerance

  21. Consider an embarrassingly parallel problem Consider an embarrassingly parallel problem public static Pair<int, string> DoWork(int index) { System.Threading.Thread.Sleep(200); return new Pair<int, string>(index, System.Environment.MachineName); } public static void Main(string[] args) { int count = 50; var seeds = Enumerable.Range(1, count); var pairs = from seed in seeds select DoWork(seed); foreach (Pair<int, string> pair in pairs) { Console.WriteLine("{0} => {1}", pair.Key, pair.Value.ToString()); } } 21

  22. An embarrassingly parallel problem An embarrassingly parallel problem Many cores, one machine with PLINQ Many cores, one machine with PLINQ public static Pair<int, string> DoWork(int index) { System.Threading.Thread.Sleep(200); return new Pair<int, string>(index, System.Environment.MachineName); } public static void Main(string[] args) { int count = 50; var seeds = Enumerable.Range(1, count); var pairs = from seed in seeds .AsParallel() select DoWork(seed); foreach (Pair<int, string> pair in pairs) { Console.WriteLine("{0} => {1}", pair.Key, pair.Value.ToString()); } } 22

  23. An embarrassingly parallel problem An embarrassingly parallel problem Many cores, many machines with DryadLINQ (& PLINQ) Many cores, many machines with DryadLINQ (& PLINQ) public static Pair<int, string> DoWork(int index) { System.Threading.Thread.Sleep(2000); return new Pair<int, string>(index, System.Environment.MachineName); } public static void Main(string[] args) { { int count = 50; var seeds = Enumerable.Range(1, count); int[] ranges = seeds.Take(count - 1).ToArray(); var pairs = from seed in seeds.ToPartitionedTable("tmp.pt").RangePartition(i => i, ranges) select DoWork(seed); foreach (Pair<int, string> pair in pairs) { Console.WriteLine("{0} => {1}", pair.Key, pair.Value.ToString()); } } 23

Recommend

DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level

DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level

DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language Overview Motivation for DryadLINQ Design Implementation Performance Q &amp; A Motivation More machines + more code =

680 views • 18 slides
DryadLINQ by Yuan Yu et al., OSDI08 Ilias Giechaskiel Cambridge University, R212

DryadLINQ by Yuan Yu et al., OSDI08 Ilias Giechaskiel Cambridge University, R212

DryadLINQ by Yuan Yu et al., OSDI08 Ilias Giechaskiel Cambridge University, R212 ig305@cam.ac.uk January 28, 2014 Conclusions Takeaway Messages SQL cannot express iteration Unsuitable for machine learning, graph processing, etc.

236 views • 20 slides
On the complex network clustering using DryadLINQ Stojan Trajanovski ( st508 ) MPhil in Advanced

On the complex network clustering using DryadLINQ Stojan Trajanovski ( st508 ) MPhil in Advanced

Data Centric Networking (R202) Open source project study On the complex network clustering using DryadLINQ Stojan Trajanovski ( st508 ) MPhil in Advanced Computer Science Motivation Why going parallel in complex networks analysis? Online

390 views • 7 slides
Distributed Aggregation for Data- Parallel Computing Interfaces and Implementations Yuan Yu

Distributed Aggregation for Data- Parallel Computing Interfaces and Implementations Yuan Yu

Distributed Aggregation for Data- Parallel Computing Interfaces and Implementations Yuan Yu Pradeep Kumar Gunda Michael Isard Microsoft Research Silicon Valley Dryad and DryadLINQ Automatic query plan generation by DryadLINQ Automatic

721 views • 30 slides
DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level

DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level

DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language Arman Idani 14 Feb 2012 R202 Data Centric Networking Background Major Distributed Computing Frameworks MapReduce Dryad

494 views • 28 slides
INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

TRAINING COURSE ON SOCIAL PROTECTION &amp; FORMALIZATION TRINIDAD AND TOBAGO MARCH 15, 2017 INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION Design of the NIS Assistance from the

547 views • 23 slides
MapReduce and Dryad CS227 Li Jin, Jayme DeDona Outline Map Reduce Dryad

MapReduce and Dryad CS227 Li Jin, Jayme DeDona Outline Map Reduce Dryad

MapReduce and Dryad CS227 Li Jin, Jayme DeDona Outline Map Reduce Dryad Computational Model Architecture Use cases DryadLINQ Outline Map Reduce Dryad Computational Model Architecture Use cases

1.29k views • 62 slides
INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

Patterns of variation in the expression of case and agreement Andrs Brny Leiden University Centre for Linguistics 9 November 2019, BaSIS Workshop a.barany@hum.leidenuniv.nl INTRODUCTION Introduction 2/42 INTRODUCTION

1.37k views • 42 slides
Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures Definition and concepts Process and product properties Principles SE Approaches Motivation Software and the economy The economies of

1.03k views • 49 slides
Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1 https://www.cdc.gov/std/stats17/infographic.htm Introduction Epidemiology in Western Colorado CPHE Surveillance Data Report, May 2018 Chlamydia Epidemiology in Western

977 views • 39 slides
DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every meeting (2) Some ideas of diagrams for section 7.1 (Introduction) of TP. Giles Barr 25 Jan 2018, Pembroke College On DUNE, the reality is there is a

474 views • 10 slides
INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

ADVANCED COMPUTER SECURITY INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT IS SECURITY? A systems security policies describe What the system is supposed to do Store and provide access

479 views • 16 slides
Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout World Wide

387 views • 14 slides
14 Introduction Introduction Bad guys can put malware into Example: hosts

14 Introduction Introduction Bad guys can put malware into Example: hosts

Introduction Introduction source Encapsulation ISO/OSI reference model message M application segment H t H t M transport network datagram H n H n H t M presentation: allow applications to frame link H l H n H t M interpret

583 views • 4 slides
Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout

322 views • 8 slides
Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Introduction Zynq Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL Processing System Processor Peripherals Data Buses AXI Bus Conclusion BTE5380 - Embedded Systems Octobre 2014 Andreas Habegger

1.81k views • 30 slides
2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04 Project References S E 5 PART.1 SMILE5 Introduction M I L Introduction STORION-SMILE5 Residential Energy Storage System Inverter 5kW Output Input

1.06k views • 25 slides
Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an application server? Why use an application server? Course introduction Services provided by CICS How CICS applications are defined Scaling CICS to meet

1.84k views • 146 slides
Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic building blocks Node Links (relationship between nodes) Spatial information Network data

584 views • 9 slides
Introduction Introduction (1) Main argument of the paper: universities are not only

Introduction Introduction (1) Main argument of the paper: universities are not only

Le jugement des pairs comme outil de management des universits Peer-review as a management tool Christine Musselin (CSO, Sciences Po et CNRS) Avril 2013, Lirrsistible ascension du capitalisme acadmique Introduction Introduction (1)

590 views • 28 slides
JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of the brand new JA-100 alarm system Visions Basic architecture System components Setting Help us with final validations You are the FIRST ones

1.69k views • 111 slides
Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Moreno Baricevic Gilberto Daz Axel Kohlmeyer Stefano Cozzini ULA ICTP CNR-IOM DEMOCRITOS Merida, VENEZUELA Trieste, ITALY Trieste, ITALY Introduction Introduction to storage and to storage and filesystems filesystems Introduction

1.19k views • 51 slides
1 Introduction Introduction Communication System Baseband an adjective that describes

1 Introduction Introduction Communication System Baseband an adjective that describes

Introduction Introduction Physical Layer Mobile network Global ISP Provides the means to transmit bits Basic Concepts of from sender to receiver, Data Transmission Defines the mechanical, electrical, and Home network timing

463 views • 7 slides
Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical

Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical

INFOGR Computer Graphics Jacco Bikker - April-July 2015 - Lecture 1: Introduction Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical Details Assignments Field Study State of

869 views • 62 slides

More recommend