P2P, DSM, and Other Products P2P, DSM, and Other Products from the Complexity Factory from the Complexity Factory Willy Zwaenepoel Willy Zwaenepoel EPFL EPFL
Impact of Research Impact of Research Not so great Not so great � Many research ideas have lost out Many research ideas have lost out � � Many non Many non- -research developments won out research developments won out �
Impact of Research Impact of Research Not so great Not so great � Many research ideas have lost out Many research ideas have lost out � � Many non Many non- -research developments won out research developments won out � Why is that? Why is that? � We make things too complex We make things too complex � � Note: not: things are too complex Note: not: things are too complex �
Impact of Research Impact of Research Not so great Not so great � Many research ideas have lost out Many research ideas have lost out � � Many non Many non- -research developments won out research developments won out � Why is that? Why is that? � We make things too complex We make things too complex � � Not: things are too complex Not: things are too complex � Why? Why? � Publishing/reviewing pushes us to complexity Publishing/reviewing pushes us to complexity �
Apologies, Caveats and Excuses Apologies, Caveats and Excuses Talk is rather polemic in nature Talk is rather polemic in nature … things are said a little crassly things are said a little crassly … Now a dean – – intellectual life prohibited intellectual life prohibited Now a dean � “ “There was once a dean who was so dumb, There was once a dean who was so dumb, � that other deans actually started noticing it” ” that other deans actually started noticing it
P2P P2P Peer- -to to- -peer peer Peer No (central) server No (central) server Easier to operate, maintain, scale, make Easier to operate, maintain, scale, make more reliable … … more reliable Started as an application Started as an application Proposed as an infrastructure for a large Proposed as an infrastructure for a large number of applications number of applications
Research on P2P Research on P2P Concentrated largely on DHTs DHTs Concentrated largely on Log(n) access ) access Log(n Chord, Pastry, … … Chord, Pastry, Applications: backup, streaming, … … Applications: backup, streaming,
The Problem with P2P The Problem with P2P Very little application other than illegal file Very little application other than illegal file sharing sharing
Reality Check Reality Check If we have learned anything about If we have learned anything about distributed computing over the last 25 distributed computing over the last 25 years, it is that anything distributed is years, it is that anything distributed is harder than anything centralized harder than anything centralized
Reasons for Distribution Reasons for Distribution You cannot handle it in one place You cannot handle it in one place � Performance Performance – – controlled replication controlled replication � � Availability Availability – – controlled replication controlled replication � Geographical distribution Geographical distribution � Google! Google! � Illegality – – P2P P2P Illegality � From Napster to Gnutella, From Napster to Gnutella, Kazaa Kazaa, , … … � � “ “Raw Raw” ” traffic numbers are high traffic numbers are high � � Much of it static Much of it static � � Could be handled by conventional replication (?) Could be handled by conventional replication (?) �
Difficulties for P2P Difficulties for P2P Hard to find anything Hard to find anything Hard to make anything secure Hard to make anything secure � Open invitation to attack Open invitation to attack � � Actively used by RIAA (pollution attacks) Actively used by RIAA (pollution attacks) � Hard to write anything Hard to write anything
Advantages for P2P Research Advantages for P2P Research Complex to find anything Complex to find anything Complex to make anything secure Complex to make anything secure Complex to write anything Complex to write anything
Advantages for P2P Research Advantages for P2P Research Complex to find anything Complex to find anything Complex to make anything secure Complex to make anything secure Complex to write anything Complex to write anything Complexity begets papers Complexity begets papers P2P = Paper- -to to- -Paper Paper P2P = Paper
There are Applications There are Applications Large file multicast Large file multicast Can be be handled handled by by very very simple techniques simple techniques Can � BitTorrent BitTorrent � It should should worry worry us us that that these these come come from from It non- -research research corners of the world! corners of the world! non
DSM DSM Distributed shared memory Distributed shared memory Parallel computing on clusters Parallel computing on clusters Distributed memories abstracted as a Distributed memories abstracted as a single shared memory single shared memory Easier to write programs Easier to write programs Usually by page faulting Usually by page faulting TreadMarks ( (ParallelTools ParallelTools) ) TreadMarks
Reality Check Reality Check Clusters are only suitable for coarse- - Clusters are only suitable for coarse grained parallel computation grained parallel computation A fortiori true for DSM A fortiori true for DSM
Problems with Fine- -Grained DSM Grained DSM Problems with Fine Expensive synchronization Expensive synchronization Expensive fine- -grained data sharing grained data sharing Expensive fine � Smaller than a page Smaller than a page � False sharing (can be solved) False sharing (can be solved) True sharing True sharing
Advantages for DSM Research Advantages for DSM Research Complex fine- -grain synchronization grain synchronization Complex fine Complex fine- -grain data sharing grain data sharing Complex fine � Compiler, language, runtime, Compiler, language, runtime, … … � Complexity begets papers … … Complexity begets papers
TreadMarks TreadMarks (Almost) every paper or grant for research (Almost) every paper or grant for research on fine- -grain DSM was accepted grain DSM was accepted on fine (Almost) every paper or grant for research (Almost) every paper or grant for research on coarse- -grained DSM was rejected grained DSM was rejected on coarse It turns out that for real applications a page It turns out that for real applications a page is not large enough! is not large enough!
Coarse- -grain grain Applications Applications Coarse Large (independent independent) ) units units of computation of computation Large ( Large chunks chunks of data of data Large � 1 page = 4k 1 page = 4k � � Not Not very very large large at at all all � � Page Page faulting faulting brings brings in one page in one page at at a time a time � � Message passing Message passing brings brings in in whole whole data segment data segment at at a a � time (> page) time (> page) Can be be and and was was done done with with DSM DSM Can � Increase Increase page size (!!) page size (!!) � � Compiler support Compiler support �
Competition is is Message Passing Message Passing Competition MPI (Message Passing Interface) MPI (Message Passing Interface) Low abstraction abstraction Low No room for complexity complexity fabrication fabrication No room for As a result result more more successful successful As a It should should worry worry us us that that MPI MPI did did not come not come It from distributed distributed systems systems research research but but from from from linear algebra algebra! ! linear
Server Performance Server Performance At the beginning of the Internet boom, At the beginning of the Internet boom, server performance was badly lagging server performance was badly lagging Multithreaded or multiprocess servers Multithreaded or multiprocess servers � Context switching Context switching � � Locking Locking � Two types of solutions Two types of solutions � Exokernel Exokernel � � Event Event- -driven servers driven servers �
Event- -Driven Servers Driven Servers Event Events Events � Incoming request, i/o completion, Incoming request, i/o completion, … … � Single thread, event loop Single thread, event loop Event handler per event Event handler per event � Straight code (no blocking) Straight code (no blocking) � � At end: At end: � nonblocking or asynchronous i/o or asynchronous i/o nonblocking create (hand- -made) continuation made) continuation create (hand
Advantages Advantages No multithreading No multithreading � No No context context switching switching � � No No locking locking ( (at at least on least on uniprocessor uniprocessor) ) � Control over order order of of event event handling handling Control over � Not Not bound bound by OS by OS scheduler scheduler �
Flash Flash Most popular popular event event- -driven driven Web server Web server Most Combined multithreaded multithreaded / / event event- -driven driven Combined Many follow follow- -ons ons Many iMimic Networking Networking iMimic
Recommend
More recommend
