Creation of a large system A complex undertaking Researchers often - PDF document

8/25/2012 Creation of a large system  A complex undertaking  Researchers often get to define the goals and assumptions at the same time as they architect the solution:  Many areas completely lack standards or prior systems THE ROLE OF BROAD  Many standards are completely ignored  Many widely adopted systems depart from the ARCHITECTURAL PRINCIPLES relevant standards IN SYSTEMS  Are there overarching goals that all systems share? CS6410 Ken Birman Candidate goals First steps in the design process  All systems should strive for the best possible  Developers often work in an iterative way performance given what they are trying to do  Identify and, if possible, separate major considerations  But of course the aspects of performance one measures  Pin down the nature of the opportunity they see, and will depend on the use case(s) envisioned from this refine their goals and assumptions  Aiming for performance in a way that ignores use cases  Eventually, begin to conceive of system in terms of an can yield a misleading conclusion architectural block diagram with (more or less) well- defined components and roles for each  A system should be an “elegant” expression of the  Walking through the main code paths may lead to desired solutions and mechanisms redesigns that aim at optimizing for main use cases  Puzzle: What metrics capture the notion of elegance? Critical-path driven process Example: Van Renesse (Horus)  If we can identify common patterns or use cases a-  Robbert was developing a fast multicast system priori (or perhaps by analysis of workloads from other  Core functionality: Reliable multicast from some sender similar systems for which data exists)... to some set of receivers  Permits us to recognize in advance that particular code  The particular system, Horus, implements the same paths will arise often and will really determine performance for the metrics of primary interest virtual synchrony model we discussed last week  In effect we can “distort” our design to support very short  Virtual synchrony platforms inevitably require a lot critical paths at the expense of shifting functionality of logic to deal with complexities of the real-world elsewhere, off the critical path  But how much of that logic needs to be on the  This sometimes permits us to use less optimized logic off the critical path without fear of huge performance hits critical path for common operations? 1

8/25/2012 Robbert adopted a layered approach Horus protocol: Stack of microprotocols  User sees some primitive like g.SafeSend(...)  Not every layer has work to  SafeSend uses an internal infrastructure, do with respect to every event SafeSend perhaps to obtain a snapshot of the group g.Send(m) Deliver(m) Add/Del members NewView(v)  Basic model: “events” that view, with a list of current members, locking flow up, or down the group against membership changes until SafeSend completes  By standardizing he ended up  Below this is a layer doing reliable sending, Horus Layer with a kind of mix-and-match Reliable Send flow control and retransmission within a set of Idea: Standard layers protocol architecture members Like Lego tm blocks  Below this is a layer establishing connections View Snapshot Each supports the  Below this one that discovers IP addresses... identical interface Total Ordering Physical Multicast Critical path analysis Elegant... but what about efficiency?  Robbert’s stacks often had 15 or 20 microprotocols  Robbert realized that his architecture would be evaluated heavily in terms of throughput and delay  By rearranging and changing selection he could build many kinds of higher level protocols in a standard way  Delay measured from when g.SafeSend was invoked until when delivery occurs  But many microprotocols just passed certain kinds of event through, taking no action of their own  Throughput: g.SafeSend completions per second  All of that “pass-through untouched” logic on the  Performance reflected very high overheads when he critical path slows Horus down “microbenchmarked” his solution  Isolate a component, then run billions of events through Drilling down Horus layers and “sub-layers”  What does the code inside a Horus layer do?  Steps in running a layer  Robbert had the idea of classifying the instructions 1 into three categories “Prepare” 2 “Touch Message” 3  Logic that “could” run before ever seeing the message “PostProcessing” 4  Logic that needs to see the actual message (cares about Send” 5 the bytes inside, or a sequence number, etc) 6  Logic that “could” run after the message is send  Do they need to happen in this order? 2

8/25/2012 Horus layers and “sub-layers” Success!  Send before post-processing, then prepare for next  Horus broke all records for multicast performance and Robbert got a great SIGCOMM publication Masking the Overhead of Layering. Robbert van Renesse. Proc. of 1 the 1996 ACM SIGCOMM Conf. Stanford University. August 1996. “Touch Message” 2  Links nicely to today’s theme: to what extent can we Send” 3 abstract the kind of reasoning we just saw into a set “PostProcessing” 5 of general design principles that “anyone” could “Prepare” “Prepare” 6 0 benefit from? 4  We’ll look first at the Internet level, then the O/S  (Scheme makes an “optimistic” guess” that next event will be a multicast, runs “unprepare” if guess was wrong) End-to-End arguments in System Design – Jerry H. End-to-End arguments in System Design – Jerry H. Saltzer, David P. Reed, David D. Clark Saltzer, David P. Reed, David D. Clark  Authors were early MIT Internet researchers who  Question posed: suppose we want a functionality such as “reliability” in the Internet. Where should we place the played key roles in understanding and solving Internet implementation of the required logic? challenges  Argue for “end-to-end” solutions, if certain conditions hold  Jerry H. Saltzer  Can the higher layer implement the functionality it needs?  A leader of Multics, key developer of the Internet, and a LAN (local area network) ring topology, project Athena  if yes - implement it there, the app knows its needs best  David P. Reed  Early development of TCP/IP , designer of UDP  Implement the functionality in the lower layer only if  David D. Clark  A) a large number of higher layers / applications use this functionality and implementing it at the lower layer improves the  I/O of Multics, Protocol architect of Internet performance of many of them AND  “We reject: kings, presidents and voting.  B) does not hurt the remaining applications  We believe in: rough consensus and running code.” Example : File Transfer (A to B) Example : File Transfer 6. Route packet 4. Pass msg/packet down the 7. Receive packet and buffer A B A B protocol stack msg. 5. Send the packet over the 8. Send data to the application network 1. Read File Data blocks 9. Store file data blocks 2. App buffers File Data 3. Pass (copy) data to the network subsystem 3

8/25/2012 Possible failures Would a reliable network help?  Suppose we make the network reliable  Reading and writing to disk  Packet checksums, sequence numbers, retry, duplicate  Transient errors in the memory chip while buffering elimination and copying  Solves only the network problem.  network might drop packets, modify bits, deliver  What about the other problems listed? duplicates  War story: Byte swapping problem while routing @ MIT  OS buffer overflow at the sender or the receiver  Not sufficient and not necessary  Either of the hosts may crash Solutions? Solutions? (cont.)  Introduce file checksums and verify once transfer  network level reliability would improve performance. completes – an end-to-end check .  But this may not benefit all applications  On failure – retransmit file.  Huge overhead for say Real-Time speech transmission  Need for optional layers  Checksum parts of the file. Formally stated Other end-to-end requirements "The function in question can completely and correctly  Delivery guarantees be implemented only with the knowledge and help  Application level ACKs of the application standing at the end points of the  Deliver only if action guaranteed communication system. Therefore, providing that  2 phase commit questioned function as a feature of the  NACKs communication system itself is not possible. (Sometimes an incomplete version of the function  End-to-end authentication provided by the communication system may be useful as a performance enhancement.)"  Duplicate msg suppression  Application level retry results in new n/w level packet 4