Communication • No-wait send • Receive with timeout • Messages are commands with zero or more arguments • Messages are sent to ports • Ports are named and typed, defining the messages they accept • Guardians have one or more ports Thursday, October 17, 13 41
Primitives for Distributed Computing • Paper finishes by showing an example of a transactional airline reservation system • One conclusion: more experience with distributed programming needed Thursday, October 17, 13 42
App-to-App Protocols • ARPANET, forerunner of the Internet, started operating in late 1969 • Early host-to-host protocols facilitated human-to-computer communications • Email in 1971 • FTP and interoperable Telnet in 1973 • Interest started growing in application-to- application protocols Thursday, October 17, 13 43
RFC 707 • J. E. White, RFC 707, “A High-Level Framework for Network-Based Resource Sharing, 1975 • Expressed concerns that programmers • didn't know how to write distributed applications • but did know how to write libraries • so, why not make distributed programming look just like library programming? Thursday, October 17, 13 44
RFC 674 and 707 • RFCs 674 and 707 basically define what would become the remote procedure call (RPC) • But questioned in RFC 684: "While the procedure call may be an appropriate basis for certain applications, we believe that it can neither directly nor accurately model the interactions and control structures that occur in many distributed multi-computer systems." —R. Schantz, RFC 684 Thursday, October 17, 13 45
See Also • Carl Hewitt's Actor Model • Smalltalk-72 • Robin Milner's Calculus of Communicating Systems • Bruce Lindsay's "Notes on Distributed Databases" • Anything by Jim Gray (in any decade) Thursday, October 17, 13 46
1980s Thursday, October 17, 13 47
Some 1980s Issues • Languages for distributed programming • Operating systems • Safety and liveness • Consensus Thursday, October 17, 13 48
Languages for Distribution • Much research in this period focused on whole programming languages and runtimes • Even whole systems consisting of unified programming language, compiler, and operating system Thursday, October 17, 13 49
Languages for Distribution • RPC was a key abstraction • Significant focus on uniformity : • local/remote transparency • location transparency • strong/static typing across the system Thursday, October 17, 13 50
Languages for Distribution • Specialized, closed protocols • Protocols were rarely the focus of research efforts, publications almost never mentioned them • Protocol was viewed as part of the RPC “black box,” hidden between client and server RPC stubs Thursday, October 17, 13 51
Liskov's Argus • Integrated programming language and system • Extension of CLU • Designed to help with reliability issues (partitions, downed nodes) • Included atomic actions to support consistency Thursday, October 17, 13 52
Xerox Cedar Programming Environment • Gave us 1984 Birrell/Nelson paper "Implementing Remote Procedure Calls" Thursday, October 17, 13 53
Interface Definition Language (IDL) • Declarative language used to define remote interface functions and types • Translated/mapped into specific programming language stubs and type definitions • There are many IDLs, not sure of the original Thursday, October 17, 13 54
IDL • In mid-80s Apollo Computer used an IDL to define system interfaces, then translated into C and Domain Pascal • Kept definitions for C and Pascal in sync • Apollo Network Computing System (NCS) also used the IDL to define remote interfaces • NCS was a forerunner of the Distributed Computing Environment (DCE) Thursday, October 17, 13 55
Some Apollo Trivia • Apollo Aegis and Domain/OS provided a native networked file system (not bolted on later) • Access to a file on host "foo" from any other host: //foo/path/to/file • Sir Tim Berners-Lee told me he later borrowed the Apollo "//" to use in URLs • Microsoft Universal Naming Convention (UNC) path uses "\\", likely due to Paul Leach who left HP/Apollo for Microsoft in 1991 Thursday, October 17, 13 56
Emerald • distributed RPC-based object language • local/remote transparency • object mobility Thursday, October 17, 13 57
Erlang • Programming language/system invented in the mid-80s at Ericsson by Joe Armstrong • Provides reliability via concurrency and distribution • Useful features, reasonable trade-offs, clear influence from work preceding it • Open source, available at erlang.org • My favorite programming language Thursday, October 17, 13 58
Vector Clocks • Independently discovered by Mattern and Fidge • Instead of just transmitting clocks or timestamps, keep a vector of clocks, one for each process • Lamport timestamps can't prove causality, vector clocks can Thursday, October 17, 13 59
Vector Clocks from Fidge "Timestamps in Message Passing Systems That Preserve the Partial Ordering" Thursday, October 17, 13 60
Consensus • Coordination and reliability • getting processes to agree • even if some are faulty or unavailable • or even if some are malicious Thursday, October 17, 13 61
Byzantine Generals • Lamport, 1982 • Proves how to achieve consensus in the presence of malicious processes Thursday, October 17, 13 62
FLP Impossibility • Fischer, Lynch, Paterson paper, 1983 • Proves that in asynchronous systems, reaching consensus in bounded time can be impossible with just one fault • Uses proof by contradiction • See also Nancy Lynch's "A Hundred Impossibility Proofs for Distributed Computing" Thursday, October 17, 13 63
Why Impossibility? "What good are impossibility results, anyway? They don't seem very useful at first... Most obviously, impossibility results tell you when you should stop trying to devise or improve an algorithm." —Nancy Lynch http://groups.csail.mit.edu/tds/papers/Lynch/podc89.pdf Thursday, October 17, 13 64
Safety and Liveness • Lamport, "Proving the Correctness of Multiprocess Programs", 1977 • See also Alpern and Schneider, "Recognizing Safety and Liveness", 1987 and their prior related work • These properties help us reason about distributed systems designs, approaches, trade-offs Thursday, October 17, 13 65
Safety and Liveness • Safety: nothing bad happens • e.g. distributed transactions ensure consistency across a system • In consensus terms, only a single proposed value is chosen • Doing nothing is considered safe! Thursday, October 17, 13 66
Safety and Liveness • Liveness: something good eventually happens • e.g., a system eventually responds to every request • In consensus terms, a proposed value is eventually chosen • Ensures system progress Thursday, October 17, 13 67
See Also • Dwork, Lynch, Stockmeyer, "Consensus in the Presence of Partial Synchrony" 1988 • Oki's and Liskov's Viewstamped Replication work for high availability Thursday, October 17, 13 68
See Also • Ken Birman's work on reliable distributed computing (Isis, Horus) • Andrew Black's Eden project, a full distributed OO operating system, RPC-based • Andrew Herbert's "Advanced Network Systems Architecture" (ANSA), models and rules for distributed systems designs. Objects, transactions, interfaces. Influenced the Object Management Group (OMG) Thursday, October 17, 13 69
1990s Thursday, October 17, 13 70
Some 1990s Issues • Distributed objects • Practical consensus • The rise of the web Thursday, October 17, 13 71
Distributed Objects • OOP grew in popularity in the 70s and 80s • Many 80s distributed systems research systems were based on objects • But research systems were often full stacks, including OS, language, and compiler Thursday, October 17, 13 72
Distributed Objects • Computer vendors ultimately had little choice but to • incorporate distributed systems research into their own stacks • but make distributed programming features available for “normal” programming languages, without changing those languages • It all led to CORBA Thursday, October 17, 13 73
Common Object request Broker Architecture • First CORBA spec published 1991 • I co-authored this 1999 book • CORBA is still alive today in 2013, and this book still sells Thursday, October 17, 13 74
Ideal Distributed Objects Architecture Example: Object Management Architecture (OMA) from the Object Management Group (OMG) AI DI DI CF OS CF OS OS CORBA ORB CF OS OS AI = Application Interfaces DI = Domain Interfaces CF = Common Facilities OS = Object Services Thursday, October 17, 13 75
Enterprise Integration Reality Thursday, October 17, 13 76
Application Integration • A "common bus" was/is an interesting but impractical goal • Even today in 2013, application integration still involves numerous approaches Thursday, October 17, 13 77
Fallacies of Distributed Computing 1. The network is 5. Topology doesn't reliable. change. 2. Latency is zero. 6. There is one administrator. 3. Bandwidth is infinite. 7. Transport cost is zero. 4. The network is secure. 8. The network is homogeneous. Thursday, October 17, 13 78
Revised Fallacies of Distributed Computing 1. Partitions do not 6. There is one occur. administrator. 2. Latency is zero. 7. Transport cost is zero. 3. Bandwidth is infinite. 8. The network is homogeneous. 4. The network is secure. 9. Clocks are synchronized. 5. Topology doesn't change. 10. Concurrency can be ignored. Thursday, October 17, 13 79
Distributed Systems Concurrency • Dist Sys are inherently concurrent • Yet the distributed objects movement largely ignored concurrent object access • there was an OMG concurrency control service, but used only for the distributed transaction service • also ignored consensus, again except for the transaction service Thursday, October 17, 13 80
A Note on Distributed Computing • 1994 paper by Jim Waldo, Geoff Wyant, Ann Wollrath, Sam Kendall • Addresses a prevalent issue of the era: that local/remote transparency was a desirable goal • Also mentions the concurrency issue Thursday, October 17, 13 81
Paxos • Lamport's "The Part-Time Parliament" defines the Paxos algorithm, still widely used today • Paper originally submitted in 1990, panned by reviewers • But others recognized its significance, so Lamport finally resubmitted for publication in 1998 Thursday, October 17, 13 82
Implementing Consensus • Butler Lampson understood the importance of Paxos • Published "How to Build a Highly Available System Using Consensus" in 1996 • Practical advice on using Paxos in real systems Thursday, October 17, 13 83
See Also • F. Schneider's "Implementing Fault Tolerant Services Using the State Machine Approach: A Tutorial", 1990 • Karger et al. paper on Consistent Hashing, 1997 • A. Fox and E.A. Brewer, "Harvest, Yield, and Scalable Tolerant Systems", 1999 Thursday, October 17, 13 84
2000s Thursday, October 17, 13 85
Some 2000s Issues • REST • Paxos made simple • CAP • Dynamo Thursday, October 17, 13 86
REST • "Representational State Transfer", defined by Roy Fielding in his doctoral thesis, 2000 based on his work on the web and HTTP • An architectural style for networked applications • Has unfortunately now become an overused & misunderstood buzzword Thursday, October 17, 13 87
REST Constraints • Constraints and trade-offs are what help define an architecture • Client-server • Statelessness • Caching • Layered system • Uniform interface • Code on demand Thursday, October 17, 13 88
REST Properties • REST imposes these constraints to induce desired properties such as • performance, scalability, portability, simplicity • visibility (monitoring and mediation) • modifiability (evolution, extension, reuse) • reliability (handling failure, failover, load balancing, redundancy) Thursday, October 17, 13 89
REST • REST works well for problems that fit its constraints • But it's not for everything • A great general lesson from REST: 1. understand the properties your apps need 2. impose the appropriate constraints/ trade-offs to get them Thursday, October 17, 13 90
Paxos Made Simple • Lamport was tired of complaints of the complexity of Paxos, so he wrote this in 2001 • It's still complex Thursday, October 17, 13 91
CAP Theorem • Eric Brewer's Consistency, Availability, Partition Tolerance conjecture, 2000 • Formally proven in 2002 by Gilbert and Lynch • Common interpretation "pick two" isn't quite right Thursday, October 17, 13 92
CAP Theorem • Distributed systems fail • So, partition tolerance (P) isn't a choice • Under partition, does your system • try to be consistent (C) • try to be available (A) • it can't be both Thursday, October 17, 13 93
Amazon Dynamo • 2007 paper from Amazon describing a large-scale highly-available eventually consistent key-value datastore • Strong influence on Cassandra, Riak, and Voldemort databases • Riak Core is a framework for implementing Dynamo-like systems https://github.com/basho/riak_core Thursday, October 17, 13 94
See Also • The Google Chubby lock service • Joe Armstrong's 2003 PhD thesis "Making reliable distributed systems in the presence of software errors" • Pastry and Chord distributed hash tables, 2001 • Multicore CPU cache coherence protocols • Papers that have won the Dijkstra Prize Thursday, October 17, 13 95
2010s Thursday, October 17, 13 96
CRDTs Convergent Commutative Replicated Data Types Conflict-free • Replicated data types that handle updates correctly in eventually consistent, highly available systems • E.g., counters, sets, maps • Automatic handling updates that can occur concurrently or under partition Thursday, October 17, 13 97
Raft • D. Ongaro and J. Ousterhout, "In Search of an Understandable Consensus Algorithm", 2013 • See also Zookeeper Atomic Broadcast (ZAB) Thursday, October 17, 13 98
CALM and Bloom • CALM: Consistency as Logical Monotonicity • Bloom: distributed programming language that helps deal with distributed consistency Thursday, October 17, 13 99
• In general, pay attention to ALL distributed systems work coming from: • Joe Hellerstein, Neil Conway, Peter Alvaro, William Marczak from the UC Berkeley Database Group • Peter Bailis and Ali Ghodsi from AMPLab at UC Berkeley Thursday, October 17, 13 100
Recommend
More recommend
