Distributed Systems What can we do now that we could not do before? - PDF document

Distributed Systems What can we do now that we could not do before? Introduction Paul Krzyzanowski pxk@cs.rutgers.edu Except as otherwise noted, the content of this presentation is licensed under the Creative Commons Attribution 2.5 License. Page 1 Page 1 Page 2 Page 2 Technology advances Networking: Ethernet - 1973, 1976 Networking Processors Memory Storage Protocols June 1976 : Robert Metcalfe presents the concept of Ethernet at the National Computer Conference 1980 : Ethernet introduced as de facto standard (DEC, Intel, Xerox) Page 3 Page 4 Network architecture Network Connectivity 570million 348 – >35,000x faster more hosts LAN speeds Then : – Original Ethernet: 2.94 Mbps – Large companies and universities on Internet – Gateways between other networks – 1985 : thick Ethernet: 10 Mbps 1 Mbps with twisted pair networking – Dial-up bulletin boards – 1991 : 10BaseT - twisted pair: 10 Mbps – 1985: 1,961 hosts on the Internet Switched networking: scalable bandwidth Now : – 1995 : 100 Mbps Ethernet – One Internet (mostly) – 1998 : 1 Gbps (Gigabit) Ethernet – 2008: 570,937,778 hosts on the Internet – 1999 : 802.11b (wireless Ethernet) standardized – Widespread connectivity – 2001 : 10 Gbps introduced High-speed WAN connectivity: 1 – >50 Mbps – Switched LANs – 2005 : 100 Gbps (over optical link) – Wireless networking Page 5 Page 6 1

Network Connectivity Computing power Computers got… – Smaller – Cheaper – Power efficient – Faster Microprocessors became technology leaders Page 7 Page 8 Computing Power Storage: RAM 9,000x cheaper 4,000x more capacity 1974: Intel 8080 2 MHz, 6K transistors year $/MB typical 2004: Intel P4 Prescott 1977 $32,000 16K 3.6 GHz, 125 million transistors 1987 $250 640K-2MB 2006: Intel Core 2 Duo 1997 $2 64MB-256MB 2.93 GHz, 291 million transistors 2007 $0.06 512MB-2GB+ Page 9 Page 10 Storage: disk Music Collection 129,000x cheaper in 20 years 18,750x more capacity 4,207 Billboard hits – 18 GB Recording density increased over – Average song size: 4.4 MB 60,000,000 times over 50 years Today 1977: 360KB floppy drive – $1480 – Download time per song @12.9 Mbps: 3.5 sec – Storage cost: $2.38 $11,529 / GB (but 2,713 5½″ disks!) 1987: 40 MB drive for – $679 Approx 20 years ago (1987) $16,975K / GB – Download time per song, V90 modem @44 Kbps: 2008: 750 GB drive for – $99 15 minutes – Storage cost: $305,55 $0.13 / GB Page 11 Page 12 2

Protocols Why do we want to network? Faster CPU • Performance ratio – Scaling multiprocessors may not be possible or cost effective more time for protocol processing • Distributing applications may make sense – ECC, checksums, parsing (e.g. XML) – ATMs, graphics, remote monitoring – Image, audio compression feasible • Interactive communication & entertainment – work and play together: email, gaming, telephony, instant messaging Faster network • Remote content bigger (and bloated) protocols – web browsing, music & video downloads, IPTV, file servers – e.g., SOAP/XML, H.323 • Mobility • Increased reliability • Incremental growth Page 13 Page 14 Problems Designing distributed software can be difficult – Operating systems handling distribution – Programming languages? – Efficiency? Building and classifying – Reliability? – Administration? distributed systems Network – disconnect, loss of data, latency Security – want easy and convenient access Page 15 Page 16 Page 16 Flynn’s Taxonomy (1972) Subclassifying MIMD number of instruction streams memory and number of data streams – shared memory systems: multiprocessors SISD – no shared memory: networks of computers, – traditional uniprocessor system multicomputers SIMD – array (vector) processor interconnect – Examples: • GPUs – Graphical Processing Units for video – bus • APU (attached processor unit in Cell processor) • SSE3: Intel’s Streaming SIMD Extensions – switch • PowerPC AltiVec (Velocity Engine) • GPGPU (General Purpose GPU): AMD/ATI, Nvidia delay/bandwidth • Intel Larrabee (late 2008?) MISD – tightly coupled systems – Generally not used and doesn’t make sense – loosely coupled systems – Sometimes (rarely!) applied to classifying redundant systems MIMD – multiple computers, each with: • program counter, program (instructions), data – parallel and distributed systems Page 17 Page 18 3

Bus-based multiprocessors Bus-based multiprocessors SMP : Symmetric Multi-Processing Dealing with bus overload All CPUs connected to one bus (backplane) - add local memory Memory and peripherals are accessed via shared bus. CPU does I/O to cache memory System looks the same from any processor. - access main memory on cache miss CPU A CPU B CPU A CPU B Device Device memory memory I/O I/O cache cache Bus Bus Page 19 Page 20 Working with a cache Working with a cache CPU A reads location 12345 from memory CPU A modifies location 12345 CPU A CPU B CPU A CPU B Device Device 12345:7 12345:7 12345: 7 I/O 12345: 3 12345: 7 I/O Bus Bus Page 21 Page 22 Working with a cache Write-through cache CPU B reads location 12345 from memory Fix coherency problem by writing all values through bus to main memory Gets old value CPU A modifies location 12345 – write-through Memory not coherent! main memory is now coherent CPU A CPU B CPU A CPU B Device Device 12345:7 12345:3 12345:7 I/O I/O 12345: 3 12345: 7 12345: 3 12345: 7 Bus Bus Page 23 Page 24 4

Write- through cache … continued Write-through cache CPU B reads location 12345 from memory CPU A modifies location 12345 - loads into cache - write-through Cache on CPU B not updated Memory not coherent! CPU A CPU B CPU A CPU B Device Device 12345:3 12345:3 12345:0 I/O I/O 12345: 3 12345: 3 12345: 0 12345: 3 12345: 3 Bus Bus Page 25 Page 26 Snoopy cache Switched multiprocessors Add logic to each cache controller: • Bus-based architecture does not scale monitor the bus to a large number of CPUs (8+) Virtually all bus-based architectures use a snoopy cache CPU A CPU B Device 12345: 3 12345: 0 12345: 0 12345: 3 12345: 0 12345: 3 12345: 3 I/O Bus write [12345] 0 Page 27 Page 28 Switched multiprocessors Crossbar alternative: omega network Divide memory into groups and connect chunks Reduce crosspoint switches by adding of memory to the processors with a crossbar more switching stages switch mem mem mem mem CPU mem CPU CPU mem CPU CPU CPU mem CPU CPU mem n 2 crosspoint switches – expensive switching fabric Page 29 Page 30 5

Crossbar alternative: omega network NUMA • Hierarchical Memory System CPU mem • Each CPU has local memory CPU mem • Other CPU’s memory is in its own address CPU mem space CPU mem – slower access with n CPUs and n memory modules: • Better average access time than omega need log 2 n switching stages, network if most accesses are local each with n/2 switches • Placement of code and data becomes difficult Total: ( n log 2 n )/2 switches . • Better than n 2 but still a quite expensive • delay increases: 1024 CPU and memory chunks overhead of 10 switching stages to memory and 10 back. Page 31 Page 32 NUMA Bus-based multicomputers • SGI Origin’s ccNUMA • No shared memory • AMD64 Opteron • Communication mechanism needed on bus – Each CPU gets a bank of DDR memory – Traffic much lower than memory access – Inter-processor communications are sent over a – Need not use physical system bus HyperTransport link • Can use LAN (local area network) instead • Linux 2.5 kernel – Multiple run queues – Structures for determining layout of memory and processors Page 33 Page 34 Bus-based multicomputers Switched multicomputers Collection of workstations on a LAN Collection of workstations on a LAN CPU CPU CPU CPU CPU CPU CPU CPU memory memory memory memory memory memory memory memory LAN LAN LAN LAN LAN LAN LAN LAN connector connector connector connector connector connector connector connector Interconnect n-port switch Page 35 Page 36 6

Software Single System Image Collection of independent computers that appears You know you have a distributed as a single system to the user(s) system when the crash of a • Independent : autonomous computer you’ve never heard of • Single system : user not aware of distribution stops you from getting any work done. Distributed systems software – Leslie Lamport Responsible for maintaining single system image Page 37 Page 38 Coupling Design issues: Transparency High level: hide distribution from users Low level: hide distribution from software – Location transparency : Tightly versus loosely coupled software users don’t care where resources are – Migration transparency : resources move at will Tightly versus loosely coupled hardware – Replication transparency : users cannot tell whether there are copies of resources – Concurrency transparency : users share resources transparently – Parallelism transparency : operations take place in parallel without user’s knowledge Page 39 Page 40 Design issues Reliability – Availability : fraction of time system is usable • Achieve with redundancy – Reliability : data must not get lost • Includes security Performance Service Models – Communication network may be slow and/or unreliable Scalability – Distributable vs. centralized algorithms – Can we take advantage of having lots of computers? Page 41 Page 42 Page 42 7