astrolabe
play

Astrolabe: A Robust and Scalable Technology for Distributed System - PowerPoint PPT Presentation

May 13, 2023 •377 likes •861 views

Astrolabe: A Robust and Scalable Technology for Distributed System R. van Renesse, K.P. Birman, W. Vogels (Cornell University) Presentation by Konrad Iwanicki October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 1

  1. Astrolabe: A Robust and Scalable Technology for Distributed System R. van Renesse, K.P. Birman, W. Vogels (Cornell University) Presentation by Konrad Iwanicki October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 1

  2. Introduction ● Imagine an organization that needs to manage large collections of distributed resources, such as: ● personal workstations ● dedicated nodes in a web farm ● or objects stored and services run on these computers. ● Think really big: ● Amazon.com, ● Google.com, ● University of Warsaw. October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 2

  3. Sample Objects & Resources Name pc372 Browser IE Printer OKI 783 Disk_total 20GB Disk_used 5GB pc372.mimuw.edu.pl Name laptop065 Browser Firefox Printer - Disk_total 500GB Disk_used 413GB laptop065.mimuw.edu.pl October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 3

  4. Sample Management duch Name pc372 laptop065 Browser - IE Firefox Printer HP 3971 OKI 783 - Disk_total 2000GB 20GB 500GB Disk_used 587GB 5GB 413GB laptop065.mimuw.edu.pl October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 4

  5. Problems ● The computers hosting the resources may be: ● co-located in a room, ● spread across a building or a campus, ● scattered around the world. ● Configurations of such systems change rapidly: ● machine failures and connectivity changes are common, ● significant adaptation may be necessary to provide the desired level of service. ● How do you manage the resources in such systems? ● In particular, how do you retrieve information about the resources? October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 5

  6. Sample Management Queries SELECT SELECT COUNT(user_name) AS FIRST(3, host_name) AS firefox_user_count host FROM processes WHERE WHERE Disk_used/Disk_total > 80% ORDER BY name = 'firefox.exe' GROUP BY Disk_used/Disk_total DESC name October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 6

  7. Requirements ● Resource information aggregation: ● A convenient way of getting some summaries regarding the resources in the system. ● Resource location: ● A means for locating resources based on the summaries. ● Scalability (in terms of the number of machines). ● Robustness to changes in the network topology. ● Security (access control and integrity). October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 7

  8. A solution: Astrolabe ● Developed at Cornell University and a start-up company, RNS. ● Used by Amazon.com to manage its huge collections of machines and the services those machines run. ● Werner Vogels is now the CTO of Amazon.com (http://www.allthingsdistributed.com/) October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 8

  9. Zone Hierarchy duch laptop065 pc372 October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 9

  10. Zone Hierarchy mim duch laptop065 pc372 wz geol October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 10

  11. Zone Hierarchy uw mim duch laptop065 pc372 wz geol October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 11

  12. Attribute List uw mim duch laptop065 pc372 wz geol October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 12

  13. Attribute List uw mim duch laptop065 pc372 wz geol October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 13

  14. Attribute List uw mim duch laptop065 pc372 wz geol October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 14

  15. Attribute List uw mim duch laptop065 pc372 wz What are these attributes? geol October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 15

  16. Aggregation Functions uw mim duch laptop065 pc372 wz Aggregated Exported attributes. attributes. geol October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 16

  17. Aggregation Functions Aggregated attributes a continuously recomputed. Function Description MIN(attribute) Find the minimum attribute. MIN(attribute) Find the maximum attribute. SUM(attribute) Sum the attributes. COUNT(attribute) Compute the attributes. AVG(attribute [, weight]) Calculate a weighted average. OR(attribute) Bit-wise OR of a bitmap. AND(attribute) Bit-wise AND of a bitmap. FIRST(n, attribute) Return a set with the first n attributes. RANDOM(n, attribute [, weight]) Return a set with n randomly selected attributes. In general, aggregation functions should compact N values into a small output value. The size of the output value should be a small function of N: - preferably O(1), - alternatively O(logN). October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 17

  18. Attribute Maintenance uw mim duch laptop065 pc372 wz Who maintains attributes for a zone? geol October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 18

  19. Attribute Maintenance duch laptop065 pc372 The members The members of the zone. of the sibling zones. October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 19

  20. Attribute Maintenance The members of the zone. mim duch laptop065 pc372 wz The members of the sibling zones. geol October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 20

  21. Attribute Maintenance uw mim me duch laptop065 pc372 My zones; wz I compute the attributes for these zones myself. Sibling zones; I get the attributes for these zones from geol other machines. October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 21

  22. Attribute Maintenance me waw /pl kra All locally maintained attributes lub for zones: /pl/waw, /pl/kra, /pl/lub me uw /pl/waw pw pjwstk me mim /pl/waw/uw wz /pl/waw/uw/mim/laptop065 geol pc372 /pl/waw/uw/mim duch me laptop065 October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 22

  23. Attribute Maintenance me waw /pl kra All locally maintained attributes lub for zones: /pl/waw, /pl/kra, /pl/lub me uw /pl/waw pw How do I get the attributes pjwstk of the sibling zones? me mim /pl/waw/uw wz /pl/waw/uw/mim/laptop065 geol pc372 /pl/waw/uw/mim duch me laptop065 October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 23

  24. Hierarchical Gossiping One of the attributes maintained null waw Locally for a zone is a list of contacts for the zone . 147.13.132.137, ... kra 182..132.137, ... lub A list of contacts for zone uw null /pl/kra pw 195.28.56.201, ... pjwstk 162.32.90.45, ... A list of contacts for zone /pl/waw/uw/geol null mim 211.123.1.15, ... wz /pl/waw/uw/mim/laptop065 193.14.16.29, ... geol 193.0.96.127 pc372 193.0.96.2 duch 193.0.96.415 laptop065 October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 24

  25. Hierarchical Gossiping ● A computer in zone */Z/X computes the attributes for */Z/X locally, based on the child zone attributes. ● To obtain attributes for a sibling zone, */Z/Y, the computer: Selects a random contact IP with zone */Z/Y. ● Communicates with this IP to exchange: ● – The attributes of all child zones of the parent zone, */Z. – The attributes of all common higher level zones. Suppose that A local is the local (the computer's) value of attribute A, ● while A remote is the remote (the contact's) value of A, as obtained in the exchange. The computer chooses the fresher attribute: – If TimeComputed(A local ) < TimeComputed(A remote ): A local := A remote October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 25

  26. Hierarchical Gossiping null ● Select zone to gossip for. waw 147.13.132.137, ... kra 182..132.137, ... lub uw null pw 195.28.56.201, ... pjwstk 162.32.90.45, ... null mim 211.123.1.15, ... /pl/waw/uw/wz wz 193.14.16.29, ... geol 193.0.96.127 pc372 193.0.96.2 duch 193.0.96.415 laptop065 /pl/waw/uw/mim/laptop065 October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 26

  27. Hierarchical Gossiping null ● Select zone to gossip for. waw 147.13.132.137, ... kra ● Randomly select a 182..132.137, ... lub contact IP. uw null pw 195.28.56.201, ... pjwstk 162.32.90.45, ... null mim 211.123.1.15 , ... wz 193.14.16.29, ... geol 193.0.96.127 pc372 193.0.96.2 duch 193.0.96.415 laptop065 /pl/waw/uw/mim/laptop065 October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 27

  28. Hierarchical Gossiping null ● Select zone to gossip for. waw 147.13.132.137, ... kra ● Randomly select a 182..132.137, ... lub contact IP. uw null ● Exchange common zone pw 195.28.56.201, ... attributes. pjwstk 162.32.90.45, ... null mim 211.123.1.15, ... wz Exchange the common attributes 193.14.16.29, ... geol with 211.123.1.15 193.0.96.127 pc372 193.0.96.2 duch 193.0.96.415 laptop065 /pl/waw/uw/mim/laptop065 October 3rd and 8th, 2012 Distributed Systems Course, University of Warsaw 28

Recommend

Construction and accuracy of a 3D Astrolabe Science in 3D - Competition STEM4youth INTRODUCTION

Construction and accuracy of a 3D Astrolabe Science in 3D - Competition STEM4youth INTRODUCTION

Construction and accuracy of a 3D Astrolabe Science in 3D - Competition STEM4youth INTRODUCTION Universe: inspirational source W e see a universe m arvellously arranged and obeying certain law s, but only dim ly understand these law s. Our

715 views • 33 slides
CS5412: BIMODAL MULTICAST ASTROLABE Lecture XIX Ken Birman Leiden; Dec 06 Gossip 201 2

CS5412: BIMODAL MULTICAST ASTROLABE Lecture XIX Ken Birman Leiden; Dec 06 Gossip 201 2

Gossip-Based Networking Workshop 1 CS5412: BIMODAL MULTICAST ASTROLABE Lecture XIX Ken Birman Leiden; Dec 06 Gossip 201 2 Recall from early in the semester that gossip spreads in log(system size) time But is this actually

607 views • 40 slides
Tools for Scalable Data Mining XANDA SCHOFIELD CS 6410 11/13/2014 1. Astrolabe Large,

Tools for Scalable Data Mining XANDA SCHOFIELD CS 6410 11/13/2014 1. Astrolabe Large,

Tools for Scalable Data Mining XANDA SCHOFIELD CS 6410 11/13/2014 1. Astrolabe Large, eventually- consistent distributed system [Source: Wikipedia] ROBERT VAN RENESSE, KEN BIRMAN, WERNER VOGELS The Problem How do we quickly find out

823 views • 35 slides
CS5412: BIMODAL MULTICAST ASTROLABE Lecture XIX Ken Birman Gossip 201 2 Recall from early

CS5412: BIMODAL MULTICAST ASTROLABE Lecture XIX Ken Birman Gossip 201 2 Recall from early

CS5412 Spring 2016 1 CS5412: BIMODAL MULTICAST ASTROLABE Lecture XIX Ken Birman Gossip 201 2 Recall from early in the semester that gossip spreads in log(system size) time But is this actually fast? 1.0 % infected 0.0 Time

643 views • 49 slides
Ken Birman i Cornell University. CS5410 Fall 2008. Background for today Consider a system like

Ken Birman i Cornell University. CS5410 Fall 2008. Background for today Consider a system like

Ken Birman i Cornell University. CS5410 Fall 2008. Background for today Consider a system like Astrolabe. Node p announces: Ive computed the aggregates for the set of leaf nodes to which I belong hi h I b l It turns out that under

677 views • 42 slides

More recommend