Scalable DDS Discovery Protocols Based on Bloom Filters Javier - PowerPoint PPT Presentation

OMG Real-time & Embedded Systems Workshop 11th July, 2007 - Arlington, VA USA Scalable DDS Discovery Protocols Based on Bloom Filters Javier Sanchez-Monedero, Javier Povedano-Molina & Juan M. Lopez-Soler University of Granada

1.Introduction  DDS needs discovery protocols: procedure to put in contact publishers and subscribers  Discovery is a common problem in distributed networking  Discovery can compromise the scalability of DDS  The state of the art in discovery is important to review New functional relationships and network topologies to consider: centralized, pure and hierarchical peer-to- peer Optimization techniques: locality, cache and Bloom filters  Main goal : to improve the scalability of DDS discovery protocols by using bloom filter technology and researches in peer-to-peer (P2P) Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 2

Contents 1. Introduction 2. The State of the Art in Discovery 3. Bloom filters 4. SDP-Bloom 5. Hierarchical P2P in DDS Discovery 6. Conclusions Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 3

2.The State of the Art in Discovery  Reference Discovery procedures − Chord: a scalable peer-to-peer lookup protocol for internet applications . − Pastry: Scalable, Decentralized Object Location, and Routing for Large- Scale Peer-to-Peer Systems − OSDA: Open Service Discovery Architecture and Distributed Search in Semantic Web Service Discovery thesis. − Directory Facilitator and Service Discovery Agent (DFSDA) . − Kademlia: A Peer-to-Peer Information System Based on the XOR Metric . − LDAP directories. − Peer-to-Peer networks like ed2k or Gnutella  Adopted terminology − Client ~ Node ~ Agent − Server ~ SuperNode ~ DirectoryFacilitator Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 4

2.The State of the Art in Discovery  Discovery procedure issues − Functional relationship between the entities of the network  Client-Server  Centralized peer-to-peer (P2P)  Pure P2P  Hierarchical P2P − Network entities distribution  Unstructured topology (ed2k)  Ring in Distributed Hash Table (DHT) implementations (Chord, Pastry...)  Trees (LDAP Directories)  Hierarchical combination (OSDA, DFSDA) − Database and information representation and storing  DHT  Bloom filters − Learning, cache and local versus global discovery Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 5

2.The State of the Art in Discovery  Pure P2P are theoretically scalable, but not in practice. For instances − Chord does not scale in scenarios with 2000 nodes − Pastry can turn out to be non-functional given the incurred management traffic overhead. Bjurefors, F. Larzon, L.A. Gold, R.: “ Performance of Pastry in a Heterogeneous System ”. Proceedings of the Fourth International Conference on Peer-to-Peer Computing, 2004.  Some hierarchical combination uses DHTs to maintain a structure for global discovery Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 6

2.The State of the Art in Discovery  State of the art main conclusions: Pure P2P does not scale well in Hierarchical P2P practice Hierarchical combination Network entities distribution (OSDA) To reduce network traffic and Bloom filters storage in Database and information representation Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 7

Contents 1. Introduction 2. The State of the Art in Discovery 3. Bloom filters 3.1. Introduction to Bloom filters 3.2. BF practical uses 4. SDP-Bloom 5. Hierarchical P2P in DDS Discovery 6. Conclusions Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 8

3.1. Introduction to Bloom filters  High Level Ideas Because DDS entities lists can be long and unwieldy to manage − We move from: “ Give me the list of what you have ” − to the paradigm of: “ Give me information so I can figure out what you have ” − Bloom filters allow to achieve the new paradigm −  Given a set S = {x1,x2,x3,…xn} the problem is to answer queries of the form: Is element y in the set S?  Bloom filter (BF) technology provides answers in: “Constant” time (time to hash). − Small amount of space. − But with some probability of being wrong (false positives). − Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 9

3.1. Introduction to Bloom filters Start with an m bit array, filled with 0s. v 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Hash each item x j in S k times. If H i ( x j ) = p , set V [ p ] = 1. v 0 1 0 0 1 0 1 0 0 1 1 1 0 1 1 0 To check if y is in S , check V at H i ( y ). All k values must be 1. v 0 1 0 0 1 0 1 0 0 1 1 1 0 1 1 0 Possible to have a false positive; all k values are 1, but y is not in S . v 0 1 0 0 1 0 1 0 0 1 1 1 0 1 1 0 Source: Michael Mitzenmacher, “ Codes, Bloom Filters, and Overlay Networks ” Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 10

3.1. Introduction to Bloom filters Bit Vector v Element a H 1 (a)=P 1 1  Given a H 2 (a)=P 2 − Vector v of size m 1 m bits − k hash functions, H i (x) H 3 (a)=P 3 1 − A set S with n elements H 4 (a)=P 4 1 The probability of a false positive can be expressed as kn k f = 1 − 1 − 1 − kn / m  k m   ≈ 1 − e Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 11

3.1. Introduction to Bloom filters Some Bloom filters additional issues  Bloom filters work better when they are not full  Two design parameters must be tuned − m the size of the filter − k the number of hash functions  It is needed to deal with false positives  Note that to delete an item implies re-building the entire filter  Depending on m and the size of a key, the updates should be done in different ways: − To send the entire filter − To send just the updated information change Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 12

Contents 1. Introduction 2. The State of the Art in Discovery 3. Bloom filters 3.1. Introduction to Bloom filters 3.2. BF practical uses 4. SDP-Bloom 5. Hierarchical P2P in DDS Discovery 6. Conclusions Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 13

3.2. BF practical uses  The first BF use was in Spell Check (~70s): − The filter represents a list of valid words  Summary Cache (Cache Digest in Squid) − Squid is a high-performance proxy caching server for web clients, supporting FTP, gopher, and HTTP data objects. − A Cache Digest is a summary of the contents of an Internet Object Caching Server. It contains, in a compact (i.e. compressed) format, an indication of whether or not particular URLs are in the cache.  OSDA ( Open Service Discovery Architecture ) − Similar use to Cache Digest − Used to reduce network traffic in local and global discovery Noura Limam, Joanna Ziembicki, Reaz Ahmed, Youssef Iraqi, Dennis Tianshu Li, Raouf Boutaba, and Fernando Cuervo. Osda: Open service discovery architecture forefficient cross-domain service provisioning . Comput. Commun., 30(3):546–563, 2007. Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 14

Contents 1. Introduction 2. The State of the Art in Discovery 3. Bloom filters 4. SDP-Bloom 4.1 Discovery Process in DDS 4.2 SDP-Bloom Overview 4.3 Algorithm Description 4.4 Design decisions 4.5 Nodes dialog 4.6 SDP-Bloom Scalability Analysis 5. Hierarchical P2P in DDS Discovery 6. Conclusions Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 15

4.1.Discovery Process in DDS  DDS SDP ( Simple Discovery Protocol )  DDS uses DDS publication for discovery purposes (different ports are used)  Two consecutive process: − First: Participant discovery protocol − Secondly: Endpoint discovery protocol  Use of special (Built-in) Topics and DataReader/DataWriter to advertise participants/publications  The Discovery process can be tuned with specific QoS policies  Discovered participants/publications are stored in a local database  The discovery process is started from a list of known host Javier Sánchez-Monedero, Javier Povedano, Juan M. Lopez-Soler: "Scalable DDS Discovery Protocols Based on Bloom Filters". University of Granada. 16