data structures from the future bloom filters distributed
play

Data Structures from the Future: Bloom Filters, Distributed - PowerPoint PPT Presentation

Apr 10, 2023 •365 likes •1.49k views

Data Structures from the Future: Bloom Filters, Distributed Hash Tables, and More! Tom Limoncelli, Google NYC tlim@google.com Thursday, November 11, 2010 1 Why am I here? I have no idea. Thursday, November 11, 2010 2 Why are you

  1. 0100100111011001 Load it up! 0001000101100011 1001110100110111 1110001010010110 Root 0011000000000100 Host 0 0 1 4 Thursday, November 11, 2010 65

  2. 0100100111011001 Split 0001000101100011 1001110100110111 1110001010010110 Root 0011000000000100 Host 0110000111101100 0100000001101011 0010111000000001 0 0011000101111000 3 4 7 1 1 2 3 Thursday, November 11, 2010 66

  3. 0100100111011001 ’01...’ 0001000101100011 1001110100110111 1110001010010110 Root 0011000000000100 Host 0110000111101100 0100000001101011 0010111000000001 0 0011000101111000 3 4 7 1 1 2 3 Thursday, November 11, 2010 67

  4. 0100100111011001 ‘0...’ 0001000101100011 1001110100110111 1110001010010110 Root 0011000000000100 Host 0110000111101100 0100000001101011 0010111000000001 0 0011000101111000 3 4 7 1 1 2 3 Thursday, November 11, 2010 68

  5. 0100100111011001 ‘1...’ 0001000101100011 1001110100110111 1110001010010110 Root 0011000000000100 Host 0110000111101100 0100000001101011 0010111000000001 0 0011000101111000 3 4 7 1 1 2 3 Thursday, November 11, 2010 69

  6. Split Root Host 0001000101100011 0011000000000100 0010111000000001 0 3 4 7 0011000101111000 1 1 0100100111011001 2 3 0110000111101100 1001110100110111 0100000001101011 1110001010010110 Thursday, November 11, 2010 70

  7. Find: 0100100111011001... Find: 0100100111011001... Find: 0100100111011001... 011 Root Root 010 010 Host Host 000 001 010 011 001 000 001 010 010 011 000 000 001 010 011 000 001 010 011 Thursday, November 11, 2010 71

  8. Find: 0100110111011... Thursday, November 11, 2010 72

  9. 011 Find: 0100110111011... Find: 0100110111011... Find: 0100110111011... Root Root 010 010 Host Host 000 001 010 011 001 000 001 010 011 011 000 000 001 010 011 000 001 010 011 Thursday, November 11, 2010 73

  10. Each host stores: All the data that “leaf” there. The list of parent nodes talking to it. The list of children it knows about. Thursday, November 11, 2010 74

  11. Dynamically Adjusting: Data hashes in “clumps” making some hosts under-full and some hosts over-full. Host running out of storage? Split in two. Give half the data to another node. Host running out of bandwidth? Clone data and load-balance. Thursday, November 11, 2010 75

  12. 011 Root Root Root 010 Root Host Host Host Host 000 001 010 011 001 001 001 001 000 001 001 001 010 011 001 000 000 000 000 000 000 000 001 010 011 000 001 010 011 010 010 Thursday, November 11, 2010 76

  13. Real DHTs in action Peer 2 Peer file-sharing networks. Content Delivery Networks (CDNs like Akamai) Cooperative Caches Thursday, November 11, 2010 77

  14. Distributed Hash Tables (DHTs) Thursday, November 11, 2010 78

  15. Key/Value Stores Thursday, November 11, 2010 79

  16. Some common Key/Value Stores “NoSQL” CouchDB MongoDB Apache Cassandra Terrastore Google Bigtable Thursday, November 11, 2010 80

  17. Name Email Address 1515 Main Tom Limoncelli tlim@google.com Street Mary Smith mary@example.com 111 One Street Joe Bond joe@007.com 7 Seventh St Thursday, November 11, 2010 81

  18. Name Email Address Tom Limoncelli tlim@google.com 1515 Main Street Mary Smith mary@example.com 111 One Street User Transaction Amount Joe Bond joe@007.com 7 Seventh St Tom Limoncelli Deposit 100 Mary Smith Deposit 200 Tom Limoncelli Withdraw 50 Thursday, November 11, 2010 82

  19. Id Name Email Address Tom 1515 Main 1 tlim@google.com Limoncelli Street mary@example.c 111 One 2 Mary Smith om Street User Id Transaction Amount 3 Joe Bond joe@007.com 7 Seventh St 1 Deposit 100 2 Deposit 200 1 Withdraw 50 Thursday, November 11, 2010 83

  20. Id Name Email Address Tom 1515 Main 1 tlim@google.com Limoncelli Street mary@example.c 111 One 2 Mary Bond om Street User Id Transaction Amount 3 Joe Bond joe@007.com 7 Seventh St 1 Deposit 100 2 Deposit 200 3 Withdraw 50 Thursday, November 11, 2010 84

  21. Relational Databases 1st Normal Form 2nd Normal Form 3rd Normal Form ACID: Atomicity, Consistency, Isolation, Durability Thursday, November 11, 2010 85

  22. Key/Value Stores Keys Values BASE: Basically Available, Soft-state, Eventually consistent Thursday, November 11, 2010 86

  23. Eventually? Who cares! This is the web, not payroll! Change the address listed in your profile. Might not propagate to Europe for 15 minutes. Can you fly to Europe in less than 15 minutes? And if you could, would you care? Thursday, November 11, 2010 87

  24. Key/Value example: Key Value tlim@google.com BLOB OF DATA mary@example.com BLOB OF DATA joe@007.com BLOB OF DATA Thursday, November 11, 2010 88

  25. Key/Value example: Key Value { ‘name’: ‘Tom Limoncelli’, tlim@google.com ‘address’: ‘1515 Main Street’ } { ‘name’: ‘Mary Smith’, mary@example.com ‘address’: ‘111 One Street’ } { ‘name’: ‘Joe Bond’, joe@007.com ‘address’: ‘7 Seventh St’ } Thursday, November 11, 2010 89

  26. Google Protobuf: http://code.google.com/p/protobuf/ Key Value message Person { " required string name = 1; tlim@google.com " optional string address = 2; repeated string phone = 3; } { ‘name’: ‘Mary Smith’, mary@example.com ‘address’: ‘111 One Street’, ‘phone’: [‘201-555-3456’, ‘908-444-1111’] } { ‘name’: ‘Joe Bond’, joe@007.com ‘phone’: [‘862-555-9876’] } Thursday, November 11, 2010 90

  27. Key/Value Stores Thursday, November 11, 2010 91

  28. Bigtable Thursday, November 11, 2010 92

  29. Bigtable Google’s very very large database. OSDI'06 http://labs.google.com/papers/bigtable.html Petabytes of data across thousands of commodity servers. Web indexing, Google Earth, and Google Finance Thursday, November 11, 2010 93

  30. Bigtable Keys Can be very huge. Don’t have to have a value! (i.e the value is “null”) Query by Key Key start/stop range (lexigraphical order) Thursday, November 11, 2010 94

  31. Long keys are cool. Key Value Query range: Main St/123/Apt1 Jones Start: “Main St/123” End: infinity Main St/123/Apt2 Smith Main St/200 Olson Thursday, November 11, 2010 95

  32. Bigtable Values Values can be huge. Gigabytes. Multiple values per key, grouped in “families”: “key:family:family:family:...” Thursday, November 11, 2010 96

  33. Families Within a family: Sub-keys that link to data. Sub-keys are dynamic: no need to pre-define. Sub-keys can be repeated. Thursday, November 11, 2010 97

  34. Example: Crawl the web For every URL: Store the HTML at that location. Store a list of which URLs link to that URL. Store the “anchor text” those sites used. <a href=”URL”>ANCHOR TEXT</a> Thursday, November 11, 2010 98

  35. http://www.cnn.com <html>.........</html> http://tomontime.com <html> <p>As you may have read on <a href=”http:// www.cnn.com”>my favorite news site</a> there is... Thursday, November 11, 2010 99

  36. Family Another family Key contents: anchor:tomontime.com anchor:cnnsi.com com.cnn.www <html>... my favorite news site CNN Key contents: anchor:everythingsysadmin.com com.tomontime <html>... videos Thursday, November 11, 2010 100

Recommend

Outline Bloom filters Applications of Bloom filters Our replacement for Bloom filters

Outline Bloom filters Applications of Bloom filters Our replacement for Bloom filters

An Optimal Bloom Filter Replacement a Rasmus Pagh, IT University of Copenhagen Joint work with Anna Pagh, IT University of Copenhagen S. Srinivasa Rao, University of Waterloo a To appear in SODA 2005 1 Outline Bloom filters

405 views • 18 slides
Bloom Filters Rapha el Clifford (Slides by Benjamin Sach and Ashley Montanaro) Introduction

Bloom Filters Rapha el Clifford (Slides by Benjamin Sach and Ashley Montanaro) Introduction

Data Structures and Algorithms COMS21103 Bloom Filters Rapha el Clifford (Slides by Benjamin Sach and Ashley Montanaro) Introduction In this lecture we are interested in space efficient data structures for storing a set S which support

1.51k views • 108 slides
Revisiting Bloom Filters Payload attribution via Hierarchiecal Bloom Filters Kulesh

Revisiting Bloom Filters Payload attribution via Hierarchiecal Bloom Filters Kulesh

Revisiting Bloom Filters Payload attribution via Hierarchiecal Bloom Filters Kulesh Shanmugasundaram, Herve Bronnimann, Nasir Memon 600.624 - Advanced Network Security version 3 Overview Questions Collaborative Intrusion Detection

1.29k views • 58 slides
Meta-Learning Neural Bloom Filters Jack Rae Sergey Bartunov Tim Lillicrap Architecture

Meta-Learning Neural Bloom Filters Jack Rae Sergey Bartunov Tim Lillicrap Architecture

Meta-Learning Neural Bloom Filters Jack Rae Sergey Bartunov Tim Lillicrap Architecture Interested in neural networks with compressive, distributed memories. Problem Trend in the use of neural networks to replace classical data-structures.

784 views • 9 slides
Bloom Filters Anna Karlin Most slides by Shreya Jayaraman, Luxi Wang, Alex Tsun Bloom Filters:

Bloom Filters Anna Karlin Most slides by Shreya Jayaraman, Luxi Wang, Alex Tsun Bloom Filters:

Bloom Filters Anna Karlin Most slides by Shreya Jayaraman, Luxi Wang, Alex Tsun Bloom Filters: Motivation Large universe of possible data items. Hash table is stored on disk or in network, so any lookup is expensive. Many (if

1.29k views • 43 slides
Bloom Filters Queries False-Positives Analysis Summary Anil Maheshwari anil@scs.carleton.ca

Bloom Filters Queries False-Positives Analysis Summary Anil Maheshwari anil@scs.carleton.ca

Bloom Filters Anil Maheshwari Bloom Filter Data Structure Bloom Filters Queries False-Positives Analysis Summary Anil Maheshwari anil@scs.carleton.ca School of Computer Science Carleton University Canada Outline Bloom Filters Anil

556 views • 13 slides
Lecture #2: Advanced hashing and concentration bounds o Bloom filters o Cuckoo hashing o Load

Lecture #2: Advanced hashing and concentration bounds o Bloom filters o Cuckoo hashing o Load

outline Lecture #2: Advanced hashing and concentration bounds o Bloom filters o Cuckoo hashing o Load balancing o Tail bounds Bloom filters Idea: For the sake of efficiency, sometime we allow our data structure to make mistakes Bloom filter:

712 views • 68 slides
Vectorized Bloom Filters for Advanced SIMD Processors Orestis Polychroniou Kenneth A. Ross

Vectorized Bloom Filters for Advanced SIMD Processors Orestis Polychroniou Kenneth A. Ross

Vectorized Bloom Filters for Advanced SIMD Processors Orestis Polychroniou Kenneth A. Ross Bloom filters Introduction Original version [Bloom 1970] Represents a set of items Answers: Does item X belong to

446 views • 29 slides
Sampling and Reconstruction Using Bloom Filters Neha Sengupta 1 , Amitabha Bagchi 1 , Srikanta

Sampling and Reconstruction Using Bloom Filters Neha Sengupta 1 , Amitabha Bagchi 1 , Srikanta

Sampling and Reconstruction Using Bloom Filters Neha Sengupta 1 , Amitabha Bagchi 1 , Srikanta Bedathur 2 , Maya Ramanath 1 1 IIT Delhi 2 IBM India Research Lab Bloom Filters Compact storage m bits, k hash functions Set Membership

348 views • 19 slides
Bloom Filters References A. Broder and M. Mitzenmacher, Network applications of Bloom A.

Bloom Filters References A. Broder and M. Mitzenmacher, Network applications of Bloom A.

2/16/2017 Bloom Filters References A. Broder and M. Mitzenmacher, Network applications of Bloom A. Broder and M. Mitzenmacher, Network applications of Bloom filters: A survey, Internet Mathematics , vol. 1 no. 4, pp. 485-509, 2004. Li

532 views • 22 slides
Building Data applications with Go from Bloom filters to Data pipelines Sergii Khomenko, Data

Building Data applications with Go from Bloom filters to Data pipelines Sergii Khomenko, Data

Building Data applications with Go from Bloom filters to Data pipelines Sergii Khomenko, Data Scientist sergii.khomenko@stylight.com, @lc0d3r FOSDEM - January 31, 2016 01 Sergii Khomenko Data scientist at one of the biggest fashion

1.22k views • 84 slides
An Examination of Bloom Filters and their Applications Jacob Honoroff March 16, 2006 Outline

An Examination of Bloom Filters and their Applications Jacob Honoroff March 16, 2006 Outline

An Examination of Bloom Filters and their Applications Jacob Honoroff March 16, 2006 Outline Bloom Filter Overview Traditional Applications Hierarchical Bloom Filters Paper Less Traditional Applications &amp; Extensions 1

1.45k views • 113 slides
Filters (Bloom &amp; Quotient) CSCI 333 Operations Filters approximately represent sets.

Filters (Bloom &amp; Quotient) CSCI 333 Operations Filters approximately represent sets.

Filters (Bloom &amp; Quotient) CSCI 333 Operations Filters approximately represent sets. Therefore, a filter must support: Insertions: insert(key) Queries: lookup(key) Filters may also support other operations: Deletion:

510 views • 24 slides
PEER-TO-PEER NUMERIC COMPUTING WITH JAVASCRIPT Athan Reines @kgryte / BLOOM FILTERS

PEER-TO-PEER NUMERIC COMPUTING WITH JAVASCRIPT Athan Reines @kgryte / BLOOM FILTERS

PEER-TO-PEER NUMERIC COMPUTING WITH JAVASCRIPT Athan Reines @kgryte / BLOOM FILTERS DBSCAN 2D CLASSIFICATION FOURIER SERIES NEURAL NETWORKS WHY JAVASCRIPT? UBIQUITY PERFORMANCE IN-BROWSER ANALYSIS DATA PIPELINES STREAMS

1.03k views • 73 slides
I great potential for representing a set in main memory [13] in NFORMATION representation and

I great potential for representing a set in main memory [13] in NFORMATION representation and

120 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 22, NO. 1, JANUARY 2010 The Dynamic Bloom Filters Deke Guo, Member , IEEE , Jie Wu, Fellow , IEEE , Honghui Chen, Ye Yuan, and Xueshan Luo Abstract A Bloom filter is an effective,

534 views • 14 slides
Indexing Encrypted Data Using Bloom Filters Claude N. Warren, Jr January 11, 2020 Email:

Indexing Encrypted Data Using Bloom Filters Claude N. Warren, Jr January 11, 2020 Email:

Indexing Encrypted Data Using Bloom Filters Claude N. Warren, Jr January 11, 2020 Email: claude@xenei.com Github: https://github.com/Claudenw LinkedIn: https://www.linkedin.com/in/claudewarren Research Gate:

457 views • 18 slides
Bloom Filters and their Applications These slides were developed by -- and used with permission

Bloom Filters and their Applications These slides were developed by -- and used with permission

Bloom Filters and their Applications These slides were developed by -- and used with permission from -- Shengquan Wang. CPSC 662 Introduction Membership Query Given a set S={x 1 , x 2 , , x n } on a universe U , want to answer the query

304 views • 12 slides
Scalable DDS Discovery Protocols Based on Bloom Filters Javier Sanchez-Monedero, Javier

Scalable DDS Discovery Protocols Based on Bloom Filters Javier Sanchez-Monedero, Javier

OMG Real-time &amp; Embedded Systems Workshop 11th July, 2007 - Arlington, VA USA Scalable DDS Discovery Protocols Based on Bloom Filters Javier Sanchez-Monedero, Javier Povedano-Molina &amp; Juan M. Lopez-Soler University of Granada

552 views • 43 slides
http://cs246.stanford.edu More algorithms for streams: (1) Filtering a data stream: Bloom

http://cs246.stanford.edu More algorithms for streams: (1) Filtering a data stream: Bloom

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu More algorithms for streams: (1) Filtering a data stream: Bloom filters Select elements with property x from stream (2) Counting distinct

842 views • 40 slides
Exercise Sheet 1: Hashing and Bloom filters COMS31900 Advanced Algorithms 2019/2020 Please feel

Exercise Sheet 1: Hashing and Bloom filters COMS31900 Advanced Algorithms 2019/2020 Please feel

Exercise Sheet 1: Hashing and Bloom filters COMS31900 Advanced Algorithms 2019/2020 Please feel free to discuss these problems on the unit discussion board. If you would like to have your answers marked, please either hand them in in person at

361 views • 5 slides
On the Privacy Provisions of Bloom Filters in Lightweight Bitcoin Clients Arthur Gervais, Ghassan

On the Privacy Provisions of Bloom Filters in Lightweight Bitcoin Clients Arthur Gervais, Ghassan

On the Privacy Provisions of Bloom Filters in Lightweight Bitcoin Clients Arthur Gervais, Ghassan O. Karame, Damian Gruber, Srdjan apkun ETH Zurich, NEC Research ACSAC 2014 Bitcoin Bitcoin Peer-to-peer decentralized currency

1.06k views • 74 slides
Service Discovery using OLSR and Bloom Filters Joakim Flathagen 4th OLSR Interop /

Service Discovery using OLSR and Bloom Filters Joakim Flathagen 4th OLSR Interop /

Service Discovery using OLSR and Bloom Filters Joakim Flathagen 4th OLSR Interop / joakim.flathagen@ffi.no Workshop, Ottawa, CA, Oct 14 2008 Presentation outline Who is involved Motivation Service Discovery General solutions

498 views • 23 slides
Compact and Distributed Data Structures Cyril Gavoille (LaBRI, University of Bordeaux) 44 th

Compact and Distributed Data Structures Cyril Gavoille (LaBRI, University of Bordeaux) 44 th

Compact and Distributed Data Structures Cyril Gavoille (LaBRI, University of Bordeaux) 44 th Ecole de Printemps en Informatique Thorique (EPIT) - Distributed Computing - le de Porquerolles, May 15-19, 2017 Information &amp; Locality

1.95k views • 181 slides
Leveraging bloom filters on Redis Cristian Castiblanco me@cristian.io | cristian@scopely.com

Leveraging bloom filters on Redis Cristian Castiblanco me@cristian.io | cristian@scopely.com

Leveraging bloom filters on Redis Cristian Castiblanco me@cristian.io | cristian@scopely.com https://cristian.io Stream processing at Scopely Stream processing at Scopely Idempotence An operation is said to be idempotent when applying it

1.17k views • 90 slides

More recommend