hyperdex
play

HyperDex A Distributed, Searchable Key-Value Store Robert Escriva - PowerPoint PPT Presentation

Aug 20, 2023 •442 likes •1.02k views

HyperDex A Distributed, Searchable Key-Value Store Robert Escriva Bernard Wong Emin Gn Sirer Department of Computer Science Cornell University School of Computer Science University of Waterloo ACM SIGCOMM Conference, August

  1. HyperDex A Distributed, Searchable Key-Value Store Robert Escriva † Bernard Wong ‡ Emin Gün Sirer † † Department of Computer Science Cornell University ‡ School of Computer Science University of Waterloo ACM SIGCOMM Conference, August 14, 2012 Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 1 / 29 http://hyperdex.org/

  2. From RDBMS to NoSQL ◮ RDBMS have difficulty with scalability and performance ◮ NoSQL systems emerged to fill the gap Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 2 / 29 http://hyperdex.org/

  3. Problems Typical of NoSQL Lack of ... ◮ Search ◮ Consistency ◮ Fault-Tolerance Specifics vary between systems Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 3 / 29 http://hyperdex.org/

  4. Typical NoSQL Architecture K Consistent hashing maps each key to a server Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 4 / 29 http://hyperdex.org/

  5. The Search Problem Searching for objects without the key involves many servers Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 4 / 29 http://hyperdex.org/

  6. The Consistency Problem Clients may read inconsistent data and writes may be lost Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 5 / 29 http://hyperdex.org/

  7. The Fault-Tolerance Problem Many systems’ default settings consider a write complete after writing to just one node Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 6 / 29 http://hyperdex.org/

  8. HyperDex: An Overview ◮ Hyperspace hashing ◮ Value-dependent chaining ⇓ ◮ High-Performance: High throughput with low variance ◮ Strong Consistency: Strong safety guarantees ◮ Fault Tolerance: Tolerates a threshold of failures ◮ Scalable: Adding resources increases performance ◮ Rich API: Support for complex datastructures and search Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 7 / 29 http://hyperdex.org/

  9. Introduction Design and Implementation Hyperspace Hashing Value-Dependent Chaining Evaluation Conclusion Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 8 / 29 http://hyperdex.org/

  10. Attributes map to dimensions in a multidimensional hyperspace Phone Number First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  11. Attribute values are hashed independently Any hash function may be used Phone Number H(“607-555-1024”) H(“Neil”) First Name H(“Armstrong”) Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  12. Objects reside at the coordinate specified by the hashes Phone Number Neil Armstrong H(“607-555-1024”) H(“Neil”) First Name H(“Armstrong”) Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  13. Phone Number Neil Armstrong First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  14. Different objects reside at different coordinates Phone Number Neil Armstrong Lance Armstrong Neil Diamond First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  15. The hyperspace is divided into regions where each object resides in exactly one region Phone Number Neil Armstrong Lance Armstrong Neil Diamond First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  16. Each server is responsible for a region of the hyperspace Phone Number Neil Armstrong Lance Armstrong Neil Diamond First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  17. Each search intersects a subset of regions of the hyperspace Phone Number Neil Armstrong Lance Armstrong Neil Diamond First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  18. All people named Neil are mapped to the yellow plane Phone Number Neil Armstrong Lance Armstrong Neil Diamond First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  19. All people named Neil are mapped to the yellow plane Phone Number Neil Armstrong Lance Armstrong Neil Diamond First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  20. All people named Armstrong are mapped to the gray plane Phone Number Neil Armstrong Lance Armstrong Neil Diamond First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  21. All people named Armstrong are mapped to the gray plane Phone Number Neil Armstrong Lance Armstrong Neil Diamond First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  22. A more restrictive search for Neil Armstrong contacts fewer servers Phone Number Neil Armstrong Lance Armstrong Neil Diamond First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  23. Range searches are natively supported Phone Number Neil Armstrong Lance Armstrong Neil Diamond First Name Last Name Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 9 / 29 http://hyperdex.org/

  24. Space Partitioning ◮ In a naive implementation, the hyperspace would grow exponentially in the number of dimensions ◮ Space partitioning prevents exponential growth in the number of searchable attributes a 5 . . . a D-2 a D-1 a D a 1 a 2 a 3 a 4 k Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 10 / 29 http://hyperdex.org/

  25. Space Partitioning ◮ In a naive implementation, the hyperspace would grow exponentially in the number of dimensions ◮ Space partitioning prevents exponential growth in the number of searchable attributes a 5 . . . a D-2 a D-1 a D a 1 a 2 a 3 a 4 k a 5 . . . a D-2 a D-1 a D a 1 a 2 a 3 a 4 k key subspace 0 subspace 1 subspace S subspace ◮ A search is performed in the most restrictive subspace Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 10 / 29 http://hyperdex.org/

  26. Space Partitioning ◮ In a naive implementation, a 9-dimensional space could require 512 machines ◮ HyperDex can store this space on just 24 machines using three subspaces Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 11 / 29 http://hyperdex.org/

  27. Hyperspace Hashing Implications ◮ search es are efficient ◮ Hyperspace hashing is a mapping, not an index ◮ No per-object updates to a shared datastructure ◮ No overhead for building and maintaining B-trees ◮ Functionality gained solely through careful placement Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 12 / 29 http://hyperdex.org/

  28. Introduction Design and Implementation Hyperspace Hashing Value-Dependent Chaining Evaluation Conclusion Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 13 / 29 http://hyperdex.org/

  29. Replication ◮ As an object changes, so too must the set of servers holding it Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 14 / 29 http://hyperdex.org/

  30. Value-Dependent Chaining A C k B D key subspace subspace 1 subspace 2 Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 15 / 29 http://hyperdex.org/

  31. Value-Dependent Chaining put(k, A=1,B=1, 1 1 1 C=1,D=1) A C 1 1 1 k B D key subspace subspace 1 subspace 2 A put includes one node from each subspace Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 15 / 29 http://hyperdex.org/

  32. Value-Dependent Chaining put(k, A=0,B=0, 2 1 2 C=1,D=1) 2 A C 1 2 2 k 2 B D key subspace subspace 1 subspace 2 When updating an object, the value-dependent chain includes the servers which hold the old and new versions of the object Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 15 / 29 http://hyperdex.org/

  33. Value-Dependent Chaining put(k, A=0,B=0, 2 2 C=1,D=1) 2 A C 2 2 k 2 B D key subspace subspace 1 subspace 2 Each put removes all state from the previous put Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 15 / 29 http://hyperdex.org/

  34. Value-Dependent Chaining put(k, A=0,B=1, 3 3 C=1,D=1) 2 3 A C 3 3 k 2 3 B D key subspace subspace 1 subspace 2 Subsequent operations involve solely the most recent nodes Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 15 / 29 http://hyperdex.org/

  35. Value-Dependent Chaining A C k B D key subspace subspace 1 subspace 2 Servers are replicated in each region to provide fault tolerance Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 16 / 29 http://hyperdex.org/

  36. Value-Dependent Chaining put(k, A=0,B=0, 2 2 1 1 2 2 C=1,D=1) 2 2 A C 1 2 2 k 2 B D key subspace subspace 1 subspace 2 The value-dependent chain includes all replicas Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 16 / 29 http://hyperdex.org/

  37. Value-Dependent Chaining put(k, A=0,B=0, 2 2 1 1 2 2 C=1,D=1) 2 2 A C 1 2 2 k 2 B D key subspace subspace 1 subspace 2 Failed nodes are removed from the chain Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 16 / 29 http://hyperdex.org/

  38. Value-Dependent Chaining Implications No extra mechanism is necessary to provide ◮ Atomicity ◮ Ordering ◮ Replication ◮ Relocation Robert Escriva Bernard Wong, Emin Gün Sirer HyperDex 17 / 29 http://hyperdex.org/

More recommend