wisckey separating keys from values in ssd conscious
play

WiscKey: Separating Keys from Values in SSD-Conscious Storage - PowerPoint PPT Presentation

Jan 01, 2024 •13 likes •1.01k views

WiscKey: Separating Keys from Values in SSD-Conscious Storage Lanyue Lu, Thanumalayan Pillai, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau University of Wisconsin-Madison Key-Value Stores Key-Value Stores Key-value stores are important

  1. WiscKey: Separating Keys from Values in SSD-Conscious Storage Lanyue Lu, Thanumalayan Pillai, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau University of Wisconsin-Madison

  2. Key-Value Stores

  3. Key-Value Stores Key-value stores are important ➡ web indexing, e-commerce, social networks ➡ various key-value stores ➡ hash table, b-tree ➡ log-structured merge trees (LSM-trees)

  4. Key-Value Stores Key-value stores are important ➡ web indexing, e-commerce, social networks ➡ various key-value stores ➡ hash table, b-tree ➡ log-structured merge trees (LSM-trees) LSM-tree based key-value stores are popular ➡ optimize for write intensive workloads ➡ widely deployed ➡ BigTable and LevelDB at Google ➡ HBase, Cassandra and RocksDB at FaceBook

  5. Why LSM-trees ?

  6. Why LSM-trees ? Good for hard drives ➡ batch and write sequentially ➡ high sequential throughput ➡ sequential access up to 1000x faster than random

  7. Why LSM-trees ? Good for hard drives ➡ batch and write sequentially ➡ high sequential throughput ➡ sequential access up to 1000x faster than random Not optimal for SSDs ➡ large write/read amplification ➡ wastes device resources

  8. Why LSM-trees ? Good for hard drives ➡ batch and write sequentially ➡ high sequential throughput ➡ sequential access up to 1000x faster than random Not optimal for SSDs ➡ large write/read amplification ➡ wastes device resources ➡ unique characteristics of SSDs ➡ fast random reads ➡ internal parallelism

  9. Our Solution: WiscKey

  10. Our Solution: WiscKey Separate keys from values

  11. Our Solution: WiscKey Separate keys from values ➡ decouple sorting and garbage collection key value LSM-tree

  12. Our Solution: WiscKey Separate keys from values ➡ decouple sorting and garbage collection value key LSM-tree Value Log

  13. Our Solution: WiscKey Separate keys from values ➡ decouple sorting and garbage collection ➡ harness SSD’s internal parallelism for range queries value key LSM-tree Value Log

  14. Our Solution: WiscKey Separate keys from values ➡ decouple sorting and garbage collection ➡ harness SSD’s internal parallelism for range queries ➡ online and light-weight garbage collection value key LSM-tree Value Log

  15. Our Solution: WiscKey Separate keys from values ➡ decouple sorting and garbage collection ➡ harness SSD’s internal parallelism for range queries ➡ online and light-weight garbage collection ➡ minimize I/O amplification and crash consistent value key LSM-tree Value Log

  16. Our Solution: WiscKey Separate keys from values ➡ decouple sorting and garbage collection ➡ harness SSD’s internal parallelism for range queries ➡ online and light-weight garbage collection ➡ minimize I/O amplification and crash consistent value key LSM-tree Value Log Performance of WiscKey ➡ 2.5x to 111x for loading, 1.6x to 14x for lookups

  17. Background Key-Value Separation Challenges and Optimizations Evaluation Conclusion

  18. LSM-trees: Insertion memory disk L0 (8MB) Log L1 (10MB) L2 (100MB) L6 (ITB)

  19. LSM-trees: Insertion memory disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  20. LSM-trees: Insertion KV memory disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  21. LSM-trees: Insertion KV memory 1 disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  22. LSM-trees: Insertion 2 KV memT memory 1 disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  23. LSM-trees: Insertion 3 2 KV memT memT memory 1 disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  24. LSM-trees: Insertion 3 2 KV memT memT memory 1 4 disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  25. LSM-trees: Insertion 3 2 KV memT memT memory 1 4 disk L0 (8MB) 5 Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  26. LSM-trees: Insertion 1. Write sequentially 2. Sort data for quick lookups 3 2 KV memT memT memory 1 4 disk L0 (8MB) 5 Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  27. LSM-trees: Insertion 1. Write sequentially 2. Sort data for quick lookups 3. Sorting and garbage collection are coupled 3 2 KV memT memT memory 1 4 disk L0 (8MB) 5 Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  28. LSM-trees: Lookup memory disk L0 (8MB) Log L1 (10MB) L2 (100MB) L6 (ITB)

  29. LSM-trees: Lookup memory disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  30. LSM-trees: Lookup K memory disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  31. LSM-trees: Lookup 1 K memT memory disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  32. LSM-trees: Lookup 1 K memT 2 memory disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  33. LSM-trees: Lookup 1 K memT 2 memory disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  34. LSM-trees: Lookup 1 K memT 2 memory 3 L1 to L6 disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  35. LSM-trees: Lookup 1. Random reads 2. Travel many levels for a large LSM-tree 1 K memT 2 memory 3 L1 to L6 disk L0 (8MB) Log L1 (10MB) LevelDB L2 (100MB) L6 (ITB)

  36. I/O Amplification in LSM-trees

  37. I/O Amplification in LSM-trees 1000 Write Read 327 Amplification Ratio 100 14 10 1 100 GB

  38. I/O Amplification in LSM-trees Random load: 1000 Write Read a 100GB database 327 Amplification Ratio Random lookup: 100 100,000 lookups 14 10 1 100 GB

  39. I/O Amplification in LSM-trees Random load: 1000 Write Read a 100GB database 327 Amplification Ratio Random lookup: 100 100,000 lookups 14 10 Problems: large write amplification 1 large read amplification 100 GB

  40. Background Key-Value Separation Challenges and Optimizations Evaluation Conclusion

  41. Key-Value Separation

  42. Key-Value Separation Main idea: only keys are required to be sorted

  43. Key-Value Separation Main idea: only keys are required to be sorted Decouple sorting and garbage collection

  44. Key-Value Separation Main idea: only keys are required to be sorted Decouple sorting and garbage collection SSD device LSM-tree Value Log

  45. Key-Value Separation Main idea: only keys are required to be sorted Decouple sorting and garbage collection value key SSD device LSM-tree Value Log

  46. Key-Value Separation Main idea: only keys are required to be sorted Decouple sorting and garbage collection value key SSD device value LSM-tree Value Log

  47. Key-Value Separation Main idea: only keys are required to be sorted Decouple sorting and garbage collection value key SSD device k, addr value LSM-tree Value Log

  48. Random Load 500 LevelDB WiscKey 450 400 Throughput (MB/s) 350 300 250 200 150 100 50 0 64B 256B 1KB 4KB 16KB 64KB 256KB Key: 16B, Value: 64B to 256KB

  49. Random Load load 100 GB database 500 LevelDB WiscKey 450 400 Throughput (MB/s) 350 300 250 200 150 100 50 0 64B 256B 1KB 4KB 16KB 64KB 256KB Key: 16B, Value: 64B to 256KB

  50. Random Load load 100 GB database 500 LevelDB WiscKey 450 400 Throughput (MB/s) 350 300 250 200 150 100 only 2 MB/s to 4.1 MB/s 50 0 64B 256B 1KB 4KB 16KB 64KB 256KB Key: 16B, Value: 64B to 256KB

  51. Random Load load 100 GB database 500 LevelDB WiscKey 450 400 Throughput (MB/s) 350 large write amplification 300 (12 to 16) in LevelDB 250 200 150 100 only 2 MB/s to 4.1 MB/s 50 0 64B 256B 1KB 4KB 16KB 64KB 256KB Key: 16B, Value: 64B to 256KB

  52. Random Load load 100 GB database 500 LevelDB WiscKey 450 400 Throughput (MB/s) 350 large write amplification 300 (12 to 16) in LevelDB 250 200 150 100 only 2 MB/s to 4.1 MB/s 50 0 64B 256B 1KB 4KB 16KB 64KB 256KB Key: 16B, Value: 64B to 256KB Small write amplification in WiscKey due to key- value separation (up to 111x in throughput)

  53. Random Load load 100 GB database 500 LevelDB WiscKey 450 400 Throughput (MB/s) 350 large write amplification 300 (12 to 16) in LevelDB 250 200 150 100 only 2 MB/s to 4.1 MB/s 50 0 64B 256B 1KB 4KB 16KB 64KB 256KB Key: 16B, Value: 64B to 256KB Small write amplification in WiscKey due to key- value separation (up to 111x in throughput)

  54. LevelDB limits of files num of files L0 9 L1 (5) 30 L2 (50) 365 L3 (500) 2184 L4 (5000) 15752 L5 (50000) 23733 L6 (500000) 0

  55. LevelDB limits of files num of files L0 9 Large LSM-tree: L1 (5) 30 Intensive compaction L2 (50) 365 ➡ repeated reads/writes ➡ stall foreground I/Os L3 (500) 2184 L4 (5000) 15752 Many levels ➡ travel several levels for L5 (50000) 23733 each lookup L6 (500000) 0

  56. LevelDB WiscKey limits of files num of files num of files L0 9 7 L1 (5) 30 11 L2 (50) 365 127 L3 (500) 2184 460 L4 (5000) 15752 0 L5 (50000) 23733 0 L6 (500000) 0 0

  57. LevelDB WiscKey limits of files num of files num of files L0 9 7 L1 (5) 30 11 L2 (50) 365 127 L3 (500) 2184 460 L4 (5000) 15752 0 L5 (50000) 23733 0 L6 (500000) 0 0 Small LSM-tree: less compaction, fewer levels to search, and better caching

  58. Random Lookup 300 LevelDB WiscKey 250 Throughput (MB/s) 200 150 100 50 0 64B 256B 1KB 4KB 16KB 64KB 256KB Key: 16B, Value: 64B to 256KB

Recommend

SEPARATING UNITS SEPARATING UNITS Application Separating workpieces and media, e.g. grinding

SEPARATING UNITS SEPARATING UNITS Application Separating workpieces and media, e.g. grinding

SEPARATING UNITS SEPARATING UNITS Application Separating workpieces and media, e.g. grinding abrasives Separating within minutes Replacing time consuming manual work PERFECT SURFACES WORLDWIDE | 2 SEPARATING UNITS FLOW SEPARATION

250 views • 11 slides
hashing Nov. 10, 2017 1 RECALL: Map keys (type K) values (type V) Each (key, value) pairs is

hashing Nov. 10, 2017 1 RECALL: Map keys (type K) values (type V) Each (key, value) pairs is

COMP 250 Lecture 27 hashing Nov. 10, 2017 1 RECALL: Map keys (type K) values (type V) Each (key, value) pairs is an entry. For each key, there is at most one value. 2 RECALL Special Case: keys are unique positive integers in

783 views • 36 slides
BinarySearchTr ADTtedtoSes : unordered collection of values Iobjects - Set insert H ) ; member lx

BinarySearchTr ADTtedtoSes : unordered collection of values Iobjects - Set insert H ) ; member lx

BinarySearchTr ADTtedtoSes : unordered collection of values Iobjects - Set insert H ) ; member lx ) ; remove 4) ; empty D; sized , dear D; ) ) ( find LX - We call the values we store keys , - We assume the keys are from some ordered sets ie , for

510 views • 36 slides
Associative arrays Associative arrays map a key to a value Keys and values can be different

Associative arrays Associative arrays map a key to a value Keys and values can be different

Associative arrays Associative arrays map a key to a value Keys and values can be different data types name => Emmett arms => 2 42 => The answer Associative arrays can be implemented in many ways

125 views • 8 slides
. ( key key - total j 'D keys closer to root float wt Ivo , . . .vn . , ) weight - and

. ( key key - total j 'D keys closer to root float wt Ivo , . . .vn . , ) weight - and

- , Implementation :( as extended BST ) weight Balance Overview : : Internal node : stores - splay trees - Given asetof keys - Static X={ Xo , Optimality . ,xn . , } More frequently accessed . . splitter - and values . ( key key -

150 views • 4 slides
Map 7 January 2019 OSU CSE 1 Map The Map component family allows you to manipulate mappings

Map 7 January 2019 OSU CSE 1 Map The Map component family allows you to manipulate mappings

Map 7 January 2019 OSU CSE 1 Map The Map component family allows you to manipulate mappings from keys (of any type K ) to values (of any type V ) A Map variable holds a very simple database of keys and their associated values

2.1k views • 66 slides
7. Separating Hyperplane Theorems I Daisuke Oyama Mathematics II May 1, 2020 Separating

7. Separating Hyperplane Theorems I Daisuke Oyama Mathematics II May 1, 2020 Separating

7. Separating Hyperplane Theorems I Daisuke Oyama Mathematics II May 1, 2020 Separating Hyperplane Theorem Proposition 7.1 (Separating Hyperplane Theorem) Suppose that B R N , B = , is convex and closed, and that b / B . Then

450 views • 29 slides
2010 2500 keys > 100 uses 1250 keys > 1000 uses 2018 11000 keys >

2010 2500 keys > 100 uses 1250 keys > 1000 uses 2018 11000 keys >

2010 2500 keys > 100 uses 1250 keys > 1000 uses 2018 11000 keys > 100 uses 4450 keys > 1000 uses An excursion in to the world of OSM tagging presets Simon Poole, SOtM 2018 In the beginning Bus

686 views • 50 slides
If the id is conscious, where and what is the unconscious? Mark Solms Cape Town 1.

If the id is conscious, where and what is the unconscious? Mark Solms Cape Town 1.

If the id is conscious, where and what is the unconscious? Mark Solms Cape Town 1. Conscious id, unconscious ego 2. Homeostasis, pleasure/unpleasure, Nirvana 3. Aims & objects; instincts & learning 4. Prediction error,

394 views • 18 slides
Chapter 4 Marketing ethics Today Conscious marketing Ethical marketing decision 2

Chapter 4 Marketing ethics Today Conscious marketing Ethical marketing decision 2

Chapter 4 Marketing ethics Today Conscious marketing Ethical marketing decision 2 Definition of conscious marketing Conscious marketing The purpose of a firm goes beyond making a profit Firms decisions should be ethical

152 views • 13 slides
Separating hyperplanes S a closed, convex set Point x not in S ==> strict

Separating hyperplanes S a closed, convex set Point x not in S ==> strict

Separating hyperplanes S a closed, convex set Point x not in S ==> strict separating hyperplane Suppose S, T two closed convex sets Can they be strictly separated? Example Intersection and union (K 1 K 2 )* =

450 views • 25 slides
Welcome IN TRODUCTION TO MON GODB IN P YTH ON Donny Winston Instructor JavaScript Object

Welcome IN TRODUCTION TO MON GODB IN P YTH ON Donny Winston Instructor JavaScript Object

Welcome IN TRODUCTION TO MON GODB IN P YTH ON Donny Winston Instructor JavaScript Object Notation (JSON) Objects {} Arrays [] String keys & values Series of values [value1, value2,...] {'key1':value1, 'key2':value2,...} Order of values

267 views • 24 slides
ONE GLOBE ONE PEOPLE The conscious human the world needs you Programme 19.00 to 22.00 The

ONE GLOBE ONE PEOPLE The conscious human the world needs you Programme 19.00 to 22.00 The

ONE GLOBE ONE PEOPLE The conscious human the world needs you Programme 19.00 to 22.00 The conscious human is able to use his influence to secure a sustainable future. And yes. It requires something from you. That you understand how you

426 views • 4 slides
Password Human beings : Short keys; possibly used to generate longer keys Dictionary

Password Human beings : Short keys; possibly used to generate longer keys Dictionary

Password Human beings : Short keys; possibly used to generate longer keys Dictionary attack: adversary tries more common keys (easy with a large set of users) Trojan horse Countermeasures: slow login, close after several

563 views • 14 slides
School to Heart to Home Conscious Discipline Parent Workshop Series 1. Composure The 7 2.

School to Heart to Home Conscious Discipline Parent Workshop Series 1. Composure The 7 2.

School to Heart to Home Conscious Discipline Parent Workshop Series 1. Composure The 7 2. Assertiveness 3. Encouragement Powers of 4. Choices Conscious 5. Empathy Adults 6. Positive Intent 7. Consequences Quick Review... Survival

594 views • 25 slides
Everglades excerpts of a talk by Fritz Davis 2004 John Kunkel Small The Keys Lower

Everglades excerpts of a talk by Fritz Davis 2004 John Kunkel Small The Keys Lower

The Keys Everglades excerpts of a talk by Fritz Davis 2004 John Kunkel Small The Keys Lower keys Upper keys Miami Rocklands (Everglades Keys) Sea of saw grass. Sea of pine trees sea of tropical

201 views • 9 slides
QUESTION: How could our conscious experiences be made out of physical stuff? Consciousness poses

QUESTION: How could our conscious experiences be made out of physical stuff? Consciousness poses

QUESTION: How could our conscious experiences be made out of physical stuff? Consciousness poses the most baffling problems in the science of the mind. There is nothing that we know more intimately than conscious experience, but there is

546 views • 17 slides
Driving the 5 Keys Smiths 5 Keys 1- Aim High in Steering 2- Get the Big Picture 3-

Driving the 5 Keys Smiths 5 Keys 1- Aim High in Steering 2- Get the Big Picture 3-

Smith System Driving the 5 Keys Smiths 5 Keys 1- Aim High in Steering 2- Get the Big Picture 3- Keep Your Eyes Moving 4- Leave Yourself an Out 5- Make Sure They See You Aim im Hig igh The first rule for this method is

682 views • 10 slides
CONSCIOUS CAPITALISM Raj Sisodia Director, Mastek Ltd. Professor of Marketing, Bentley

CONSCIOUS CAPITALISM Raj Sisodia Director, Mastek Ltd. Professor of Marketing, Bentley

The New Realities for Boards in the Age of CONSCIOUS CAPITALISM Raj Sisodia Director, Mastek Ltd. Professor of Marketing, Bentley University Chairman, Conscious Capitalism Institute What Capitalism has Enabled Real GDP per Capita (2000 $)

736 views • 28 slides
Visualizing Tables Tables Rows of records (items), and

Visualizing Tables Tables Rows of records (items), and

Visualizing Tables Tables Rows of records (items), and each item has one or mul;ple columns A table has keys and values Keys:

346 views • 17 slides
Values Learning Outcomes Define what values are Identify your personal values Relate

Values Learning Outcomes Define what values are Identify your personal values Relate

Values Learning Outcomes Define what values are Identify your personal values Relate your values to your current goals in college What are Values? Write down your ideas What are Values? (cont.) According to google,

608 views • 8 slides
IDENTITY BRANDING The Effect of Packaging Design for Eco-Conscious Cleaning Products on

IDENTITY BRANDING The Effect of Packaging Design for Eco-Conscious Cleaning Products on

IDENTITY BRANDING The Effect of Packaging Design for Eco-Conscious Cleaning Products on Consumer Self-Expression and Brand Identity Tate Ragland the premise. Buyers adopt products, not just as consumer choices, but as conscious

731 views • 56 slides
(CAN,Tapestry, Pastry) Freenet: search towards keys, but no guarantees Chord: Map keys

(CAN,Tapestry, Pastry) Freenet: search towards keys, but no guarantees Chord: Map keys

Napster: central search engine (CAN,Tapestry, Pastry) Freenet: search towards keys, but no guarantees Chord: Map keys to linear search space Stoica, R. Morris, D. Karger, F. Kaashoek, and Keep pointers (fingers) into

281 views • 12 slides
Distributed Hash Tables What is a DHT? Hash Table data structure that maps keys to

Distributed Hash Tables What is a DHT? Hash Table data structure that maps keys to

Distributed Hash Tables What is a DHT? Hash Table data structure that maps keys to values essen=al building block in so?ware systems Distributed Hash Table (DHT) similar, but spread across many hosts Interface

455 views • 32 slides

More recommend