Reducing the Storage Overhead of Main-Memory OLTP Databases with C o - a u t h o r s : Hybrid Indexes D a v i d G . A n d e r s e n A n d r e w P a v l o M i c h a e l K a m i n s k y Huanchen Zhang L i n M a R u i S h e n
You are running out of memory 2
You are running out of memory 2
? Buy more You are running out of memory 2
TPC-C on -Store Memory Limit = 5GB Throughput 60K 20K 8M 10M 0 2M 4M 6M Transactions Executed Memory (GB) 8 Disk tuples 4 In-memory tuples Indexes 0 3
4
The better way: Use memory more efficiently 5
Indexes are LARGE Hybrid Index Benchmark % space for index 58% 34% TPC-C 55% 41% Voter 34% 18% Articles 6
Our Contributions The hybrid index architecture The Dual-Stage Transformation Applied to 4 index structures - B+tree - Skip List - Masstree - Adaptive Radix Tree (ART) Performance Space 30 – 70% 7
Did we solve this problem? -Store Throughput (txn/s) 60K 20K 8M 10M 0 2M 4M 6M TPC-C on Stay tuned Transactions Executed 8
How do hybrid indexes achieve memory savings ? Static 9
Hybrid Index: a dual-stage architecture dynamic stage static stage 10
Inserts are batched in the dynamic stage write merge dynamic stage static stage 11
Reads search the stages in order read read dynamic stage static stage 12
A Bloom filter improves read performance read read dynamic stage static stage 13
Memory-efficient Skew-aware read read write merge ~ ~ ~ ~ ~ ~ ~ ~ dynamic stage static stage 14
The Dual-Stage Transformation merge dynamic stage static stage 15
The Dynamic-to-Static Rules Compaction Reduction Compression 16
Compaction: minimize # of memory blocks 4 2 4 6 8 10 1 2 5 5 5 6 7 8 9 10 11 12 3 4 e f g h i j k l m n a b c d 17
Compaction: minimize # of memory blocks 3 6 9 1 2 3 7 8 9 10 11 12 4 5 6 i j k l m n a b c d h e f g 17
Reduction: minimize structural overhead 3 6 9 10 11 12 1 2 3 7 8 9 4 5 6 a b c i j k l m n d h e f g 18
Reduction: minimize structural overhead 4 3 6 9 2 4 6 8 10 1 2 3 7 8 9 4 5 6 10 11 12 a b c d h i j k l m n 1 2 5 5 5 6 7 8 9 10 11 12 3 4 e f g e f g h i j k l m n a b c d 18
The Dual-Stage Transformation Merge Questions: 1. Partial? ? ? 2. When? ? merge 3. Blocking? dynamic stage static stage 19
Did we solve this problem? B+tree -Store Throughput (txn/s) 60K 20K 8M 10M 0 2M 4M 6M TPC-C on Transactions Executed 20
Yes, we improved the DBMS’s capacity! B+tree -Store Throughput (txn/s) 60K 20K 8M 10M 0 2M 4M 6M TPC-C on Hybrid 60K 20K Transactions Executed 20
Throughput (txn/s) B+tree 60K -Store 20K Hybrid 60K 20K 4M 8M 10M 0 2M 6M TPC-C on 8 B+tree Disk tuples Memory (GB) 4 In-memory tuples Indexes 8 Hybrid 4 Transactions Executed 21
Throughput (txn/s) B+tree 60K -Store 20K Hybrid 60K 20K 4M 8M 10M 0 2M 6M TPC-C on 8 B+tree Disk tuples Memory (GB) 4 In-memory tuples Indexes 8 Hybrid 4 Transactions Executed 21
Throughput (txn/s) B+tree 60K -Store 20K Hybrid 60K 20K 4M 8M 10M 0 2M 6M TPC-C on 8 B+tree Disk tuples Memory (GB) 4 In-memory tuples Indexes 8 Hybrid 4 Transactions Executed 21
Throughput (txn/s) B+tree 60K -Store 20K Hybrid 60K 20K 4M 8M 10M 0 2M 6M TPC-C on 8 B+tree Disk tuples Memory (GB) 4 In-memory tuples Indexes 8 Hybrid 4 Transactions Executed 21
Throughput (txn/s) B+tree 60K Take Away: -Store 20K Higher Memory saved Larger working Hybrid throughput by indexes set in memory 60K 20K 4M 8M 10M 0 2M 6M TPC-C on 8 B+tree Disk tuples Memory (GB) 4 In-memory tuples Indexes 8 Hybrid 4 Transactions Executed 21
This is just the BEGINNING 22
Conclusions The hybrid index architecture GENERAL The Dual-Stage Transformation PRACTICAL Applied to 4 index structures USEFUL - B+tree - Skip List - Masstree - Adaptive Radix Tree (ART) 23
Toll-Free Hotline: 1-844-88-CMUDB
Recommend
More recommend
