Hybrid Indexes Huanchen Zhang You are running out of memory 2 You - PowerPoint PPT Presentation

Reducing the Storage Overhead of Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang

You are running out of memory 2

You are running out of memory 2

? Buy more You are running out of memory 2

TPC-C on -Store Memory Limit = 5GB Throughput 60K 20K 8M 10M 0 2M 4M 6M Transactions Executed Memory (GB) 8 Disk tuples 4 In-memory tuples Indexes 0 3

4

The better way: Use memory more efficiently 5

Indexes are LARGE Hybrid Index Benchmark % space for index 58% 34% TPC-C 55% 41% Voter 34% 18% Articles 6

Our Contributions The hybrid index architecture The Dual-Stage Transformation Applied to 4 index structures - B+tree - Skip List - Masstree - Adaptive Radix Tree (ART) Performance Space 30 – 70% 7

Did we solve this problem? -Store Throughput (txn/s) 60K 20K 8M 10M 0 2M 4M 6M TPC-C on Stay tuned Transactions Executed 8

How do hybrid indexes achieve memory savings ? Static 9

Hybrid Index: a dual-stage architecture dynamic stage static stage 10

Inserts are batched in the dynamic stage write merge dynamic stage static stage 11

Reads search the stages in order dynamic stage static stage 12

A Bloom filter improves read performance read dynamic stage static stage 13

Memory-efficient Skew-aware read write merge ~ ~ ~ ~ ~ ~ ~ ~ dynamic stage static stage 14

The Dual-Stage Transformation merge dynamic stage static stage 15

The Dynamic-to-Static Rules Compaction Reduction Compression 16

Compaction: minimize # of memory blocks 4 2 4 6 8 10 11 12 1 2 5 5 5 6 7 8 9 10 3 4 g h i j k l m n a b c d e f 17

Compaction: minimize # of memory blocks 3 6 9 1 2 3 7 8 9 10 11 12 4 5 6 l m n a b c d h i j k e f g 17

Reduction: minimize structural overhead 3 6 9 10 11 12 1 2 3 7 8 9 4 5 6 a b c i j k l m n d h e f g 18

Reduction: minimize structural overhead 4 3 6 9 2 4 6 8 10 1 2 3 7 8 9 4 5 6 10 11 12 a b c d h i j k l m n 11 12 1 2 5 5 5 6 7 8 9 10 3 4 e f g g h i j k l m n a b c d e f 18

The Dual-Stage Transformation merge dynamic stage static stage 19

The Dual-Stage Transformation Merge Questions: 1. Partial? ? ? 2. When? ? merge 3. Blocking? dynamic stage static stage 19

Did we solve this problem? B+tree -Store Throughput (txn/s) 60K 20K 8M 10M 0 2M 4M 6M TPC-C on Transactions Executed 20

Yes, we improved the DBMS’s capacity! B+tree -Store Throughput (txn/s) 60K 20K 8M 10M 0 2M 4M 6M TPC-C on Hybrid 60K 20K Transactions Executed 20

Throughput (txn/s) B+tree 60K -Store 20K Hybrid 60K 20K 4M 8M 10M 0 2M 6M TPC-C on 8 B+tree Disk tuples Memory (GB) 4 In-memory tuples Indexes 8 Hybrid 4 Transactions Executed 21

Throughput (txn/s) B+tree 60K -Store 20K Hybrid 60K 20K 4M 8M 10M 0 2M 6M TPC-C on 8 B+tree Disk tuples Memory (GB) 4 In-memory tuples Indexes 8 Hybrid 4 Transactions Executed 21

Throughput (txn/s) B+tree 60K -Store 20K Hybrid 60K 20K 4M 8M 10M 0 2M 6M TPC-C on 8 B+tree Disk tuples Memory (GB) 4 In-memory tuples Indexes 8 Hybrid 4 Transactions Executed 21

Throughput (txn/s) B+tree 60K -Store 20K Hybrid 60K 20K 4M 8M 10M 0 2M 6M TPC-C on 8 B+tree Disk tuples Memory (GB) 4 In-memory tuples Indexes 8 Hybrid 4 Transactions Executed 21

Throughput (txn/s) B+tree 60K -Store 20K Hybrid 60K 20K 4M 8M 10M 0 2M 6M TPC-C on 8 B+tree Disk tuples Memory (GB) 4 In-memory tuples Indexes 8 Hybrid 4 Transactions Executed 21

Throughput (txn/s) B+tree 60K Take Away: -Store 20K Higher Memory saved Larger working Hybrid throughput by indexes set in memory 60K 20K 4M 8M 10M 0 2M 6M TPC-C on 8 B+tree Disk tuples Memory (GB) 4 In-memory tuples Indexes 8 Hybrid 4 Transactions Executed 21

This is just the BEGINNING 22

Conclusions The hybrid index architecture GENERAL The Dual-Stage Transformation PRACTICAL Applied to 4 index structures USEFUL - B+tree - Skip List - Masstree - Adaptive Radix Tree (ART) 23

Toll-Free Hotline: 1-844-88-CMUDB