The Case for RAMClouds: Scalable High-Performance Storage Entirely in DRAM Harel Cohen Tel Aviv University
Outline ● Introduction ● RAMCloud Overview ● Motivation ● Research Issues ● RAMCloud Disadvantages ● Related Work ● Conclusion ● RAMCloud Project ● Acknowledgments ● References May 26, 2013 RAMCloud Slide 2
Introduction ● DRAM usage limited/specialized Facebook: ● Clumsy (consistency with 200 TB total data 150 TB cache! backing store) ● Lost performance (cache memcached misses, backing store) Web indexes Main-memory entirely in DRAM databases Large file caches UNIX buffer cache Main-memory DBs, again 1970 1980 1990 2000 2010 May 26, 2013 RAMCloud Slide 3
Introduction RAMCloud – harness full performance potential of large-scale DRAM storage: ● General-purpose storage system ● All data always in DRAM (no cache misses) ● Durable and available (no backing store) ● Scale: 1000+ servers, 100+ TB ● Low latency: 5-10µs remote access (100 - 1000x lower than disk) ● High throughput (100 - 1000x higher than disk) Potential impact: enable new class of applications � Large-scale Web applications � Richer query models that enable a new class of data-intensive � Scalable storage substrate May 26, 2013 RAMCloud Slide 4
RAMCloud Overview ● Storage for datacenters Application Servers ● 1000-10000 commodity servers ● 32-64 GB DRAM/server ● All data always in RAM ● Scale automatically ● Durable and available ● 100 – 1000x better disk-based storage systems ● Performance goals: � High throughput: 1M ops/sec/server Storage Servers � Low-latency access: 5-10µs Datacenter May 26, 2013 RAMCloud Slide 5
RAMCloud Overview 2009 5-10 years # servers 1000 4000 GB/server 64GB 256GB Total capacity 64TB 1PB Total server cost $4M $5M $/GB $60 $5 For $100-200K today: � One year of Amazon customer orders � One year of United flight reservations May 26, 2013 RAMCloud Slide 6
RAMCloud Overview 1000 – 100,000 Application Servers Appl. Appl. Appl. Appl. … Library Library Library Library Datacenter Coordinator Network Master Master Master Master … Backup Backup Backup Backup 1000 – 10,000 Storage Servers May 26, 2013 RAMCloud Slide 7
Motivation Application scalability ● Relational databases don’t scale ● Every large-scale Web application has problems: � Facebook: 4000 MySQL instances + 2000 memcached servers ● Major system redesign for every 10x increase in scale ● New forms of storage appearing: � Bigtable � Dynamo � PNUTS ● RAMCloud may provide a general-purpose storage system that scales far beyond existing systems May 26, 2013 RAMCloud Slide 8
Motivation Technology trends Disk access rate not keeping up with capacity: Mid-1980’s 2009 Change Disk capacity 30 MB 500 GB 16667x Max. transfer rate 2 MB/s 100 MB/s 50x Latency (seek & rotate) 20 ms 10 ms 2x Capacity/bandwidth 15 s 5000 s 333x (large blocks) Capacity/bandwidth 600 s 58 days 8333x (1KB blocks) Jim Gray's rule 5 min 30 hours 360x May 26, 2013 RAMCloud Slide 9
Motivation Technology trends ● It is not possible to access information on disk very frequently ● Disks must become more archival ● More information must move to memory ● Jim Gary’s Rule May 26, 2013 RAMCloud Slide 10
Motivation Caching ● Lost performance: � 1% misses → 10x performance degradation � Hard to approach 1% misses (Example: Facebook ~ 5-7% misses) ● Won’t save much money: � Already have to keep information in memory � Example: Facebook caches ~75% of data size ● Changes disk management issues: � Optimize for reads, vs. writes & recovery May 26, 2013 RAMCloud Slide 11
Motivation Does latency matter? Traditional Application Web Application Application Servers UI Storage Servers UI Bus. App. Logic Logic Data Structures Single machine Datacenter << 1µs latency 0.5 - 10ms latency ● Large-scale apps struggle with high latency � Facebook: make average of 130 internal requests per page � Amazon: 100 – 200 internal requests per page May 26, 2013 RAMCloud Slide 12
Motivation Does latency matter? Traditional Application Web Application Application Servers UI Storage Servers UI Bus. App. Logic Logic Data Structures Single machine Datacenter << 1µs latency 0.5 - 10ms latency 5 - 10µs ● RAMCloud goal: large scale and low latency ● New “one size fits all” database ● Enable a new breed of information-intensive applications May 26, 2013 RAMCloud Slide 13
Motivation Use flash memory instead of DRAM? ● Many candidate technologies besides DRAM � Flash Memory (NAND, NOR) � PCRAM (Phase-Change Memory) ● FlashCloud would be cheaper and consume less energy than a DRAM-based approach ● DRAM enables lowest latency today: � 5 - 10x faster than flash memory ● RAMCloud provide higher throughput than FlashCloud ● Most RAMCloud techniques will apply to other technologies May 26, 2013 RAMCloud Slide 14
Motivation Use flash memory instead of DRAM? May 26, 2013 RAMCloud Slide 15
Motivation RAMCloud applicability today ● Facebook: 200 TB of (non-image) data in 2009 ● Online Retailer – (Example : Amazon): Revenues/year: ~ $16B Orders/year: ~ 400M ($40/order) Bytes/order: ~ 1000 - 10000 Order data/year: ~ 400 GB - 4.0 TB RAMCloud cost: ~ $24 - 240K ● Airline Reservations – (Example: United Airlines): Total flights/day: ~ 4000 (30,000 for all airlines in U.S.) Passenger flights/year: ~ 200M Bytes/passenger-flight: ~ 1000 - 10000 Order data/year: ~ 220 GB - 2.2 TB RAMCloud cost: ~ $13 - 130K Ready today for almost all online data; media soon… May 26, 2013 RAMCloud Slide 16
Research Issues Some challenging research issues: ● Low latency RPC ● Durability and availability ● Data model ● Distribution and scaling ● Concurrency, transactions, and consistency ● Multi-tenancy ● Server-client functional distribution ● Self-management May 26, 2013 RAMCloud Slide 17
Research Issues Low latency RPC Achieving 5 - 10µs will impact every layer of the system: ● Must reduce network latency: � Typical today: Switch 300-500 µs roundtrip (10-30 µs/switch, 5 switches each way) Switch Switch � Arista 7100S 10GB switch : 9 µs roundtrip (0.9 µs/switch) Switch Switch � Need cut-through routing Client Server ● Tailor OS on server side: � Dedicated cores � No interrupts? � No virtual memory? May 26, 2013 RAMCloud Slide 18
Research Issues Low latency RPC ● Network protocol stack: � TCP too slow (especially with packet loss) � Must avoid copies ● Client side: need efficient path through VM: � User-level access to network interface? ● Preliminary experiments: � 10 - 15 µs roundtrip � Direct connection: no switches � Biggest current problem: NICs (3-5 µs each way) May 26, 2013 RAMCloud Slide 19
Research Issues Durability and availability ● Data must be durable when write RPC returns ● Unattractive approaches: � Replicate in other memories (too expensive – money and energy ) � Synchronous disk write (100 - 1000x too slow) ● One possibility: buffered logging write log log DRAM DRAM DRAM Storage async, batch Servers disk disk disk May 26, 2013 RAMCloud Slide 20
Research Issues Durability and availability ● Potential problem: crash recovery � If master crashes, data unavailable until read from disks on backups � Read 64 GB from one disk? 10 minutes � Our goal: recover in 1 - 2 seconds ● Solution: take advantage of system scale � Shard backup data across many servers � Recover in parallel � Truncated disk logs ● System throughput will be limited by disk bandwidth for writing: 50,000 updates/second (vs. 1M read/second ) � Single disk per server with 100 MB/second throughput � Typical updates of a few hundred bytes � It make sense to use Flash memory instate of disk May 26, 2013 RAMCloud Slide 21
Research Issues Durability and availability ● Potential problem: power failures ● Solution: one of three ways � Servers continue operating long to flush data to disk (buffered logging – few seconds, replication – 10 minutes) � Applications tolerate the loss of unwritten log data (buffered logging ) � All data committed to stable storage ● Potential problem: high bit error rates for DRAM ● Solution: special attention � ECC memory is not enough � Augment stored objects with check-sums � Scan stored data periodically to detect latent errors. May 26, 2013 RAMCloud Slide 22
Research Issues Data model ● Data model aspects � What is the nature of the basic objects stored in the system? � How are basic objects organized into higher-level structures? � When retrieving or modifying basic objects? how are the objects named? ● Highly-structured data models � may not be practical to scale the relational structures over large numbers of servers ● Unstructured models � may not be rich enough to support a variety of applications conveniently May 26, 2013 RAMCloud Slide 23
Research Issues Data model Identifier (64b ) Richer model in the future: Version (64b) • Indexes? • Transactions? Blob ( ≤ 1MB) • Graphs? Object One-Size-Fits-All May 26, 2013 RAMCloud Slide 24
Recommend
More recommend
