md raid 456 write journal and cache
play

MD/RAID-456 Write Journal and Cache Shaohua Li & So Song g Liu - PowerPoint PPT Presentation

Jul 14, 2023 •7 likes •847 views

MD/RAID-456 Write Journal and Cache Shaohua Li & So Song g Liu Software Engineer, Facebook MD/RAID-456 Write Journal and Cache Write holes of RAID-456 Hardware RAID: benefits and challenges Write operation in MD/RAID-456

  2. MD/RAID-456 Write Journal and Cache Shaohua Li & So Song g Liu Software Engineer, Facebook

  3. MD/RAID-456 Write Journal and Cache • Write holes of RAID-456 • Hardware RAID: benefits and challenges • Write operation in MD/RAID-456 • RAID-456 write journal: plug the write hole • RAID-456 write cache: fast fsync(), more full stripe writes • Examples

  4. Write Hole of RAID-456

  5. Failure Recovery of RAID-456 • Disk failure recovery: rebuild data from parity • Power failure recovery: resync stripes with mis-matched data and parity

  6. RAID-5: Data and Parity in Sync xor xor = D D D P

  7. RAID-5 Disk Failure D D D P

  8. RAID-5 Degraded Mode D D P

  9. RAID-5 Rebuild Data for Disk Failure D = P-D 1 -D 2 D 1 D 2 D P

  10. RAID-5 Power Failure D D P D D D D P

  11. RAID-5 Power Failure D D P D D D D P

  12. RAID-5 after Power Failure D D D P

  13. RAID-5 Resync after Power Failure drop old D D D parity

  14. RAID-5 Resync after Power Failure calculate xor xor = D D D P new parity

  15. Write Hole: Disk Failure + Power Failure D D P

  16. Write Hole: Rebuild Wrong Data D D D P

  17. Hardware RAID

  18. Hardware RAID Battery-backed RAM D D D P

  19. Hardware RAID: No Write Hole Battery-backed RAM D D P D D P

  20. Hardware RAID: No Write Hole Battery-backed RAM D D P D D P

  21. Hardware RAID: No Write Hole Battery-backed RAM D D P D D P

  22. Hardware RAID: Fast fsync() Battery-backed RAM return writes from RAM D D D D P

  23. Hardware RAID: Fast fsync() Battery-backed RAM return writes from RAM D D D D D P

  24. Hardware RAID: Full Stripe Write Battery-backed RAM D D D D D D P

  25. Hardware RAID: Full Stripe Write Battery-backed RAM D D D P D D D P

  26. Hardware RAID: Full Stripe Write Battery-backed RAM D D D P D D D P

  27. Hardware RAID @ Facebook • RAID-6 • Haystack: photo storage • GlusterFS: scalable network filesystem • RAID-0 of single HDD, for fast fsync()

  28. Challenges with Hardware RAID • Black box solution • Low transparency; low flexibility • Vendor specific toolset

  29. Make Software RAID Better • Write journal: plug write hole • Write cache: accelerate fsync(); more full stripe writes

  30. MD/RAID-456 Write

  31. RAID-456: Stripe Cache Stripe Cache (System Memory) D D D P

  32. RAID-456 Write • Step 1: update data and parity in stripe cache • Option 1: Reconstruct • Option 2: R-M-W • Step 2: write data and parity to RAID disks • Step 3: bio_endio

  33. RAID-456 Reconstruct Write Stripe Cache D D D D P

  34. RAID-456 Reconstruct Write Stripe Cache D D D D D D P

  35. RAID-456 Reconstruct Write Stripe Cache xor xor = D D D P D D D P

  36. RAID-456 Reconstruct Write Stripe Cache D D D P D D D P

  37. RAID-456 R-M-W Write Stripe Cache D D D D P

  38. RAID-456 R-M-W Write Stripe Cache D D P D D D P

  39. RAID-456 R-M-W Write Stripe Cache P=P-D+D D D P P D D D P

  40. RAID-456 R-M-W Write Stripe Cache D P D D D P

  41. MD/RAID-456 Write Journal

  42. RAID-456 Write Journal • Use block device (SSD, NVM, etc.) as the journal • No change to read path • All writes (data and parity) hit journal before committing to RAID array • For each stripe, “commit all” or “commit nothing” • Journal replay after power failure (no need for resync)

  43. RAID-456 Write Journal: Write Path • Step 1: update data and parity in stripe cache • Step 2: write data and parity to journal device • Step 3: flush journal device cache • Step 4: write data and parity to RAID disks • Step 5: bio_endio

  44. RAID-456 Write Journal: Disk Format meta block magic, checksum, version, data block meta size, seq, position data block type: data/parity/flush … flags: discard, reshape size, sector, checksum data block type: data/parity/flush meta block flags: discard, reshape data block size, sector, checksum data block … … data block

  45. RAID-456 Write Journal Stripe Cache Journal D D D P

  46. RAID-456 Write Journal Stripe Cache D P Journal D D D P

  47. RAID-456 Write Journal Stripe Cache D P Journal D D D P D P

  48. RAID-456 Write Journal Stripe Cache D P Journal D D D P D P

  49. RAID-456 Write Journal: Reclaim Path • Step 1: update journal device super block • Step 2: issue discard to journal device

  50. RAID-456 Write Journal: Recovery Path • For complete stripe (with data and parity) in journal • Replay all data/parity to RAID disks • For partial stripe (data only) in journal • Drop the journal entry

  51. After Power Failure: Replay Writes Stripe Cache Journal D D D P D P

  52. After Power Failure: Replay Writes Stripe Cache Journal D D D P D P

  53. After Power Failure: Drop Partial Journal Stripe Cache Journal D D D P D

  54. After Power Failure: Drop Partial Journal Stripe Cache Journal D D D P

  55. MD/RAID-456 Write Cache

  56. RAID-456 Write Cache • Use same disk format as the write journal • Move bio_endio to a much earlier stage • Hold data in stripe cache • Read path must look up in stripe cache • Need reclaim and smarter recovery • Opportunity for more full stripe writes

  57. RAID-456 Write Cache: Read Path • Chunk aligned read (bypass stripe cache, optimal state) • Step 1: look up data in stripe cache • Step 2: when missed stripe cache, read from disk • Step 3: amend data from disk with latest data in stripe cache • None chunk aligned read • No changes

  58. RAID-456 Write Cache: Write Path • Step 1: write data to journal device • Step 2: flush journal device cache • Step 3: bio_endio • Step 4: update data and parity in stripe cache • Step 5: write parity to journal device • Step 6: flush journal device cache • Step 7: write data and parity to RAID disks

  59. RAID-456 Write Cache: Write Path Stripe Cache D Journal D D D P

  60. RAID-456 Write Cache: Write Path Stripe Cache D Journal D D D P D

  61. RAID-456 Write Cache: Write Path Stripe Cache bio_endio D Journal D D D P D

  62. RAID-456 Write Cache: Write Path Stripe Cache D D Journal D D D P D

  63. RAID-456 Write Cache: Write Path Stripe Cache D D Journal D D D P D D

  64. RAID-456 Write Cache: Write Path Stripe Cache bio_endio D D Journal D D D P D D

  65. RAID-456 Write Cache: Reclaim Path • Step 1: update data and parity in stripe cache • Step 2: write parity to journal device • Step 3: flush journal device cache • Step 4: write data and parity to RAID disks

  66. RAID-456 Write Cache: Reclaim Path Stripe Cache D D Journal D D D P D D

  67. RAID-456 Write Cache: Reclaim Path Stripe Cache D D D Journal D D D P D D

  68. RAID-456 Write Cache: Reclaim Path Stripe Cache xor xor = D D D P Journal D D D P D D

  69. RAID-456 Write Cache: Reclaim Path Stripe Cache D D D P Journal D D D P D D P

  70. RAID-456 Write Cache: Reclaim Path Stripe Cache D D D P Journal D D D P D D P

  71. RAID-456 Write Cache: Recover Path • Stripes with data and parity in journal • Replay writes of data and parity • Stripes with data in journal • Repeat full recontruct write or R-M-W write

  72. Recover Stripe Data: Reconstruct Stripe Cache Journal D D D P D

  73. Recover Stripe Data: Reconstruct Stripe Cache D Journal D D D P D

  74. Recover Stripe Data: Reconstruct Stripe Cache D D D Journal D D D P D

  75. Recover Stripe Data: Reconstruct Stripe Cache xor xor = D D D P Journal D D D P D

  76. Recover Stripe Data: Reconstruct Stripe Cache D D D P Journal D D D P D P

  77. Recover Stripe Data: Reconstruct Stripe Cache D D D P Journal D D D P D P

  78. Recover Stripe Data: R-M-W Stripe Cache D Journal D D P D

  79. Recover Stripe Data: R-M-W Stripe Cache P=P-D+D D D P P Journal D D P D

  80. Recover Stripe Data: R-M-W Stripe Cache D P Journal D D P D P

  81. Recover Stripe Data: R-M-W Stripe Cache D P Journal D D P D P

  82. Current Status • Write Journal • Kernel changes released with kernel 4.4 • mdadm changes released with mdadm-3.4 • Write Cache • Kernel changes in progress • No change required for mdadm

  83. Examples # create array with write journal mdadm --create -f /dev/md0 -c 64 --raid-devices=4 -- level=5 /dev/sd[b-e] --write-journal /dev/sdf # check array with journal cat /proc/mdstat Personalities : [raid6] [raid5] [raid4]md0 : active raid5 sdf[4](J) sde[3] sdd[2] sdc[1] sdb[0] # add journal to existing array mdadm --manage /dev/md0 --add-journal /dev/sdf

Recommend

Distributed Systems - II Delayed-write modifications written to the cache and then

Distributed Systems - II Delayed-write modifications written to the cache and then

CSC 4103 - Operating Systems Cache Update Policy Spring 2007 Write-through write data through to disk as soon as they are placed on any cache simple reliable (little information is lost if client crashes) Lecture - XXIII

177 views • 4 slides
Lecture 23: Multiprocessors Todays topics: RAID Multiprocessor taxonomy

Lecture 23: Multiprocessors Todays topics: RAID Multiprocessor taxonomy

Lecture 23: Multiprocessors Todays topics: RAID Multiprocessor taxonomy Snooping-based cache coherence protocol 1 RAID 0 and RAID 1 RAID 0 has no additional redundancy (misnomer) it uses an array of disks and stripes

193 views • 16 slides
Write Through No Write Allocate Cache Write Reference Check tag and index Yes Tag AND

Write Through No Write Allocate Cache Write Reference Check tag and index Yes Tag AND

Write Through No Write Allocate Cache Write Reference Check tag and index Yes Tag AND No Index are Hit Miss in cache Memref Yes Memref Yes Update Cache - No Update Cache - Yes Write Through No Write Allocate Cache

345 views • 3 slides
1 Implementation Snoop Caches Implementing Snooping Caches Write Races: Multiple processors

1 Implementation Snoop Caches Implementing Snooping Caches Write Races: Multiple processors

An Example Snoopy Protocol Invalidation protocol, write-back cache Each block of memory is in one state: Cache Coherence and Memory Clean in all caches and up-to-date in memory (Shared) Consistency OR Dirty in exactly one cache

507 views • 3 slides
2 ND GRADE TUESDAY 3 / 17 H A P P Y S T. PAT R I C K S DAY JOURNAL 8:00-8:30

2 ND GRADE TUESDAY 3 / 17 H A P P Y S T. PAT R I C K S DAY JOURNAL 8:00-8:30

2 ND GRADE TUESDAY 3 / 17 H A P P Y S T. PAT R I C K S DAY JOURNAL 8:00-8:30 Objective: I can interpret my thoughts in writing. Journal Question: (In Journal Notebook) In honor of St. Patricks Day write about something

448 views • 8 slides
Lecture 22: I/O, Disk Systems Todays topics: I/O overview Disk basics RAID

Lecture 22: I/O, Disk Systems Todays topics: I/O overview Disk basics RAID

Lecture 22: I/O, Disk Systems Todays topics: I/O overview Disk basics RAID Reminder: Assignment 8 due Tue 11/21 1 Input/Output CPU Cache Bus Memory Disk Network USB DVD 2 I/O Hierarchy CPU Cache Disk

892 views • 21 slides
HOW TO WRITE A CASE COMMENT Georgetown Journal of Gender and the Law Georgetown Journal of Legal

HOW TO WRITE A CASE COMMENT Georgetown Journal of Gender and the Law Georgetown Journal of Legal

1 HOW TO WRITE A CASE COMMENT Georgetown Journal of Gender and the Law Georgetown Journal of Legal Ethics Georgetown Journal of Law & Public Policy 2 Introductions Georgetown Journal of Gender and the Law K. Daw,

622 views • 48 slides
Lecture 23: Cache, Memory, Security Todays topics: Caching policies Main memory

Lecture 23: Cache, Memory, Security Todays topics: Caching policies Main memory

Lecture 23: Cache, Memory, Security Todays topics: Caching policies Main memory system Hardware security intro 1 Cache Misses On a write miss, you may either choose to bring the block into the cache (write-allocate) or

460 views • 11 slides
Lecture 24: Cache, Memory, Security Todays topics: Caching policies Main memory

Lecture 24: Cache, Memory, Security Todays topics: Caching policies Main memory

Lecture 24: Cache, Memory, Security Todays topics: Caching policies Main memory system Hardware security intro 1 Cache Misses On a write miss, you may either choose to bring the block into the cache (write-allocate) or

146 views • 10 slides
Direct-Mapped Cache: Write Allocate with Write-Through Protocol Block size in bytes: B = 2 b WRITE

Direct-Mapped Cache: Write Allocate with Write-Through Protocol Block size in bytes: B = 2 b WRITE

Direct-Mapped Cache: Write Allocate with Write-Through Protocol Block size in bytes: B = 2 b WRITE data to address [x] n-m [w] m [d] b Cache size in blocks: M = 2 m (2 b+m bytes) Block Address A = [x] n-m [w] m Memory size in blocks = 2 n (2 b+n

529 views • 25 slides
1 Classifying cache misses Cache Organization Classifying misses by causes (3Cs) Cache size,

1 Classifying cache misses Cache Organization Classifying misses by causes (3Cs) Cache size,

Cache Performance Metrics Cache miss rate: number of cache misses divided by number of accesses Lecture 14: Hardware Approaches for Cache Optimizations Cache hit time: the time between sending address and data returning from cache Cache

608 views • 4 slides
Web Cache Consistency Web Cache Consistency Web Cache Consistency Web Cache Consistency

Web Cache Consistency Web Cache Consistency Web Cache Consistency Web Cache Consistency

Web Cache Consistency Web Cache Consistency Web Cache Consistency Web Cache Consistency Requirements of performance, availability, and disconnected operation require us to relax the goal of semantic transparency. - HTTP 1.1 specification

598 views • 13 slides
CS356 : Discussion #10 Dynamic Memory and Cache Lab Illustrations from CS:APP3e textbook Cache

CS356 : Discussion #10 Dynamic Memory and Cache Lab Illustrations from CS:APP3e textbook Cache

CS356 : Discussion #10 Dynamic Memory and Cache Lab Illustrations from CS:APP3e textbook Cache Lab Goal To write a small C simulator of caching strategies. Expect about 200-300 lines of code. Starting point in your repository.

498 views • 26 slides
Cache Coherency Cache coherent processors most current value for an address is the last

Cache Coherency Cache coherent processors most current value for an address is the last

Cache Coherency Cache coherent processors most current value for an address is the last write all reading processors must get the most current value Cache coherency problem update from a writing processor is not known to other

719 views • 30 slides
The Virtual Write Queue: Coordinating DRAM and Last-Level Cache Policies Jeffrey Stuecheli 1,2 ,

The Virtual Write Queue: Coordinating DRAM and Last-Level Cache Policies Jeffrey Stuecheli 1,2 ,

ISCA 2010 The Virtual Write Queue: Coordinating DRAM and Last-Level Cache Policies Jeffrey Stuecheli 1,2 , Dimitris Kaseridis 1 , David Daly 3 , Hillery C. Hunter 3 & Lizy K. John 1 1 ECE Department, The University of Texas at Austin 2 IBM

1.57k views • 17 slides
Algorithm Engineering (aka. How to Write Fast Code) CS260 Lecture 1 Yan Gu I/O (Cache)

Algorithm Engineering (aka. How to Write Fast Code) CS260 Lecture 1 Yan Gu I/O (Cache)

Algorithm Engineering (aka. How to Write Fast Code) CS260 Lecture 1 Yan Gu I/O (Cache) Efficiency Many slides in this lecture are borrowed from Lecture 14 in 6.172 Performance Engineering of Software Systems at MIT. The credit is to Prof.

842 views • 44 slides
L09: Cache Name: ID: Question: Direct Mapping Cache Hit Rate Consider a 4-block empty Cache,

L09: Cache Name: ID: Question: Direct Mapping Cache Hit Rate Consider a 4-block empty Cache,

L09: Cache Name: ID: Question: Direct Mapping Cache Hit Rate Consider a 4-block empty Cache, and all blocks initially marked as not valid , Given the main memory word addresses 0 1 2 3 4 3 4 15, calculate Cache hit rate. Cache Index

654 views • 9 slides
CS356 : Discussion #9 Cache Lab & Review for Midterm II Illustrations from CS:APP3e textbook

CS356 : Discussion #9 Cache Lab & Review for Midterm II Illustrations from CS:APP3e textbook

CS356 : Discussion #9 Cache Lab & Review for Midterm II Illustrations from CS:APP3e textbook Cache Lab Goal To write a small C simulator of caching strategies. Expect about 200-300 lines of code. Starting point in your

469 views • 28 slides
data level parallelism / exceptions 1 1 last time (1) PRIME+PROBE attacker fjll cache set(s)

data level parallelism / exceptions 1 1 last time (1) PRIME+PROBE attacker fjll cache set(s)

data level parallelism / exceptions 1 1 last time (1) PRIME+PROBE attacker fjll cache set(s) with attacker data let victim run, use cache set(s) cache coherency solution: invalidate or update all other caches on write glossed over details:

1.7k views • 123 slides
38. RAID Operating System: Three Easy Pieces 1 Youjip Won RAID (Redundant Array of Inexpensive

38. RAID Operating System: Three Easy Pieces 1 Youjip Won RAID (Redundant Array of Inexpensive

38. RAID Operating System: Three Easy Pieces 1 Youjip Won RAID (Redundant Array of Inexpensive Disks) Use multiple disks in concert to build a faster , bigger , and more reliable disk system. RAID just looks like a big disk to the host

884 views • 30 slides
Tutorial 9 : cache memory Why use a cache ? Main memory (VRAM/DRAM) is slow ! To deal with

Tutorial 9 : cache memory Why use a cache ? Main memory (VRAM/DRAM) is slow ! To deal with

Tutorial 9 : cache memory Why use a cache ? Main memory (VRAM/DRAM) is slow ! To deal with this, the -machine speed is reduced to match the memory read and write speed To make the machine faster, one can use a intermediate smaller

219 views • 8 slides
A RAID AT THE HEART OF THE OILIBYA RALLY OF MOROCCO Discover the Cross- Country Raid in the

A RAID AT THE HEART OF THE OILIBYA RALLY OF MOROCCO Discover the Cross- Country Raid in the

A RAID AT THE HEART OF THE OILIBYA RALLY OF MOROCCO Discover the Cross- Country Raid in the heart of the Moroccan desert! BIKE / QUAD / SSV / 4WD No need for specific preparation, you can use your own vehicle! Bring An accessible

186 views • 6 slides
distributed 1 / sockets 1 last time RAID ordering writes carefully waste space rather than

distributed 1 / sockets 1 last time RAID ordering writes carefully waste space rather than

distributed 1 / sockets 1 last time RAID ordering writes carefully waste space rather than point to unallocated/wrong fsck (fjlesystem check) recovery redo logging (writeahead logging) write intention to log; redo committed parts of

1.34k views • 96 slides
What Is Memory Hierarchy A typical memory hierarchy today: Lecture 13: Cache Basics and Cache

What Is Memory Hierarchy A typical memory hierarchy today: Lecture 13: Cache Basics and Cache

What Is Memory Hierarchy A typical memory hierarchy today: Lecture 13: Cache Basics and Cache Performance Proc/Regs L1-Cache Bigger Faster L2-Cache Memory hierarchy concept, cache L3-Cache (optional) design fundamentals, set-associative

303 views • 4 slides

More recommend