BitRot detection in GlusterFS Venky Shankar Gaurav Garg We are, - PowerPoint PPT Presentation

BitRot detection in GlusterFS Venky Shankar Gaurav Garg

We are, Gluster developers at Red Hat ... participate in meetups, open source events ... hang out on #freenode: gluster, gluster-dev nick: overclk, ggarg ... interact with community: gluster-devel@gluster.org gluster-users@gluster.org

OK, enough. Let’s get started...

GlusterFS Quick Tour

● Distributed ● Local filesystem ( brick ) XFS ○ Where’s my data? EXT3, EXT4 ○ ○ BTRFS ● Prerequisite POSIX compatible ○ Xattr support ○

Understanding data corruption

● Direct “brick” manipulation ○ Script bug Corruption? ○ Admin Malicious ○ How ?

● Silent corruption Corruption? Disk itself ○ ■ Firmware bug ■ Mechanical wear How ? ■ Ageing (cont..)

Illustration

Solution: Integrity checks

● Track data modifications Checksum (signature) ○ ○ Persistent Integrity Check ● Verify during access Recompute and check ○ ● Repair if corrupted Consistency

Enough of theory, show me how it’s done.

● Big fat-file story Implementation ● Deployments Distribute + Replicate ○ ○ Stripe, now [3.7+] sharding Constraints on choices Erasure coded ○

● In-band data signing Costly ○ Implementation RMW cycle ○ ○ Degraded I/O performance ● Verification Constraints on choices (cotd..) “ Ditto ” ○

● Out-of-band data signing ○ Daemon ○ Asynchronous Implementation ■ Policy ■ Strong hash (reason ?) ● Verification Details ○ Daemon (scrubber) On-demand ○ ○ Pre-scrubbed

● Object versioning ○ Versioned upon modification Implementation Versioning xattr (64 bit) ○ ○ Reflect “object state” ● Signature Details (cotd..) ○ xattr Attached to a “version” ○

● Integrity checking Periodic ○ ■ daily, weekly, etc.. ○ Filesystem scan ■ Signature mismates Implementation ■ Matching version ○ QoS ■ Controlled crunching Details (cotd..) ● Corrupted objects Denies access (EIO) ○ ○ Repairable ■ Replica, Codes

Use cases

● Small files Use Cases ● Long lived data Archival storage ○ WORM workload ○

● Replica consistency ● Metadata checksumming ● Offloading BTRFS ○ Future ● Sharding adaption ● GlusterFS 4.0 [Interesting!] ○ In-band ( weaker hash ) Checksum everything ○ Default ○ ○ Lost (phantom) writes

● Bitrot detection ● Bug fixes ● Hell of a change ● No recovery ● Scrub status ● Sharding by default ● In comes sharding ● Checksum everything 3.7 3.7.2 3.7.4 3.8 4.0 ● Recovery support ● Sharding ready ● Bitrot adaption

Q & A