Providing Flexible Consistency Levels with Manhattan at Twitter - PowerPoint PPT Presentation

Providing Flexible Consistency Levels with Manhattan at Twitter Boaz Avital @bx

BOAZ AVITAL Tech Lead, Core Storage @bx

M A N H AT TA N

USING MANHATTAN Self service creation of applications and datasets Large multitenant clusters Seamless global replication Automatic observability and alerting # T W I T T E R M A N H AT TA N

ADOPTION More use cases need more features Use cases you designed for Use cases you didn't # T W I T T E R M A N H AT TA N

ADOPTION More use cases need more features { Use cases you Pain! designed for Use cases you didn't # T W I T T E R M A N H AT TA N

BUILDING NEW FEATURES ONE THING ALL THINGS EVERYTHING DO ONE AND THE THING WELL KITCHEN SINK Too many running services with slightly di ff erent code and Bloated software that’s not that tooling good at anything # T W I T T E R M A N H AT TA N

A R C H I T E C T U R E # T W I T T E R M A N H AT TA N

ARCHITECTURE: DATA MODEL partitioning key local key value 437698567 profile, username “@womeng” # T W I T T E R M A N H AT TA N

ARCHITECTURE: DATA MODEL partitioning key local key value 437698567 profile, username “@womeng” 437698567 profile, image “https:/ /pbs.twimg.com…” 437698567 tweets, 70309… “In which our Periscope…” 437698567 tweets, 70260… “Boston, @WomenWhoC…” 53205685 profile, username “@bx” 53205685 profile, image “https:/ /pbs.twimg.com…” 53205685 tweets, 710573… “Strong consistency in…” 53205685 tweets, 709182… “Anyone else having issu…” # T W I T T E R M A N H AT TA N

ARCHITECTURE: PARTITIONING shards partitioning key 437698567 53205685 # T W I T T E R M A N H AT TA N

ARCHITECTURE: PARTITIONING shards partitioning key 437698567 53205685 # T W I T T E R M A N H AT TA N

ARCHITECTURE: PARTITIONING shards partitioning key 437698567 53205685 # T W I T T E R M A N H AT TA N

ARCHITECTURE: MESSAGING C C C C # T W I T T E R M A N H AT TA N

ARCHITECTURE: CONSISTENCY Dynamo + Last Write Wins # T W I T T E R M A N H AT TA N

ARCHITECTURE: CONSISTENCY Dynamo + Last Write Wins # T W I T T E R M A N H AT TA N

ARCHITECTURE: CONSISTENCY Dynamo + Last Write Wins # T W I T T E R M A N H AT TA N

ARCHITECTURE: CONSISTENCY Replica Reconciliation # T W I T T E R M A N H AT TA N

BENEFITS OF EVENTUAL CONSISTENCY AVAILABILITY LATENCY SIMPLICITY # T W I T T E R M A N H AT TA N

WHEN IT’S NOT ENOUGH ? reserve() @justinbieber ? reserve() # T W I T T E R M A N H AT TA N

WHEN IT’S NOT ENOUGH reserve() reserve() # T W I T T E R M A N H AT TA N

WHEN IT’S NOT ENOUGH @username is free @username is free @username claimed by 53205685 # T W I T T E R M A N H AT TA N

ADAPTING ARCHITECTURE Logs! TWITTER DISTRIBUTEDLOG APACHE BOOKKEEPER # T W I T T E R M A N H AT TA N

ADAPTING ARCHITECTURE Operations read write compare-and-set increment # T W I T T E R M A N H AT TA N

ADAPTING ARCHITECTURE Check: @username → free Set: @username → 53205685 Check: @username → free Set: @username → 20719205 # T W I T T E R M A N H AT TA N

ADAPTING ARCHITECTURE C C C C # T W I T T E R M A N H AT TA N

ADAPTING ARCHITECTURE C C C C # T W I T T E R M A N H AT TA N

ADAPTING ARCHITECTURE C C C C # T W I T T E R M A N H AT TA N

ADAPTING ARCHITECTURE C C C C # T W I T T E R M A N H AT TA N

ADAPTING ARCHITECTURE C C C C # T W I T T E R M A N H AT TA N

RESULTS WHAT WE’VE GAINED In order updates for keys (but not full transactions) Failure isolation at shard level Mixing strong and eventual consistency in a cluster Mixing strong and eventual consistency in a dataset WHAT WE’VE LOST A few 10s of milliseconds added to average latency Latency hiccups during failures Potential for stream halts on error # T W I T T E R M A N H AT TA N

C O N S E Q U E N C E S # T W I T T E R M A N H AT TA N

A DIFFERENT CONSISTENCY MODEL EVENTUAL CONSISTENCY y, v2 x, v1 y, v2 x, v1 y, v2 y, v2 x, v1 y, v2 # T W I T T E R M A N H AT TA N

A DIFFERENT CONSISTENCY MODEL STRONG CONSISTENCY x, a1 y y x, b1 y, b2 y x, c1 y, c2 # T W I T T E R M A N H AT TA N

A DIFFERENT CONSISTENCY MODEL NO MORE REPLICA RECONCILIATION # T W I T T E R M A N H AT TA N

A DIFFERENT CONSISTENCY MODEL BACKFILL AND DATA MIGRATION Live requests Backfill data # T W I T T E R M A N H AT TA N

A DIFFERENT TOPOLOGY TRANSITION Old New # T W I T T E R M A N H AT TA N

A DIFFERENT TOPOLOGY TRANSITION 2 3 1 # T W I T T E R M A N H AT TA N

A DIFFERENT DEFINITION OF TIME x x, ttl=1001 now=1000 x, ttl=1001 ⍉ now=1003 x, ttl=1001 ⍉ now=1002 # T W I T T E R M A N H AT TA N

A DIFFERENT DEFINITION OF TIME expire value x write value x check for x, set y write value x expire value x check for x, set y write value x check for x, set y expire value x external time # T W I T T E R M A N H AT TA N

A DIFFERENT DEFINITION OF TIME x, ttl=1001 now=1000 now = 1002 x, ttl=1001 now=1002 x, ttl=1001 now=1002 # T W I T T E R M A N H AT TA N

WHAT THE USER SEES Provisioning Querying # T W I T T E R M A N H AT TA N

T H A N K YO U @ B X