Bimodal Multicast And Cache Invalidation Who/What/Where Bruce - PowerPoint PPT Presentation

Bimodal Multicast And Cache Invalidation

Who/What/Where • Bruce Spang • Software Engineer/Student • Fastly

`

Powderhorn

Outline • Frame the problem • The papers we looked at • Bimodal Multicast • What we built

Content Delivery Networks

Cache Invalidation We would like to be able to update a piece of content globally

Cache Invalidation

Cache Invalidation

Cache Invalidation

Approach Notify all servers to remove a piece of content

Naïve Approach Central Service.

Central Service e g a s s e M

Central Service k c A

Central Service

Central Service e g a s s e M

Central Service k c A

Central Service Works

Problems • Unreliable • High Latency

Another Idea Cache servers can send purges themselves

Fixes • More Reliable • Lower Latency

Another Idea e g a s s e M

Another Idea k c A

Problem Every server sends an ack to the sender

Conclusion This is hard.

Reliable Broadcast

Basic Problem Send a message to a set of servers

Applications • Stock Exchanges • Control Systems • Configuration • Cache Invalidation

Atomic Broadcast • Paxos • ZooKeeper Atomic Broadcast • etc…

Strong Guarantees • Guaranteed Delivery • Total Order

Too Many Guarantees • Don’t need Ordering • Don’t need Atomicity

“Best-Effort” Broadcast “Try very hard” to deliver a message

Algorithms • Scalable Reliable Multicast • Bimodal Multicast • Plumtree • Sprinkler • etc…

` `

Goals • “Best-Effort” Delivery • Predictable Performance

Algorithm • Dissemination • Anti-Entropy (Gossip)

Dissemination • Unreliably broadcast a message to all other servers • IP multicast, UDP in a for loop, etc…

Anti-Entropy • Each server sends a random server a digest of the messages it knows about • If the other server hasn’t received some of those messages, it requests that they be retransmitted

Example

Example

Example

Gossip

Convergence

Goals • “Best-Effort” Delivery • Predictable Performance

Stable Throughput e g a s s e M

Stable Throughput Ack

Stable Throughput e g a s s e M

Stable Throughput d n e s e R

Independence A server that’s behind will recover from many servers in the cluster.

Retransmission Limits Don’t DDoS servers that are trying to recover.

Soft Failure Detection Ignore servers that are running behind.

“Best effort” is ok! • Messages are delivered • Predicable Performance

Powderhorn Bimodal Multicast in the Wild

Development All the logic is in failure handling.

Jepsen • Five nodes • Simulated partitions, packet loss • http://github.com/aphyr/jepsen

“Scale” Testing • EC2 • 200 m1.mediums

Avalanche • Random, repeatable network fault injection • https://github.com/fastly/avalanche

Garbage Collection “I have messages {1,2,3,4,5,6,7,8,9,…}”

Garbage Collection

Computers are Horrible We see high packet loss and network partitions all the time .

Convergence < 40%

The Digest List of Message IDs

The Digest Doesn’t Have to be a List

The Digest • send ranges of ids of known messages • “messages 1 to 1,000,000 from server 1" • can represent large numbers of messages

The Digest

Behavior

End-to-End Latency Density plot and 95 th percentile of purge latency by server location New York 0.10 42ms 0.05 0.00 London 0.10 74ms 0.05 Density 0.00 San Jose 0.10 83ms 0.05 0.00 Tokyo 0.10 133ms 0.05 0.00 0 50 100 150 Latency (ms)

Firewall Partition Purge performance under network partition 120 60 Throughput (messages/s) 95 th percentile latency (s) 10 90 Cache server A 1 B 60 C D 0.1 30 0 02:30 03:00 03:30 04:00 02:30 03:00 03:30 04:00 Time Time

NYC to London Partition Purge performance 120 15 Recovered purges (messages/s) Throughput (messages/s) 90 10 Cache server NYC 60 London 5 30 0 0 06:00 06:10 06:20 06:30 06:00 06:10 06:20 06:30 Time Time

APAC Packet Loss Purge performance 400 125 Recovered purges (messages/s) Throughput (messages/s) 100 300 75 Cache server Affected 200 Unaffected 50 100 25 0 0 16:30 17:00 17:30 18:00 16:30 17:00 17:30 18:00 Time Time

DDoS Purge performance under denial − of − service attack 150 60 Throughput (messages/s) 95 th percentile latency (s) 10 100 Cache server • ` 1 Victim Unaffected 50 0.1 0 23:40 23:50 00:00 00:10 00:20 23:40 23:50 00:00 00:10 00:20 Time Time

Bimodal Multicast is Great We generally don’t have to worry about purging failing, even when the network does.

Fin. brucespang.com/bimodal

Questions? brucespang.com/bimodal

We're Hiring www.fastly.com/about/jobs

Decent Hash Table Purge performance with linear probing hash − table 95 th percentile latency (ms) 10 5 0 May 07 May 08 May 09 May 10 May 11 May 12 May 13 Date