Programming Distributed Systems 08 Replicated Data Types Annette Bieniusa AG Softech FB Informatik TU Kaiserslautern Summer Term 2018 Annette Bieniusa Programming Distributed Systems Summer Term 2018 1/ 37
Motivation So far, we resolved conflicting updates (i.e. non-commutative) updates simply by sequencing operations using arbitration order ( ar ). But sometimes, applications do not want to depend on a global order such as ar do want to be made aware of conflicts do want to resolve conflicts in a type-specific way. Annette Bieniusa Programming Distributed Systems Summer Term 2018 2/ 37
Example: Multi-value register Annette Bieniusa Programming Distributed Systems Summer Term 2018 3/ 37
How can we determine the state? Annette Bieniusa Programming Distributed Systems Summer Term 2018 4/ 37
Formal model Annette Bieniusa Programming Distributed Systems Summer Term 2018 5/ 37
Sequential semantics for registers S : Op × Op ∗ → V al S ( rd, ǫ ) = undef (read returns initial value) S ( rd, wr (2) · wr (8)) = 8 (read returns last value written) S ( wr (3) , rd · wr (2) · wr (8)) = ok (write always returns ok) Annette Bieniusa Programming Distributed Systems Summer Term 2018 6/ 37
Operation Context An operation context is a finite event graph C = ( E, op, vis, ar ) . Events in E capture what prior operations are visible to the operation that is to be performed. Annette Bieniusa Programming Distributed Systems Summer Term 2018 7/ 37
Concurrent semantics for Multi-Value Register F : Op × C → V al F mvr ( wr ( x ) , C ) = ok F mvr ( rd () , C ) = { x | exists e in C such that op ( e ) = wr ( x ) and e is vis-maximal in C } Annette Bieniusa Programming Distributed Systems Summer Term 2018 8/ 37
Quizz: What do the read ops return? Annette Bieniusa Programming Distributed Systems Summer Term 2018 9/ 37
Annette Bieniusa Programming Distributed Systems Summer Term 2018 10/ 37
Return values in Abstract Executions revisited Last lecture: Consistency Notions can be defined alternatively with concurrency semantics An abstract execution A = ( E, op, rval, rb, ss, vis, ar ) satisfies a concurrent semantics F if rval ( e ) = F ( op ( e ) , A | vis − 1 ( e ) ,op,vis,ar Annette Bieniusa Programming Distributed Systems Summer Term 2018 11/ 37
Semantics of a set data type Sequential specification of abstract data type Set S : { true } add(e) { e ∈ S } { true } rmv(e) { e / ∈ S } The following pairs of operations are commutative (for two elements e, f and e � = f ): { true } add(e); add(e) { e ∈ S } { true } add(e); add(f) { e, f ∈ S } { true } rmv(e); rmv(e) { e / ∈ S } { true } rmv(e); rmv(f) { e, f / ∈ S } { true } add(e); rmv(f) { e ∈ S, f / ∈ S } For these ops, the concurrent execution should yield the same result as executing the ops in any order. Annette Bieniusa Programming Distributed Systems Summer Term 2018 12/ 37
What are the options regarding a concurrency semantics for add(e) and rmv(e)? The operations add(e) and rmv(e) are not commutative { true } add(e); rmv(e) { e / ∈ S } { true } rmv(e); add(e) { e ∈ S } Options sequential consistency / linearizability → not really. . . add-wins: e ∈ S remove-wins: e / ∈ S erroneous state last-writer wins (i.e. define arbitration order through total order, for example, via totally- ordered timestamps) Annette Bieniusa Programming Distributed Systems Summer Term 2018 13/ 37
Set Semantics Annette Bieniusa Programming Distributed Systems Summer Term 2018 14/ 37
Formal Semantics for the Add-Wins Set F aws ( add ( x ) , C ) = ok F aws ( rmv ( x ) , C ) = ok F aws ( rd () , C ) = { x | exists e in C such that op ( e ) = add ( x ) and there exists no e’ in C such that op ( e ′ ) = rmv ( x ) and e vis → e ′ } − − Annette Bieniusa Programming Distributed Systems Summer Term 2018 15/ 37
Sets with odd semantics Grow-only set Convergence by union on element set No remove operation 2P-Set (Wuu & Bernstein PODC 1984) Set of added elements + set of tombstones Add/remove once Violates sequential spec c-set (Sovran et al., SOSP 2011) Maintain add/remove counter Violates sequential spec Annette Bieniusa Programming Distributed Systems Summer Term 2018 16/ 37
Conflict-free Replicated Data Types (CRDTs) [3] Same API as sequential abstract data type, but with concurrency semantics If operations are commutative, same semantics as in sequential execution Otherwise, need arbitration to resolve conflict Don’t loose updates! Results should not depend on the order received Semantics close to sequential version Annette Bieniusa Programming Distributed Systems Summer Term 2018 17/ 37
CRDTs and Consistency Strong Eventual Consistency Eventual delivery: Every update is eventually applied at all correct replicas Termination: Update operation terminates Strong convergence: Correct replicas that have applied the same update have equivalent state Annette Bieniusa Programming Distributed Systems Summer Term 2018 18/ 37
CRDT catalogue Register (Laster-writer wins, Multi-value) Set (Grow-Only, Add-Wins, Remove-Wins) Flags Counter (unlimited, restricted/bounded) Graph (directed, monotone DAG) Sequence / List Map, JSON Annette Bieniusa Programming Distributed Systems Summer Term 2018 19/ 37
Specification: Replicated counter All operations commute, no conflict resolution is needed Value returned depends only on E and op , but not on vis and ar F ctr ( rd () , ( E, op, vis, ar )) = |{ e ′ ∈ E | op ( e ′ ) = inc }| Next: Operational Semantics Annette Bieniusa Programming Distributed Systems Summer Term 2018 20/ 37
State-based CRDTs Annette Bieniusa Programming Distributed Systems Summer Term 2018 21/ 37
State-based specifications Synchronization by propagating replica state Updates must inflate the state State must form a join semi-lattice wrt merge ⇒ Merge must be idempotent, commutative, associative Annette Bieniusa Programming Distributed Systems Summer Term 2018 22/ 37
Join-semilattice A join-semilattice S is a set that has a join (i.e. a least upper bound) for any nonempty finite subset: For all elements x, y ∈ S , the least upper bound (LUB) x ⊔ y exists. A semilattice is a commutative, idempotent and associative. A partial order on the elements of S is induced by setting x ≤ y iff x ⊔ y = y . Annette Bieniusa Programming Distributed Systems Summer Term 2018 23/ 37
Examples Annette Bieniusa Programming Distributed Systems Summer Term 2018 24/ 37
Example: Counter Annette Bieniusa Programming Distributed Systems Summer Term 2018 25/ 37
Operation-based CRDTs Annette Bieniusa Programming Distributed Systems Summer Term 2018 26/ 37
Op-based specifications Synchronization by propagating operations and re-apply Concurrent updates must commute Requires reliable causal delivery Annette Bieniusa Programming Distributed Systems Summer Term 2018 27/ 37
Example: Counter Annette Bieniusa Programming Distributed Systems Summer Term 2018 28/ 37
Example: Add-wins Set (Observed-remove Set) Annette Bieniusa Programming Distributed Systems Summer Term 2018 29/ 37
Example: Add-wins Set (Observed-remove Set) Annette Bieniusa Programming Distributed Systems Summer Term 2018 30/ 37
Optimized version of Add-wins Set Possible to garbage-collect the tombstone after remove Trick: Assuming causal delivery, element will never be re-introduced (with the same id)[2] Annette Bieniusa Programming Distributed Systems Summer Term 2018 31/ 37
Optimized version of Add-wins Set Annette Bieniusa Programming Distributed Systems Summer Term 2018 32/ 37
Challenges with CRDTs Meta-data overhead for CRDTs that require causal contexts Version vectors track concurrent modifications (O(N) ⇒ Churn!) Monotonically growing state with state-based approach Tombstones ⇒ Garbage collection Composability for recursive CRDTs (Maps, Sequences; Transactions) What is the desirable semantics of the composed entity? Annette Bieniusa Programming Distributed Systems Summer Term 2018 33/ 37
Delta-based CRDTs State-based CRDTs suffer from monotonically growing state Op-based CRDTs require causal delivery Delta-based CRDTs[1] Small message comprising a set of incremental updates Work over unreliable communication channels A delta-mutator m δ is a function, corresponding to an update operation, which takes a state X in a join-semilattice S as parameter and returns a delta-mutation m δ ( X ) , also in S . X ′ = X ⊔ m δ ( X ) Annette Bieniusa Programming Distributed Systems Summer Term 2018 34/ 37
Adoption of CRDTs in industry Annette Bieniusa Programming Distributed Systems Summer Term 2018 35/ 37
Conclusion CRDTs provide Strong Eventual Consistency (sometimes even more) Data-type specific conflict resolution Deterministic (independent of local update order) Relation to sequential semantics Meta-data overhead can be substantial Annette Bieniusa Programming Distributed Systems Summer Term 2018 36/ 37
Recommend
More recommend
