Lost in transaction? Strategies to deal with (in)consistency in distributed systems @berndruecker
Once upon a time: try { tx.begin(); doA(); Do A doB(); tx.commit(); All or + } catch (Exception e) { tx.rollback(); nothing } Do B Or simply: @Transactional public void createCustomer(Customer cust) { // ... }
A Atomicity C Consistency I Isolation D Durability
Distributed systems
Distributed systems
Distributed systems
But there is two-phase commit (XA)!! Resource TX Managers Coordinator Prepare Phase Commit Phase
“ Pat Helland Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce
“ Grown- Ups Don’t Use Distributed T ransactions Pat Helland Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce
Starbucks does not use two phase commit https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html Photo by John Ingle
Atomicity Eric Brewer Consistency Isolation Durability http://pld.cs.luc.edu/courses/353/spr11/notes/brewer_keynote.pdf
That means Consistent Violates „I“ Local Do A ACID of ACID Temporarily inconsistent Local Do B ACID Eventually consistent 1 (micro-)service again 1 aggregate t 1 program 1 resource
Consistent You might know this from: Do A Temporarily inconsistent Do B Eventually consistent again t Photo by Gerhard51, available under Creative Commons CC0 1.0 license.
A C I D 2.0 Pat Helland „Building on Quicksand“ Paper
Associative (a + b) + c = a + (b + c) Commutative a + b = b + a Idempotent f(x) = f( f(x) ) Distributed 2.0 Pat Helland „Building on Quicksand“ Paper
Photo by pixabay, available under Creative Commons CC0 1.0 license.
Requirement: Idempotency of services! Photo by pixabay, available under Creative Commons CC0 1.0 license.
Requirement: Idempotency of services! Photo by Chr.Späth , available under Public Domain.
Distributed
It is impossible to Client Service Provider differentiate certain failure scenarios: Independant of communication style!
Network problems Credit Payment Card charge
Strategy: retry has to be idempotent Credit Payment Card charge Charge Credit Card Not Not idempotent cardNumber amount Charge Credit Card cardNumber Idempotent amount transactionId
Strategy: Cleanup Credit Payment Card charge Raise payment failed Make sure it is not charged! Cancel charge cardNumber amount transactionId
Some communication challenges require state.
Strategy: Stateful retry Credit Payment Card charge
Strategy: Stateful retry Credit Payment Card charge Make sure it is not charged!
Warning: Contains Opinion
Bernd Ruecker Co-founder and Chief T echnologist of Camunda Berlin, Germany bernd.ruecker@camunda.com @berndruecker
Let‘s use a lightweight OSS workflow engine for this:
Stateful retry Credit Payment Card REST
Stateful retry & cleanup Credit Payment Card REST Cancel charge
Live hacking https://github.com/flowing/flowing-retail/tree/master/rest
Embedded Engine Example (Java) https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
Remote Engine Example (Polyglot) https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
A relatively common pattern Kafka / Rabbit 1. Receive 3. Send response ? ACK 4. Send additional events Service (e.g. Go) 2. Business Logic RDMS
That means Consistent Do A Temporarily inconsistent Do B In case of failure: Compensate or apologise Eventually consistent t
https://blogs.msdn.microsoft.com/pathelland/2007/05/15/memories-guesses-and-apologies/
Compensation – the classical example Saga 1. 2. 3. book book book hotel car flight book trip In case of failure cancel cancel trigger compensations hotel car 6. 5.
2 alterntive approaches: choreography & orchestration
Event-driven choreography Hotel booked Hotel T rip Car requested Request trip T rip Car booked T rip booked Flight Flight booked
Event-driven choreography Perform undo (cancel hotel) Hotel booked Perform undo (cancel car booking) Hotel T rip Car requested Request Car trip canceled Hotel T rip canceled Car booked T rip Flight failed Flight failed
The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
Classical example Saga 1. 2. 3. book book book hotel car flight book trip In case of failure cancel cancel trigger compensations hotel car 6. 5.
If your transaction involves 2 to 4 steps, choreography might be a very good fit. However, this approach can rapidly become confusing if you keep adding extra steps in your transaction as it is difficult to track which services listen to Denis Rosa Couchbase which events. Moreover, it also might add a cyclic dependency between services as they have to subscribe to one another’s events. https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part/
Microservice pioneers have become aware
We have a new basic agreement Implementing changes in the process with the car rental agency and can cancel for free within 1 hour – do that first! Hotel booked Hotel T rip Car requested Request Car trip canceled Hotel T rip canceled Car booked T rip Flight failed Flight failed
We have a new basic agreement Implementing changes in the process with the car rental agency and can cancel for free within 1 hour – do that first! Hotel booked Hotel T rip Car requested Request Car trip canceled Hotel T rip canceled Car booked T rip Flight failed Flight failed You have to adjust all services and redeploy at the same time!
Photo by born1945, available under Creative Commons BY 2.0 license.
What we wanted Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0
Orchestration Hotel Book Car Book hotel car Request Hotel trip booked Car T rip booked T rip Book booked flight Flight Flight booked
We have a new basic agreement Orchestration with the car rental agency and can cancel for free within 1 hour – do that first! Hotel Book Car Book hotel car Request Hotel trip booked Car T rip booked T rip Book booked flight Flight Flight booked You have to adjust one service and redeploy only this one!
Describe orchestration with BPMN Request trip T rip T rip booked
The workflow is part of the service T rip
The workflow is part of the service T rip Payment
The workflow is part of the service T rip Workflow engine itself can run decentralized or centralized Payment
Caitie McCaffrey | @caitie
„Can this really handle 15k requests per second ?“
Graphical models?
Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
BPMN Business Process Model and Notation ISO Standard
Living documentation for long-running behaviour
Visual HTML reports for test cases
BizDevOps
Saga with AWS Step Functions https://theburningmonk.com/2017/07/applying-the- saga-pattern-with-aws-lambda-and-step-functions/
Thoughts on the state machine | workflow engine market
Thoughts on the state machine | workflow engine market Camunda, Zeebe, jBPM, Activiti , Mistral, … Stack Vendors, OSS Workflow or Integration Frameworks Pure Play BPMS Orchestration Engines Apache Camel, Low Code Platforms Balerina , … PEGA, IBM, SAG, … Data Homegrown frameworks Pipelines Cloud Offerings to scratch an itch Apache Airflow, Spring Data Flow, … AWS Step Functions, Uber, Netflix, AirBnb , ING, … Azure Durable Functions , …
Does it support stateful operations? Does it support the necessary flow logic? Does it support BizDevOps? Does it scale?
Recommend
More recommend
