Opportunities & Pitfalls of Event-Driven Utopia @berndruecker
Why this talk
Why this talk
Agenda Events on the outside 3 Events on the inside 1 Service 1 Service 2
Agenda Events on the outside 3 Events on the inside 1 Service 1 Service 2 Events inside out 2
Agenda Events on the outside 3 Events on the inside 1 Service 1 Service 2 Events inside out 2
Once upon a time… Client BBC architecture Application (box - arrow – box – arrow - cylinder) Every architecture diagram you'll ever need RDMS
The great thing about this architecture Application DB gurantees RDMS (e.g. ACID)
The problem Application RDMS RDMS not webscale resiliency is expensive
Pat Helland https://vimeo.com/52831373
Persistent change Persistent state Append-only Log RDMS +2,500 $ … … transfered Account # Balance 12345 2,500$ bank -14.99$ account paid by created credit card Current Balance = 2,485.01 $
Persistent change Event Bank Account Created Append-only 2019/04/16 11:00 Log # 12345 +2,500 $ … … transfered Event bank -14.99$ Money T ransfer Received account paid by created credit card 2,500$ 2019/04/16 11:00 # 12345
Event Sourcing in a nutshell Customer Internal State 6. Async publish 3. Build 4. Validation and 1. Create Customer Customer Created Domain Event internal state Invariant Checks Business Save Replay Logic event 2. Read events 5. Add Customer Created Event … Customer Event Store e.g. Customer Created, Customer Credit Limit Approved , …
Working without distributed transactions Customer Internal State 6. Async publish 3. Build 4. Validation and 1. Create Customer Customer Created Domain Event internal state Invariant Checks Business Save This is the only Replay Logic event atomic operation required 2. Read events 5. Add Customer Created Event … Customer Event Store e.g. Customer Created, Customer Credit Limit Approved , …
T raditional Architecture Customer Account 3. Remote Communication Business Open 1. Create Customer Logic Account 2. Persist state changes RDMS
@berndruecker “ Pat Helland Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce
“ Grown- Ups Don’t Use Distributed T ransactions Pat Helland Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce
Outbox pattern in traditional architectures Customer Account TX 2 TX 1 3. Remote Job: Execute: Communication Open 1. Create Customer Business „Open „Open Account Logic Account“ Account“ Async after first transaction! 2. Persist state changes RDMS
Outbox pattern – Implementation Approaches Scheduler Database T ransaction Log Workflow Automation
Idempotency Customer Account TX 2 TX 1 3. Remote 1. Create Job: Execute: Communication Capture Business Customer „Open „Open Request Logic Async after Account“ Account“ transaction! 2. Persist state changes RDMS
Idempotency Customer Account TX 3 TX 1 TX 2 3. Remote 1. Create Job: Execute: Communication Capture Business Customer „Open „Open Request Logic Async after Account“ Account“ transaction! 2. Persist state changes RDMS
Working without distributed transactions Customer Internal State 6. Async publish 3. Build 4. Validation and 1. Create Customer Customer Created Domain Event internal state Invariant Checks Business Save This is the only Replay Logic event atomic operation required 2. Read events 5. Add Customer Created Event … Customer Event Store e.g. Customer Created, Customer Credit Limit Approved , …
Events on the inside. An example from my world
Bernd Ruecker Co-founder and Chief T echnologist of Camunda mail@berndruecker.io @berndruecker
We offer two different workflow engines. Why? Camunda Persistent Zeebe Persistent State change
2. 1. 3. Workflow Engine 1. INSERT RDMS Workflow Current State Instance Activity Id 2. UPDAT E 2 RetrievePayment running Workflow Current State Instance Activity Id 3. UPDAT E 2 ShipGoods running Workflow Current State Instance Activity Id 2 OrderDelivered ended
2. 1. Workflow Engine Append-only 1. 2. Log create start workflow event activity lock complete activity instance occured activated created task completed … … workflow sequence task task task sequence instance flow taken created locked completed flow taken created
Event Handling, Replication & Single Writer 1 complete task send command Broker (Leader) 2 append command task completed store & replicate 3 event command store & replicate 7 5 respond event 6 append event Follower 4 process Follower Single Writer Stream (single thread) Processor
What we do different Store and replay 1 complete task send command commands Leader 2 append command task completed event store & replicate 3 command store & replicate 7 event 6 append event 5 respond Follower 4 process Persist & replicate Follower Delete records that Single Writer Stream internal state (single thread) are fully processed Processor
Consistency Availability Partition
Zeebe is CP 1 complete task send command Leader 2 append command task completed event store & replicate 3 command store & replicate 7 event 6 append event 5 respond Follower 4 process Follower Single Writer Stream (single thread) Processor
Horizontal scalability by partitioning Stream Single Writer Processor (single thread) Partition 1 instance id: 2-42 Partition 2 instance id: 3-66 Partition 3 Every workflow instance is exactly handled by one partition Partition 4
Queries and read models Zeebe ask Zeebe Broker Broker Streaming ask Exporter
@berndruecker Recap 1 – Events on the inside # Natural mechanism to build scalable services in distributed systems (with Outbox & co included) But # You have to think about reads, queries & eventual consistency # Much less industry experience available
Agenda Events on the outside 3 Events on the inside 1 Service 1 Service 2 Events inside out 2
Event Store and Messaging Customer 1. Create Customer … Customer Event Store
Merge Messaging and Event Store Customer 1. Create Customer … Customer Event Store
Merge messaging and event store Customer 1. Create Customer … Shared Event Store
Enter the world of Kafka…
Merge messaging and event store Customer 1. Create Customer … Shared Event Store
Kafka as transport 1. Create Customer … Used as queue (but persistent!) Customer
Agenda Events on the outside 3 Events on the inside 1 Service 1 Service 2 Events inside out 2
Once upon a time Customer Change Address Billing
Event Notification Customer Address changed Billing
What‘s Event Notification Reverse direction direction of dependency of dependency Customer Customer Address Change changed Address Billing Billing
Event Notification Customer AdressChanged { Address customerId: 42 changed } Ask for details Billing
AddressChanged { Event-carried State T ransfer customerId: 42, oldAddress: ... newAddress: ... } AddressChanged { Customer customerId: 42, address: ... } Address changed CustomerChanged { customerId: 42, status: A, Billing address: ..., } CustomerMoved { ..., }
What‘s This decision is complex Reverse direction direction of dependency of dependency Customer Customer Address Change changed Address Billing Billing
Example Change Address Address Incoming Email From bla@company.com Submit Date 2019-04-23 09.05 To confirm your address change please click on this link: http://company.com/confirm?id=82e97d49-166c-4862- 9973-4db348e6225d
Example direction of dependency Notification http://company.com/confirm?id=82e97 Change Adress d49-166c-4862-9973-4db348e6225d Address change requested Address change Customer confirmed
Example direction of dependency Notification http://company.com/confirm?id=82e97 Change Adress d49-166c-4862-9973-4db348e6225d Send mail ‚ Confirmation ‘ ‚ Confirmation ‘ Customer approved Address changed
Challenge: Command Command vs. Event vs Event
It is NOT about communication protocols Customer Customer It can be messaging, Address Change changed REST, whatever , …. Address Billing Billing
Manifold ways of transport …
Manifold ways of transport …
Message Event ? Record Query Command Event Intend, Fact, Want s.th. to happen, happened in the past, The intention itself is a fact immutable
Message Event ? Record Query Command Event
Commands in disguise Send Wording of Message recipient The Customer Needs T o Be Sent A Message T o Confirm Address Change Wording of Sender Event
Recommend
More recommend
