Concursus Event Sourcing Evolved GOTO London 2016 Introductions - PowerPoint PPT Presentation

Concursus Event Sourcing Evolved GOTO London 2016

Introductions Dominic Fox Tareq Abedrabbo Twitter: @dynamic_proxy Twitter: @tareq_abedrabbo Email: dominic.fox@opencredo.com Email: tareq.abedrabbo@opencredo.com Concursus Page: https://opencredo.com/publications/concursus/ Github: http://github.com/opencredo/concursus

Agenda • History • Concepts • Example • Domain model • Processing model • Programming model • Future directions

What is Concursus? A toolkit for processing and organising messy data in an distributed context .

The Concursus Timeline Observations Conception and design Prototype Open source implementation Technical report and blogs

Event Sourcing “Event Sourcing ensures that all changes to application state are stored as a sequence of events . Not just can we query these events , we can also use the event log to reconstruct past states , and as a foundation to automatically adjust the state to cope with retroactive changes.” http://martinfowler.com/eaaDev/EventSourcing.html

What is Concursus? Problems Concursus addresses: ü Processing events in a scalable and reliable way ü Processing guarantees and ordering: exactly once, out of order, repeated or missed delivery, etc.. ü Building meaningful domain models to reason about and build business logic around ü Flexibility: building additional views as needed

Why Concursus? Tendencies: • From internet of users to internet of things • From “presence” to “presents” • From monoliths to microservices

From Internet of Users to Internet of Things

From Presence to Presents

From Monoliths to Microservices

“Write First, Reason Later” 2016-10-12 2016-10-12 2016-10-12 09:06:31.432 09:06:32.106 09:06:34.740 Received at Depot Received at Depot Received at Depot 2016-10-12 2016-10-12 2016-10-12 11:35:02.163 11:40:21.032 11:38:51.204 Loaded onto Truck Loaded onto Truck Loaded onto Truck 2016-10-12 2016-10-12 2016-10-12 14:12:44.021 15:00:31.322 15:11:05.038 Delivery Failed Delivered Delivered

“Write First, Reason Later”

Handling Events ü Delivery constraints out of order, repeated, delayed or missed delivery ü Processing guarantees at least once or exactly once processing, idempotency ü Ordering partial ordering across aggregates (with reasonable assumptions)

Data Processing Layers ü Durable sufficiently durable buffer for async processing (what’s happening) ü Persistent a permanent record of everything that has happened (what happened) ü Transient fast and consistent, but also disposable state (what happens)

Building Blocks • Java 8 and Kotlin : APIs • Cassandra : Persistent state (Event store) • Kafka : Durable state (Message broker) • Hazelcast : Transient state (cache, idempotency filters) • Also, RabbitMQ and Redis

Sources of Inspiration Stream processing frameworks such as Apache Storm and Spark Google papers: Cloud dataflow, MillWheel Apache Spark papers The Axon CQRS framework Domain Driven Design Functional programming

Summary Concursus = Event sourcing + Stream processing + Bounded contexts (DDD) + Distributed computing

Received at Loaded onto Depot Truck Delivered Delivery Failed

Domain Model: Events Received Loaded Received Loaded Delivered Delivery at onto at onto Failed Depot Truck Depot Truck

Domain Model: Events aggregateType : parcel aggregateId : 69016fb5-1d69-4a34- 910b-f8ff5c702ad9 Received Loaded Received Loaded Delivered Delivery at onto at onto Failed Depot Truck Depot Truck eventTimestamp : 2016-03-31 10:31:17.981 parameters : { “depotId”: “Lewisham” }

Domain Model: Events aggregateType : parcel aggregateId : 69016fb5-1d69-4a34- 910b-f8ff5c702ad9 Received Loaded Received Loaded Delivered Delivery at onto at onto Failed Depot Truck Depot Truck eventTimestamp : 2016-03-38 08:15:23.104 parameters : { “truckId”: “J98 257” }

Domain Model: Events aggregateType : parcel aggregateId : 69016fb5-1d69-4a34- 910b-f8ff5c702ad9 eventTimestamp : 2016-03-31T10:36:42.171Z processingTimestamp : 2016-03-31T10:36:48.3904Z Delivered parameters : { “deliveryAddress”: “123 Sudbury Loaded onto Truck Avenue, Droitwich DR4 8PQ”} Received at Depot Delivery Failed Loaded onto Truck Received at Depot

Domain Model: Summary Every Event occurs to an Aggregate , identified by its type and id . Every Event has an eventTimestamp , generated by the source of the event. An Event History is a log of Events , ordered by eventTimestamp , with an additional processingTimestamp which records when the Event was captured.

Processing Model: Ordering Network Event sources Event processors Events arrive: • Partitioned • Interleaved • Out-of-order

Processing Model: Ordering Log is: • Partitioned by aggregate id • Ordered by event timestamp

Cassandra Schema CREATE TABLE IF NOT EXISTS concursus.Event ( aggregateType text, aggregateId text, eventTimestamp timestamp, streamId text, processingId timeuuid, name text, version text, parameters map<text, text>, characteristics int, PRIMARY KEY((aggregateType, aggregateId), eventTimestamp, streamId) ) WITH CLUSTERING ORDER BY (eventTimestamp DESC);

Cassandra & AMQP Publish events RabbitMQ Topic Downstream processing Log events Cassandra Event Store

Cassandra & AMQP out-of-order events RabbitMQ Topic Downstream processing ordered query results Cassandra Event Store

Cassandra & Kafka Publish events Kafka Topic Downstream processing Log events Cassandra Event Store Event store listener

Processing Model: Summary Events arrive partitioned, interleaved and out-of-order. Events are sorted into event histories by aggregate type and id. Events are sorted within event histories by event timestamp, not processing timestamp. Event consumers need to take into account the possibility that an event history may be incomplete at the time it is read – consider using a watermark to give incoming events time to “settle”.

Programming Model: Core Metaphor Received Loaded Received Loaded Delivered Delivery at onto at onto Failed Depot Truck Depot Truck

Programming Model: Core Metaphor Received Loaded Received Loaded Delivered Delivery at onto at onto Failed Depot Truck Depot Truck Consumer<Event>

Emitting Events You give me a Consumer<Event> , and I send Event s to it one at a time:

Handling Events I implement Consumer<Event> , and handle Event s that are sent to me.

Java 8 Mapping

Kotlin Mapping sealed class ParcelEvent { class ReceivedAtDepot( val depotId : String): ParcelEvent() class LoadedOntoTruck( val truckId : String): ParcelEvent() class Delivered( val destinationId : String): ParcelEvent() class DeliveryFailed(): ParcelEvent() }

Kotlin Mapping eventBus.dispatchTo(parcelId, ReceivedAtDepot(depotId = "Lewisham Depot" ) at start, LoadedOntoLorry(lorryId = "Truck CU50 ZCV" ) at start.plus(2, DAYS ) )

Kotlin Mapping fun describeEvent(event: ParcelEvent): Unit = when (event) { is ReceivedAtDepot -> println ( "Received at depot: ${ event. depotId}" ) is LoadedOntoTruck -> println ( "Loaded onto truck: ${ event. truckId}" ) is Delivered -> println ( "Delivered to: ${ event. destinationId}" ) is DeliveryFailed -> println ( "Delivery failed" ) }

Event-Handling Middleware Event-handling middleware is a chain of Consumer<Event> s that transforms, routes, persists and dispatches events. A single event submitted to this chain may be: ■ Checked against an idempotency filter (e.g. a Hazelcast distributed cache) ■ Serialised to JSON ■ Written to a message queue topic ■ Retrieved from the topic and deserialised ■ Persisted to an event store (e.g. Cassandra) ■ Published to an event handler which maintains a query-optimised view of part of the system ■ Published to an event handler which maintains an index of aggregates by event property values (e.g. lightbulbs by wattage)

Future Directions • Kafka Streams • Narrative threads across event histories • Generic Attribute indexing • State management and caching • Improved cloud tooling

Thank you for listening Any questions?

Three Processing Schedules 1.Transient 2.Durable 3.Persistent

Three Processing Schedules 1.Transient - what happens 2.Durable - what’s happening 3.Persistent - what happened

“Write First, Reason Later”