monolith to micro-services? event sourcing can help Doug
Client legacy Culture Amp (2012 - 2015) Rails App (Murmur) Query Server Read-write Read-only DB
Our journey Our meandering path to CQRS & event sourcing • Attempt 1 - Asynchronous micro-services Event Log the monolith and introduce query services • Attempt 2 - Greenfield systems of record Rebuild part of the monolith as a stand alone command side service • Attempt 3 - Refactor the monolith first And start with the least depended upon aggregates
Why micro-services? Attempt 1: Asynchronous • Too many developers! Micro-services • Monolith was becoming too complex. • Want teams to control their own assets. • Flexibility to introduce fit for purpose technologies. (for example, elastic search for our comments search) • Scale different parts of the app independently. (for example, capture has a very different profile)
Attempt 1: Why asynchronous? Asynchronous Micro-services • Resilience is architected into the platform. If a micro-service is down, all depended micro-services remain available and are eventually consistent. Also protects yourself from self inflicted DOS. • Ability to replay history for consuming services Much easier to refactor, or to introduce new services. • Decoupled architecture enabling platform flexibility Consumers don’t necessarily need to know where the producers are deployed
Client Reporting Service Rails App Event (Murmur) Log Reporting Query Server Service Read-write Read-only DB
• Plagued with bad data issues as we tried to convert What went our current production database to a stream of events wrong • Ongoing source-of-truth issues between the monolith persistence layer and the synthesised event log. • Events were generated at the wrong business points intended for consumption but ironically problematic for consumers • Events were too coarse-grained and slow to process. • Some of the events weren't real business events. • Multiple event streams - one for each event type - hurt.
Asynchronous micro-services are the What we right long term aim. learned But, don’t start with event consumers, start with event producers. Event logging is not event sourcing!
Add Add Remove Add What is Employee A Employee B Employee A Employee Event Sourcing? Add Add Employee B Employee • Store the business events as the source of truth • Can answer questions you don’t yet know you have. • Legal, accounting, traditional professions have used event sourcing for centuries.
“ Every method should either be a command that performs an action, or a query that returns data to the caller, but not both. In other words, asking a question should not change the answer. — Bertrand Meyer (designer of Eiffel language) on CQS
Command Query Responsibility Segregation What is CQRS? With CQRS Without CQRS EmployeeService EmployeeReadService + createEmployee(employee): void + getEmployee(id): Employee + getEmployee(id): Employee + getEmployees(account): + updateEmployeeName(id, name): void + makeEmployeeAdmin(id) : void + getEmployees(account): EmployeeWriteService + createEmployee(employee): void + updateEmployeeName(id, name): void + makeEmployeeAdmin(id) : void
Client CQRS & event sourcing architecture Command Query Handler Handler Aggregate Projection Eventually event consistent! store
Client Attempt 2: Greenfield systems of Rails App Event record (Murmur) Log Reporting Employee Query Server Service SoR Event Projector DB Store
• CQRS and event sourcing helped us be clear about the What worked boundaries of context. well? • The new service owns the validation, so bad data issues in accounts we migrated were found and fixed upfront • We could treat the monolith as one massive projection minimising work required on the Query side, and UI. • Removed our source of truth problems. Domain models can be deterministically built from events, not vice versa. • We could incrementally migrate accounts
• Required a leap of faith for the team to agree to try. What didn’t • Slow to develop as we were learning a new paradigm, work well? and rolling our own event sourcing system. • Eventual consistency hurts: • Upfront we needed to solve how to update the monolith in a reasonable timeframe. • We needed to refactor parts of the UI to cope with the eventual consistency. • Disaster recovery becomes a distributed problem.
“ With all the troubles of evented architecture - it is a blessing when "shit hits the fan”. I don't know if in the past we would even be able to tell the customers affected by the issue. — Unsolicited feedback from our Tech Lead for the Employee Service
1. Start with the Command Side. Treat your existing application like one big projection. Attempt 3: 2. Tackle a single Aggregate at a time. Refactor the No need for a big-bang migration. monolith first 3. Generate UUIDS in the legacy monolith first. These will help you link the migration scripts back. 4. Start by synchronously updating your existing application. Your legacy UI will thank you for it. 5. When all writes are through CQRS Commands, refactor the UI to use new dedicated asynchronous projections. 6. When nothing still uses the legacy domain model,
Demographi * Identifying c aggregates 1 Start Here! 1 * Account Employee Question * 1 1 * 1 1 * * Survey Response Comment 1 * 1 * Report ActionItem
Client Monolith Monolith Rails Controllers phase 1 Participant Comment Survey C Handler C Handler C Handler Legacy Domain Models Participant Comment Survey Aggregate Aggregate Aggregate Synchronous update! Legacy Event Projector DB Store
Client Client Client Monolith Authentication layer phase 2 Query Query Question Participant Survey C Handler C Handler C Handler handler handler Question Participant Survey Projection Projection Aggregate Aggregate Aggregate Asynchronous Projector Projector updates PostGres
Beyond the Employees Capture Survey Account monolith Service Service Designer Admin Message Bus Ratings Text Public Email Data Data Labs Analitycs Analytics APIs Sender Warehous Experiments
Key learnings 1. Event Logging is not event sourcing and just leads to source of truth confusion. 2. CQRS and event sourcing can help you move to micro-services, but start by refactoring the monolith 3. If you choose to event source your monolith, take one step at a time, and focus on the command side.
Questions?
Recommend
More recommend
