Microservices Lessons Learned From a Startup Perspective Susanne - PowerPoint PPT Presentation

Microservices Lessons Learned From a Startup Perspective Susanne Kaiser @suksr CTO at Just Software @JustSocialApps

Each journey is different “People try to copy Netflix, but they can only copy what they see. They copy the results, not the process.” Adrian Cockcroft, AWS VP Cloud Archtitect, former Netflix Chief Cloud Architect

Affecting Circumstances Team Legacy-System Strategy ● Size ● Maintenance effort ● New Features ● Skillset ● Environment ● Timeline/Milestones ● Structure

Background JUST PAGE JUST CONNECT Social Network Real-time collaboration JUST TASKS JUST DRIVE Task Management Document Sharing JUST PEOPLE User Management

Background At The Beginning … A Monolith In Every Aspect One One team Single Unit technology stack One collaboration product

Background After An Evolving Time ... New features Productivity suffered released slowly Usability and UX suffered

Background Separate Collaboration Apps JUST PAGE JUST CONNECT Social Network Real-time collaboration JUST TASKS JUST DRIVE Task Management Document Sharing JUST PEOPLE User Management

Background Small, Autonomous Teams With well-defined responsibilities JUST PAGE Social Network JUST PEOPLE User Management JUST CONNECT Real-time collaboration JUST DRIVE Document Sharing JUST TASKS Task Management

Background In The Long Run ... Product Organization Software architecture

Straightforward Process? Start End

No Straightforward Process! Start End Start Theory Reality

First Approach As Co-Existing Service Web App JUST CONNECT JUST DRIVE REST API JUST PAGE r Message Broker e k o r JUST DRIVE B r Adapter e e t g p a a s d s e A M DB Adapter JUST LIST Message DB Adapter Broker    

Lesson #1: Too Many Steps At Once Slow You Down New UI Maintain & run current system New data structure Timelines More features

Start With One Manageable Step At A Time Changing Easy to extract frequently Different resource requirements Split in steps, e.g. top/down

Lesson #2: Deferring Solving Authz Handling Hurts I have a new service I have a new service that needs authorization. Where is that needs authorization. Where is the authz service I could use? the authz service I could use? Not there, yet. Sorry! Not there, yet. Sorry! Ok, than I am putting my code Ok, than I am implementing authz to the place where authz handling in my local service. exists … to the monolith. Feeding the monolith Re-implementing authz w/ every service

Lesson #2: Deferring Solving Authz Handling Hurts Solve Authz Handling Early!

Lesson #3: Less Aligned Strategy Is Expensive Separate Apps Separate Teams Separate Services Bundled Deployment

Lesson #4: Data Related Overhead Keep in sync Setup Setup Maintain Setup Maintain

Lesson #4: Data Related Overhead Streams

How To Interact Between Services? Event-Driven Hybrid Request-Driven command/query command Event-Stream Event-Stream query consume consume produce produce

How To Manage Data? Remote query Hybrid Model getProfile(ProfileId) directly to source Document Service REST Profile Service Task Service Event stream purely for notifications

How To Manage Data? Event-Driven State Transfer Document Service Profile Local copy/cache Service Task ProfileUpdated Service Event

How To Manage Data? Source Of Truth Database Events “Traditional” Event-Driven System

How To Manage Data? Multiple Sources Of Truth Internal source of truth Database Events Dual writes External source of truth “Traditional” Event-Driven System

How To Manage Data? Multiple Sources Of Truth Single Source Of Truth Internal source of truth Database Derive state from events Event-Store Events Dual writes Events as first-class entities External source of truth Event Sourcing “Traditional” Event-Driven System

Apache Kafka Producer writes Immutable, ordered sequence of Topic records per partition 3 3 3 Each consumer controls 2 2 2 2 1 1 1 1 its position per partition 0 0 0 0 (offset) P0 P1 P2 P3 Each consumer subscribes reads to a topic Consumer Consumer Consumer Consumer Consumer Consumer Each partition is assigned to one consumer within Consumer Group Consumer Group a consumer group

Apache Kafka Scalable Producer Topic can be scaled out ● writes to several nodes Messages load-balanced Node/Broker ● Topic between consumer of one group Add partitions for more parallelism ● 3 3 3 Adding capacity with 0 downtime 2 2 2 2 ● 1 1 1 1 Fault-tolerant 0 0 0 0 P0 P1 P2 P3 Data stored to disk ● reads Replicated partitions ● Consumer conrols its offset ● Fast Consumer Consumer Consumer Consumer Consumer Consumer O(1) to append messages ● LinkedIn 2016: ● 1.4 trillion messages/day Consumer Group Consumer Group across over 1400 brokers

Apache Kafka Combines ... Messaging System Storage System Streaming Platform

Primary Use Cases Of Streaming Platforms Stream Processing Data Integration Source Sink

Kafka Streams Unbounded, ordered sequence Continuously Key/value-pair Topic updating

Kafka Streams Loads topic on startup Running in the Service Topic process of Microservice Streams can be: ● Joined ● Filtered State Store (disk backed) ● Grouped ● Aggregated Streams make data available wherever it’s needed ● etc.

Kafka Streams For Materialized Views Stream-Table-Join Stream Document Service Document Topic A P I Matererialized View Profile Topic as State Store Table for enrichment

Low Barrier To Entry For New Service ● No separate data storage to set up ● No extra local copies / caches to set up and to keep in sync ● No remote calls ● Materialized View always up to date ● Scalable, fault-tolerant, fast => reducing overhead, increasing performance & autonomy

If We Could Start The Journey Again ... ● Start with one manageable step at a time ● Take care of Authorization handling early ● Align strategy w/ Microservices goals ● Using Kafka Streams for Materialized Views

THANK YOU! Susanne Kaiser @suksr CTO at Just Software @JustSocialApps