from http to kafka based from http to kafka based
play

FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES - PowerPoint PPT Presentation

Sep 18, 2022 •304 likes •1.15k views

7/17/2019 From HTTP to Kafka-based microservices FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES Wojciech Rzsa, FLYR Poland @wrzasa localhost:4567/index.html?print-pdf#/ 1/83 From HTTP to Kafka-based

  1. 7/17/2019 From HTTP to Kafka-based microservices FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES Wojciech Rząsa, FLYR Poland @wrzasa localhost:4567/index.html?print-pdf#/ 1/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  2. 7/17/2019 From HTTP to Kafka-based microservices ABOUT ME ABOUT ME Informatics specialist by passion and by profession 15 years of academic work PhD but primarily an engineer FLYR Inc. http://flyrlabs.com Distributed systems Rzeszow Ruby User Group http://rrug.pl localhost:4567/index.html?print-pdf#/ 2/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  3. 7/17/2019 From HTTP to Kafka-based microservices FLYR FLYR Revenue management system for airlines Offices in San Francisco, USA (PST) Kraków, Poland (CEST) Machine Learning Microservices Python GCloud Kubernetes localhost:4567/index.html?print-pdf#/ 3/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  4. 7/17/2019 From HTTP to Kafka-based microservices FLYR DEVS ON PYCON CZ FLYR DEVS ON PYCON CZ localhost:4567/index.html?print-pdf#/ 4/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  5. 7/17/2019 From HTTP to Kafka-based microservices IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP localhost:4567/index.html?print-pdf#/ 5/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  6. 7/17/2019 From HTTP to Kafka-based microservices IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case localhost:4567/index.html?print-pdf#/ 6/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  7. 7/17/2019 From HTTP to Kafka-based microservices FAN-OUT REQUESTS FAN-OUT REQUESTS localhost:4567/index.html?print-pdf#/ 7/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  8. 7/17/2019 From HTTP to Kafka-based microservices PARTIAL RESPONSES PARTIAL RESPONSES localhost:4567/index.html?print-pdf#/ 8/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  9. 7/17/2019 From HTTP to Kafka-based microservices PARTIAL RESPONSES PARTIAL RESPONSES localhost:4567/index.html?print-pdf#/ 9/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  10. 7/17/2019 From HTTP to Kafka-based microservices IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case partial responses localhost:4567/index.html?print-pdf#/ 10/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  11. 7/17/2019 From HTTP to Kafka-based microservices IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case partial responses performance localhost:4567/index.html?print-pdf#/ 11/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  12. 7/17/2019 From HTTP to Kafka-based microservices OK, LET'S SWITCH FROM HTTP OK, LET'S SWITCH FROM HTTP TO... A... MQ? TO... A... MQ? localhost:4567/index.html?print-pdf#/ 12/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  13. 7/17/2019 From HTTP to Kafka-based microservices BUT... BUT... localhost:4567/index.html?print-pdf#/ 13/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  14. 7/17/2019 From HTTP to Kafka-based microservices BUT... BUT... We have HTTP-based infrastructure localhost:4567/index.html?print-pdf#/ 14/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  15. 7/17/2019 From HTTP to Kafka-based microservices BUT... BUT... We have HTTP-based infrastructure We have HTTP developers experience and habits localhost:4567/index.html?print-pdf#/ 15/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  16. 7/17/2019 From HTTP to Kafka-based microservices BUT... BUT... We have HTTP-based infrastructure We have HTTP developers experience and habits We lack experience with MQ-based IPC localhost:4567/index.html?print-pdf#/ 16/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  17. 7/17/2019 From HTTP to Kafka-based microservices BUT... BUT... We have HTTP-based infrastructure We have HTTP developers experience and habits We lack experience with MQ-based IPC We need to do it well ;-) localhost:4567/index.html?print-pdf#/ 17/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  18. 7/17/2019 From HTTP to Kafka-based microservices WITH MQ WE GET WITH MQ WE GET Flexibility Reliability Scalability Robustness localhost:4567/index.html?print-pdf#/ 18/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  19. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... localhost:4567/index.html?print-pdf#/ 19/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  20. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES localhost:4567/index.html?print-pdf#/ 20/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  21. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues localhost:4567/index.html?print-pdf#/ 21/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  22. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions localhost:4567/index.html?print-pdf#/ 22/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  23. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice localhost:4567/index.html?print-pdf#/ 23/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  24. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker localhost:4567/index.html?print-pdf#/ 24/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  25. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver localhost:4567/index.html?print-pdf#/ 25/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  26. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker localhost:4567/index.html?print-pdf#/ 26/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  27. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker . . . localhost:4567/index.html?print-pdf#/ 27/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  28. 7/17/2019 From HTTP to Kafka-based microservices LET'S CONTAIN THE RISKS IN LET'S CONTAIN THE RISKS IN ONE PLACE ONE PLACE localhost:4567/index.html?print-pdf#/ 28/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  29. 7/17/2019 From HTTP to Kafka-based microservices LET'S CONTAIN THE RISKS IN LET'S CONTAIN THE RISKS IN ONE PLACE ONE PLACE (A LIBRARY) (A LIBRARY) localhost:4567/index.html?print-pdf#/ 29/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  30. 7/17/2019 From HTTP to Kafka-based microservices AND CALL THIS PLACE AND CALL THIS PLACE async_calls async_calls localhost:4567/index.html?print-pdf#/ 30/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  31. 7/17/2019 From HTTP to Kafka-based microservices AND CALL THIS PLACE AND CALL THIS PLACE async_calls async_calls (FOR THE LACK OF BETTER CONCEPT) (FOR THE LACK OF BETTER CONCEPT) localhost:4567/index.html?print-pdf#/ 31/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

Recommend

Apache Kafka Real-Time Data Pipelines http://kafka.apache.org/ Joe Stein Developer,

Apache Kafka Real-Time Data Pipelines http://kafka.apache.org/ Joe Stein Developer,

Apache Kafka Real-Time Data Pipelines http://kafka.apache.org/ Joe Stein Developer, Architect & Technologist Founder & Principal Consultant => Big Data Open Source Security LLC - http://stealth.ly Big Data Open Source

637 views • 40 slides
Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has gotten its mileage out of

Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has gotten its mileage out of

1 Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has gotten its mileage out of Zookeeper But it is still a second system KIP-500 has been adopted by the community This is not a 1-1 replacement Weve been

1.92k views • 124 slides
KAFKA STREAMS CLOUD MONITORING AWS CLOUD MONITORING AWS APP CLOUD MONITORING AWS HTTP APP

KAFKA STREAMS CLOUD MONITORING AWS CLOUD MONITORING AWS APP CLOUD MONITORING AWS HTTP APP

FROM USE CASE TO PRODUCTION KAFKA STREAMS CLOUD MONITORING AWS CLOUD MONITORING AWS APP CLOUD MONITORING AWS HTTP APP CLOUD MONITORING AWS METRICS NRDB HTTP APP CLOUD MONITORING SCHDLR JOBS AWS NRDB METRICS HTTP WORKER CLOUD

1.43k views • 124 slides
Introduction to Kafka Instructor: Ekpe Okorafor 1. Big Data Academy - Accenture 2. Computer

Introduction to Kafka Instructor: Ekpe Okorafor 1. Big Data Academy - Accenture 2. Computer

Introduction to Kafka Instructor: Ekpe Okorafor 1. Big Data Academy - Accenture 2. Computer Science - African University of Science & Technology Agenda Introduction - Messaging Basics Kafka Architecture Kafka

555 views • 26 slides
Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu

Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu

Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu Clusters Adem Efe Gencer Senior Software Engineer LinkedIn Kafka: A Distributed Stream Processing Platform : High throughput & low latency :

989 views • 50 slides
Kafka in Jail Running Kafka in container orchestrated clusters Sean Glover, Lightbend @seg1o

Kafka in Jail Running Kafka in container orchestrated clusters Sean Glover, Lightbend @seg1o

Kafka in Jail Running Kafka in container orchestrated clusters Sean Glover, Lightbend @seg1o Who am I? Im Sean Glover Senior Software Engineer at Lightbend Member of the Fast Data Platform team Contributor to various projects in

788 views • 60 slides
Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai

Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai

Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai Waehner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.kai-waehner.de Confidential 1 Abstract Microservices establish many

977 views • 44 slides
Day 3 Lab1: Spark Streaming with Kafka Example Introductions In this example, we will write a

Day 3 Lab1: Spark Streaming with Kafka Example Introductions In this example, we will write a

Day 3 Lab1: Spark Streaming with Kafka Example Introductions In this example, we will write a Spark Streaming program that consumes messages from Kafka. We will reuse the whole setup from the previous lab, so this lab is best done as a

233 views • 4 slides
High throughput High throughput kafka for science kafka for science Testing Kafkas limits

High throughput High throughput kafka for science kafka for science Testing Kafkas limits

High throughput High throughput kafka for science kafka for science Testing Kafkas limits for science J Wyngaard, PhD wyngaard@jpl.nasa.gov UTLINE O UTLINE O Streaming Science Data Benchmark Context Tests and Results

473 views • 30 slides
Day 4 Lab1: Docker container for Kafka - Spark streaming - Cassandra This Dockerfile sets up

Day 4 Lab1: Docker container for Kafka - Spark streaming - Cassandra This Dockerfile sets up

Day 4 Lab1: Docker container for Kafka - Spark streaming - Cassandra This Dockerfile sets up a complete streaming environment for experimenting with Kafka, Spark streaming (PySpark), and Cassandra. It installs Kafka 0.10.2.1 Spark 2.1.1

156 views • 4 slides
READING KAFKA IN QATAR Qatar-TESOL Conference, April 2011 Magdalena Rostron Academic Bridge

READING KAFKA IN QATAR Qatar-TESOL Conference, April 2011 Magdalena Rostron Academic Bridge

READING KAFKA IN QATAR Qatar-TESOL Conference, April 2011 Magdalena Rostron Academic Bridge Program Qatar Foundation, Doha, Qatar mrostron@qf.org.qa Introduction my story Reading Kafka In Qatar Reading Kafka in Qatar

562 views • 10 slides
Blockchain consensus Protocols in the Wild Tao Wang, Lihang Pan ECS 265 Apache Kafka

Blockchain consensus Protocols in the Wild Tao Wang, Lihang Pan ECS 265 Apache Kafka

Blockchain consensus Protocols in the Wild Tao Wang, Lihang Pan ECS 265 Apache Kafka (CFT) PBFT (BFT) Kafka Architecture ISR (in- sync) {b1,b2,b3} b1 b2 b3 Photo credit to Apache

2.57k views • 32 slides
INTEGRATING SYSTEMS INTEGRATING SYSTEMS IN THE AGE OF IN THE AGE OF QUARKUS, KNATIVE AND KAFKA

INTEGRATING SYSTEMS INTEGRATING SYSTEMS IN THE AGE OF IN THE AGE OF QUARKUS, KNATIVE AND KAFKA

Sponsorslides INTEGRATING SYSTEMS INTEGRATING SYSTEMS IN THE AGE OF IN THE AGE OF QUARKUS, KNATIVE AND KAFKA CONNECT QUARKUS, KNATIVE AND KAFKA CONNECT Peter Palaga @ppalaga The original reveal.js presentation this PDF was created from is

1.05k views • 93 slides
Cloud-Native and Scalable Kafka Allen Wang @allenxwang About Me Real Time Data

Cloud-Native and Scalable Kafka Allen Wang @allenxwang About Me Real Time Data

Cloud-Native and Scalable Kafka Allen Wang @allenxwang About Me Real Time Data Infrastructure @ Netflix Apache Kafka contributor (KIP-36 Rack Aware Assignment) NetflixOSS contributor (Archaius and Ribbon) Previously Cloud

722 views • 44 slides
BUBUKU kafka supervisor Dmitry Sorokin Zalando SE What is supervisor From wikipedia:

BUBUKU kafka supervisor Dmitry Sorokin Zalando SE What is supervisor From wikipedia:

BUBUKU kafka supervisor Dmitry Sorokin Zalando SE What is supervisor From wikipedia: blablabla controls the execution of other routines and regulates work scheduling, input/output operations, error actions, and similar functions and

133 views • 9 slides
PG Change Data Capture with Debezium Hannu Valtonen Kafka Meetup | PG CDC with Debezium |

PG Change Data Capture with Debezium Hannu Valtonen Kafka Meetup | PG CDC with Debezium |

PG Change Data Capture with Debezium Hannu Valtonen Kafka Meetup | PG CDC with Debezium | 2018-11-04 Me Aiven co-founder Maintainer of Open Source projects: PGHoard, pglookout and pgmemcache PostgreSQL user for last 18 years

894 views • 27 slides
The Best of Apache Kafka Architecture Ranganathan Balashanmugam @ran_than Hell Budapest

The Best of Apache Kafka Architecture Ranganathan Balashanmugam @ran_than Hell Budapest

Apache: Big Data 2015 The Best of Apache Kafka Architecture Ranganathan Balashanmugam @ran_than Hell Budapest About Me Graduated as Civil Engineer. <dev> 10+ years </dev> <Thoughtworker from=India/>

515 views • 33 slides
VLTAVA VLTAVA Franz Kafka "Oh, God", he thought, "what a what I'll do. That's

VLTAVA VLTAVA Franz Kafka "Oh, God", he thought, "what a what I'll do. That's

VLTAVA VLTAVA Franz Kafka "Oh, God", he thought, "what a what I'll do. That's when I'll make the Metamorphosis strenuous career it is that I've chosen! big change. First of all though, I've got Travelling day in and day out.

257 views • 11 slides
How-to for real-time streaming and analytics at scale with Apache Kafka and Apache Ignite Viktor

How-to for real-time streaming and analytics at scale with Apache Kafka and Apache Ignite Viktor

How-to for real-time streaming and analytics at scale with Apache Kafka and Apache Ignite Viktor Gamov, Confluent, @gamussa Denis Magda, GridGain, @denismagda Digital Transformations Challenges Application Layer 10-100x more queries and

347 views • 18 slides
Apache Spark Streaming, Kafka and HarmonicIO: A Performance Benchmark and Architecture Comparison

Apache Spark Streaming, Kafka and HarmonicIO: A Performance Benchmark and Architecture Comparison

Apache Spark Streaming, Kafka and HarmonicIO: A Performance Benchmark and Architecture Comparison for Enterprise and Scientific Computing Ben Blamey, Andreas Hellander and Salman Toor Uppsala University, Sweden Ben.Blamey@it.uu.se Bench19,

474 views • 20 slides
VoIP Technologie, Implementationsbeispiel und wirtschaftliche Aspekte Karl-Heinz Kafka Senior

VoIP Technologie, Implementationsbeispiel und wirtschaftliche Aspekte Karl-Heinz Kafka Senior

4/9/2003 VoIP Technologie, Implementationsbeispiel und wirtschaftliche Aspekte Karl-Heinz Kafka Senior Consultant HP Services Network Solutions Group (NSG) 9. April 2003 Agenda Die Voraussetzungen der Implementierung einer VoIP

434 views • 9 slides
STREAM PROCESSING AT LINKEDIN: APACHE KAFKA & APACHE SAMZA Processing billions of events

STREAM PROCESSING AT LINKEDIN: APACHE KAFKA & APACHE SAMZA Processing billions of events

STREAM PROCESSING AT LINKEDIN: APACHE KAFKA & APACHE SAMZA Processing billions of events every day Neha Narkhede Co-founder and Head of Engineering @ Stealth Startup Prior to this Lead, Streams Infrastructure @ LinkedIn

494 views • 45 slides
Running Kafka on Kubernetes with Strimzi Sean Glover, Lightbend @seg1o Who am I? Im Sean

Running Kafka on Kubernetes with Strimzi Sean Glover, Lightbend @seg1o Who am I? Im Sean

Running Kafka on Kubernetes with Strimzi Sean Glover, Lightbend @seg1o Who am I? Im Sean Glover Principal Engineer at Lightbend Member of the Lightbend Pipelines team Organizer of Scala Toronto (scalator) Author and

720 views • 44 slides
Internet and Web Based Technology http://144.16.192.60/~isg/IWT/ About the Course I will be

Internet and Web Based Technology http://144.16.192.60/~isg/IWT/ About the Course I will be

Internet and Web Based Technology http://144.16.192.60/~isg/IWT/ About the Course I will be covering half the course (2 hours / week) Tuesday 9:30 AM 11:25 AM Topics to be covered How Internet works, HTML, HTTP, CGI scripts,

1.74k views • 142 slides

More recommend