kafka streams
play

KAFKA STREAMS CLOUD MONITORING AWS CLOUD MONITORING AWS APP - PowerPoint PPT Presentation

Jun 25, 2023 •183 likes •1.43k views

FROM USE CASE TO PRODUCTION KAFKA STREAMS CLOUD MONITORING AWS CLOUD MONITORING AWS APP CLOUD MONITORING AWS HTTP APP CLOUD MONITORING AWS METRICS NRDB HTTP APP CLOUD MONITORING SCHDLR JOBS AWS NRDB METRICS HTTP WORKER CLOUD

  1. ENRICHMENT ▸ same partitioning key ▸ same number of partitions ▸ same hash function of producer

  2. ENRICHMENT METRICS PO JAVA METRICS P1 APP METRICS P2 METADATA PO GO METADATA P1 APP METADATA P2

  3. ENRICHMENT METRICS PO K:1, V:42 JAVA APP GO APP METADATA P2 K:1, V:eu

  4. ENRICHMENT METRICS PO K:1, V:42 JAVA APP METADATA KSTREAMS APP PO P0 GO APP METADATA P2 K:1, V:eu

  5. ENRICHMENT METRICS PO K:1, V:42 JAVA APP METADATA KSTREAMS APP PO P0 GO APP METADATA P2 K:1, V:eu

  6. ENRICHMENT METRICS MURMUR2 PO JAVA APP METADATA KSTREAMS APP PO P0 FNV-1A GO APP METADATA P2

  7. ENRICHMENT METRICS PO K:1, V:42 JAVA METRICS P1 APP METRICS P2 METADATA KSTREAMS APP REPARTITION PO P0’ METADATA’ METADATA PO P0’ K:1, V:eu GO METADATA’ METADATA P1 P1’ APP METADATA’ METADATA P2 P2’ K:1, V:eu

  8. ENRICHMENT METRICS PO K:1, V:42 JAVA METRICS P1 APP METRICS P2 METADATA KSTREAMS APP REPARTITION PO P0’ METADATA’ METADATA PO P0’ K:1, V:eu GO METADATA’ METADATA P1 P1’ APP METADATA’ METADATA P2 P2’ K:1, V:eu

  9. ENRICHMENT ▸ enrichment at scale thanks to co-partitioning ▸ in case of doubt — repartition ▸ a bit more kafka topics

  10. ENRICHMENT 🏞 VACATIONS?

  11. PRODUCTION READINESS 2 CPU 4096 MB APP

  12. PRODUCTION READINESS 2 CPU 3x 4096 MB APP APP APP

  13. PRODUCTION READINESS HEALTH-CHECK HEALTH-CHECK HEALTH-CHECK APP APP APP

  14. PRODUCTION READINESS HEALTH-CHECK HEALTH-CHECK HEALTH-CHECK APP APP APP

  15. PRODUCTION READINESS HEALTH-CHECK HEALTH-CHECK APP APP

  16. PRODUCTION READINESS HEALTH-CHECK HEALTH-CHECK HEALTH-CHECK APP APP APP

  17. PRODUCTION READINESS HEALTH-CHECK APP KSTREAMS + LIBRARY DSL

  18. PRODUCTION READINESS HEALTH-CHECK STREAM APP THREAD

  19. PRODUCTION READINESS HEALTH-CHECK STREAM APP THREAD

  20. PRODUCTION READINESS HEALTH-CHECK STREAM APP THREAD IT’S FINE

  21. PRODUCTION READINESS HEALTH-CHECK STREAM APP THREAD

  22. ENRICHMENT 👃 LISTENERS

  23. PRODUCTION READINESS RUNNING

  24. PRODUCTION READINESS PENDING NOT RUNNING ERROR SHUTDOWN RUNNING

  25. PRODUCTION READINESS RUNNING REBALANCING

  26. PRODUCTION READINESS RUNNING REBALANCING

  27. PRODUCTION READINESS RESTORE BATCH START RESTORED

  28. PRODUCTION READINESS RESTORE BATCH RESTORE START RESTORED END

  29. PRODUCTION READINESS RESTORE BATCH START RESTORED

  30. PRODUCTION READINESS ‣ consuming/producing throughput ▸ missing joins ▸ rebalancing loops ▸ storage-specific metrics ▸ business-specific metrics

  31. PRODUCTION READINESS

  32. PRODUCTION READINESS

  33. PRODUCTION READINESS

Recommend

Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai

Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai

Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai Waehner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.kai-waehner.de Confidential 1 Abstract Microservices establish many

977 views • 44 slides
Kafka Streams: Hands-on Session A.A. 2017/18 Matteo Nardelli Laurea Magistrale in Ingegneria

Kafka Streams: Hands-on Session A.A. 2017/18 Matteo Nardelli Laurea Magistrale in Ingegneria

Universit degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Kafka Streams: Hands-on Session A.A. 2017/18 Matteo Nardelli Laurea Magistrale in Ingegneria Informatica - II anno The reference

300 views • 13 slides
Fast Data apps with Alpakka Kafka connector and Akka Streams Sean Glover, Lightbend @seg1o Who

Fast Data apps with Alpakka Kafka connector and Akka Streams Sean Glover, Lightbend @seg1o Who

Fast Data apps with Alpakka Kafka connector and Akka Streams Sean Glover, Lightbend @seg1o Who am I? Im Sean Glover Principal Engineer at Lightbend Member of the Fast Data Platform team Organizer of Scala Toronto (scalator)

925 views • 69 slides
FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES Wojciech Rzsa,

FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES Wojciech Rzsa,

7/17/2019 From HTTP to Kafka-based microservices FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES Wojciech Rzsa, FLYR Poland @wrzasa localhost:4567/index.html?print-pdf#/ 1/83 From HTTP to Kafka-based

1.15k views • 83 slides
Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has gotten its mileage out of

Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has gotten its mileage out of

1 Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has gotten its mileage out of Zookeeper But it is still a second system KIP-500 has been adopted by the community This is not a 1-1 replacement Weve been

1.92k views • 124 slides
STREAM PROCESSING AT LINKEDIN: APACHE KAFKA & APACHE SAMZA Processing billions of events

STREAM PROCESSING AT LINKEDIN: APACHE KAFKA & APACHE SAMZA Processing billions of events

STREAM PROCESSING AT LINKEDIN: APACHE KAFKA & APACHE SAMZA Processing billions of events every day Neha Narkhede Co-founder and Head of Engineering @ Stealth Startup Prior to this Lead, Streams Infrastructure @ LinkedIn

494 views • 45 slides
Introduction to Kafka Instructor: Ekpe Okorafor 1. Big Data Academy - Accenture 2. Computer

Introduction to Kafka Instructor: Ekpe Okorafor 1. Big Data Academy - Accenture 2. Computer

Introduction to Kafka Instructor: Ekpe Okorafor 1. Big Data Academy - Accenture 2. Computer Science - African University of Science & Technology Agenda Introduction - Messaging Basics Kafka Architecture Kafka

555 views • 26 slides
Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu

Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu

Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu Clusters Adem Efe Gencer Senior Software Engineer LinkedIn Kafka: A Distributed Stream Processing Platform : High throughput & low latency :

989 views • 50 slides
Kafka in Jail Running Kafka in container orchestrated clusters Sean Glover, Lightbend @seg1o

Kafka in Jail Running Kafka in container orchestrated clusters Sean Glover, Lightbend @seg1o

Kafka in Jail Running Kafka in container orchestrated clusters Sean Glover, Lightbend @seg1o Who am I? Im Sean Glover Senior Software Engineer at Lightbend Member of the Fast Data Platform team Contributor to various projects in

788 views • 60 slides
CSE 143 Streams as C++ Classes Streams are C++ classes Streams have lots of built-in

CSE 143 Streams as C++ Classes Streams are C++ classes Streams have lots of built-in

CSE 143 Streams as C++ Classes Streams are C++ classes Streams have lots of built-in methods We use the . syntax to access member More Stream I/O functions, as usual. inFile.get(ch); // get a character outFile.put(ch); //

209 views • 3 slides
Day 3 Lab1: Spark Streaming with Kafka Example Introductions In this example, we will write a

Day 3 Lab1: Spark Streaming with Kafka Example Introductions In this example, we will write a

Day 3 Lab1: Spark Streaming with Kafka Example Introductions In this example, we will write a Spark Streaming program that consumes messages from Kafka. We will reuse the whole setup from the previous lab, so this lab is best done as a

233 views • 4 slides
High throughput High throughput kafka for science kafka for science Testing Kafkas limits

High throughput High throughput kafka for science kafka for science Testing Kafkas limits

High throughput High throughput kafka for science kafka for science Testing Kafkas limits for science J Wyngaard, PhD wyngaard@jpl.nasa.gov UTLINE O UTLINE O Streaming Science Data Benchmark Context Tests and Results

473 views • 30 slides
Day 4 Lab1: Docker container for Kafka - Spark streaming - Cassandra This Dockerfile sets up

Day 4 Lab1: Docker container for Kafka - Spark streaming - Cassandra This Dockerfile sets up

Day 4 Lab1: Docker container for Kafka - Spark streaming - Cassandra This Dockerfile sets up a complete streaming environment for experimenting with Kafka, Spark streaming (PySpark), and Cassandra. It installs Kafka 0.10.2.1 Spark 2.1.1

156 views • 4 slides
Apache Kafka Real-Time Data Pipelines http://kafka.apache.org/ Joe Stein Developer,

Apache Kafka Real-Time Data Pipelines http://kafka.apache.org/ Joe Stein Developer,

Apache Kafka Real-Time Data Pipelines http://kafka.apache.org/ Joe Stein Developer, Architect & Technologist Founder & Principal Consultant => Big Data Open Source Security LLC - http://stealth.ly Big Data Open Source

637 views • 40 slides
READING KAFKA IN QATAR Qatar-TESOL Conference, April 2011 Magdalena Rostron Academic Bridge

READING KAFKA IN QATAR Qatar-TESOL Conference, April 2011 Magdalena Rostron Academic Bridge

READING KAFKA IN QATAR Qatar-TESOL Conference, April 2011 Magdalena Rostron Academic Bridge Program Qatar Foundation, Doha, Qatar mrostron@qf.org.qa Introduction my story Reading Kafka In Qatar Reading Kafka in Qatar

562 views • 10 slides
Blockchain consensus Protocols in the Wild Tao Wang, Lihang Pan ECS 265 Apache Kafka

Blockchain consensus Protocols in the Wild Tao Wang, Lihang Pan ECS 265 Apache Kafka

Blockchain consensus Protocols in the Wild Tao Wang, Lihang Pan ECS 265 Apache Kafka (CFT) PBFT (BFT) Kafka Architecture ISR (in- sync) {b1,b2,b3} b1 b2 b3 Photo credit to Apache

2.57k views • 32 slides
INTEGRATING SYSTEMS INTEGRATING SYSTEMS IN THE AGE OF IN THE AGE OF QUARKUS, KNATIVE AND KAFKA

INTEGRATING SYSTEMS INTEGRATING SYSTEMS IN THE AGE OF IN THE AGE OF QUARKUS, KNATIVE AND KAFKA

Sponsorslides INTEGRATING SYSTEMS INTEGRATING SYSTEMS IN THE AGE OF IN THE AGE OF QUARKUS, KNATIVE AND KAFKA CONNECT QUARKUS, KNATIVE AND KAFKA CONNECT Peter Palaga @ppalaga The original reveal.js presentation this PDF was created from is

1.05k views • 93 slides
Cloud-Native and Scalable Kafka Allen Wang @allenxwang About Me Real Time Data

Cloud-Native and Scalable Kafka Allen Wang @allenxwang About Me Real Time Data

Cloud-Native and Scalable Kafka Allen Wang @allenxwang About Me Real Time Data Infrastructure @ Netflix Apache Kafka contributor (KIP-36 Rack Aware Assignment) NetflixOSS contributor (Archaius and Ribbon) Previously Cloud

722 views • 44 slides
Stream Bank Stabilization in Open Space Streams in open space There are approximately 35

Stream Bank Stabilization in Open Space Streams in open space There are approximately 35

Stream Bank Stabilization in Open Space Streams in open space There are approximately 35 miles of streams that flow through Open Space. The streams range in size from small head water streams to the Middle Patuxent River. Streams

349 views • 30 slides
Data Streams Many large sources of data are generated as streams of updates: IP Network

Data Streams Many large sources of data are generated as streams of updates: IP Network

Summarizing and Mining Skewed Data Streams Graham Cormode cormode@bell-labs.com Flip Korn, S. Muthukrishnan, Divesh Srivastava Data Streams Many large sources of data are generated as streams of updates: IP Network traffic data Text:

485 views • 44 slides
Data Streams Many large sources of data are generated as streams of updates: IP Network

Data Streams Many large sources of data are generated as streams of updates: IP Network

Summarizing and Mining Skewed Data Streams Graham Cormode cormode@bell-labs.com S. Muthukrishnan muthu@cs.rutgers.edu Data Streams Many large sources of data are generated as streams of updates: IP Network traffic data Text: email/

535 views • 20 slides
Streams and File I/O Fundamentals of Computer Science Outline Overview of Streams and File

Streams and File I/O Fundamentals of Computer Science Outline Overview of Streams and File

Streams and File I/O Fundamentals of Computer Science Outline Overview of Streams and File I/O Buffering Text File I/O Binary File I/O Streams Stream : an object that either delivers data to its destination (screen, file,

770 views • 33 slides
CS 61A Discussion 10 Streams (and Interpreters) Albert Xu Kaavya Shah Slides:

CS 61A Discussion 10 Streams (and Interpreters) Albert Xu Kaavya Shah Slides:

CS 61A Discussion 10 Streams (and Interpreters) Albert Xu Kaavya Shah Slides: albertxu.xyz/teaching/cs61a/ *part of these slides sourced from Jemmy and Is Final Review Fall 18 Streams Streams are lazy linked lists - the first element

334 views • 21 slides
BUBUKU kafka supervisor Dmitry Sorokin Zalando SE What is supervisor From wikipedia:

BUBUKU kafka supervisor Dmitry Sorokin Zalando SE What is supervisor From wikipedia:

BUBUKU kafka supervisor Dmitry Sorokin Zalando SE What is supervisor From wikipedia: blablabla controls the execution of other routines and regulates work scheduling, input/output operations, error actions, and similar functions and

133 views • 9 slides

More recommend