Building Scalable Real-Time Data Pipeline Data Fridge Vicente Valls - PowerPoint PPT Presentation

Sep 30, 2022 •13 likes •283 views

Building Scalable Real-Time Data Pipeline Data Fridge Vicente Valls Rios Software Engineering Manager !!! 2 million orders delivered in one day (July 2019) www.deliveryhero.com Data Volume in Numbers + User clicks + Logistics data

Building Scalable Real-Time Data Pipeline “Data Fridge” Vicente Valls Rios Software Engineering Manager
!!! 2 million orders delivered in one day (July 2019) www.deliveryhero.com
Data Volume in Numbers + User clicks + Logistics data 10M + Restaurant Order-Related availability Events Per Day + Menu items + Customer data
House of Brands & Global Services Logistics Search Recommendation ...
Challenges
Data Producers Data Consumers Search Recommendation Logistics ... ...
Data Producers Data Consumers Search Recommendation Data Fridge Logistics ... ...
Mission ● Unify the data structure across all entities ● Provide different data consumptions types: ○ Near real-time ○ Low-latency ○ High-latency (analytics) ● Become a data producer for ML applications
Architecture
Architecture Data Fridge Low-latency any API Data Consumer Ingestion Events any Streaming API subscription Data Consumer any Data Producer Long-term any storage Analytics
Ingestion API
Ingestion API ● Verifies quality of data using complex validations ● Single entrypoint AWS API Gateway ● Batch import ● AWS Lambda for event processing ● IP Whitelisting ● JWT authentication AWS Lambda
Canary Deployment v1 ● Custom solution Alias: PROD 90% ● 2 versions under the same alias 10% v2 ● Metrics monitoring
Streaming
Streaming AWS Kinesis Lambda AWS SNS any any Ingestion Data Data API Producer Consumer ● Kinesis stream preserves messages up to 7 days ● Ability to replay data ● Ability to scale up/down
SNS instead of Kinesis ? ● We do not care about order of events. ● Having Message Filtering. ● Kinesis requires to scale up shards as our Consumers grow. ● Scale up Kinesis is harder and more expensive than SNS. ● SNS->HTTP/SQS service provides PUSH-PULL data consumption.
Streaming ● 2 types for subscriptions ● Filtering based on event attributes HTTPS AWS PUSH SNS any Data Consumer AWS SQS PULL any Data Consumer
Stream Aggregation Order Event Order Event - AWS SNS AWS DynamoDB Order + Order Status Event AWS SNS AWS Lambda Order Status Event AWS SNS /SQS
Low-latency API
Low-latency API ● Dead Letter Queue AWS DynamoDB ● On-demand scaling AWS AWS API Lambda Gateway AWS SQS DLQ
Analytics
Analytics AWS Google ● Bigquery as OLAP db Lambda BigQuery ● Data quality visualization ● Bigquery is scalable DWH solution AWS SQS DLQ
Any Producer Ingestion API Streaming First use case: Second use case: Third use case: Near real-time Low-latency API Analytics Any Consumer
Tech Stack
Challenges
Challenges ● SLAs ( latency, durability, etc.) for some cloud services. ● Data Quality & Data documentation ● GDPR ● Automating new pipeline creation ● Automating SNS subscriptions / BigQuery Access
We Are Hiring! www.deliveryhero.com

Recommend

Scalable Data Analytics Pipeline for Real-Time Attack Detection; Design, Validation, and

Scalable Data Analytics Pipeline for Real-Time Attack Detection; Design, Validation, and Deployment in a Honeypot Environment Eric Badger Masters Student Computer Engineering 1 Overview Introduction/Motivation Challenges

508 views • 30 slides

Scalable Data Analytics Pipeline for Validation of Real-Time Attack Detection Eric Badger , Phuong

Scalable Data Analytics Pipeline for Validation of Real-Time Attack Detection Eric Badger , Phuong Cao, Alex Withers, Adam Slagell, Zbigniew Kalbarczyk, Ravi Iyer University of Illinois Urbana-Champaign 1 Overview Introduction/Motivation

283 views • 24 slides

Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned Yaroslav

Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned Yaroslav Tkachenko Senior Data Engineer at Activision 1+ Data lake size (AWS S3) PB Number of topics in the biggest cluster 500+ (Apache Kafka)

548 views • 44 slides

Fully Fault Tolerant Real Time Data Pipeline with Docker and Mesos Rahul Kumar Technical Lead

Fully Fault Tolerant Real Time Data Pipeline with Docker and Mesos Rahul Kumar Technical Lead LinuxCon / ContainerCon - Berlin, Germany Data Pipeline Agenda Mesos + Docker Reactive Data Pipeline Goal Analyzing data always

504 views • 37 slides

Deep Learning Frameworks with Spark and GPUs Abstract Spark is a powerful, scalable, real-time

Deep Learning Frameworks with Spark and GPUs Abstract Spark is a powerful, scalable, real-time data analytics engine that is fast becoming the de facto hub for data science and big data. However, in parallel, GPU clusters is fast becoming the

389 views • 26 slides

Building Multi-Model Big Data Platform for Real Estate Analytics Karthik Karuppaiya ApacheCon Big

Building Multi-Model Big Data Platform for Real Estate Analytics Karthik Karuppaiya ApacheCon Big Data - 05/18/2017 1 Agenda Company Overview 1 Overview of Ten-X Datasets 2 Ten-X Data Platform Architecture 3 Data-pipeline Implementation

652 views • 31 slides

THE DATA MINING PIPELINE What is data? The data mining pipeline: collection, preprocessing,

DATA MINING THE DATA MINING PIPELINE What is data? The data mining pipeline: collection, preprocessing, mining, and post-processing Sampling, feature extraction and normalization Exploratory analysis of data basic statistics What is data

2.11k views • 139 slides

Cloud-Native and Scalable Kafka Allen Wang @allenxwang About Me Real Time Data

Cloud-Native and Scalable Kafka Allen Wang @allenxwang About Me Real Time Data Infrastructure @ Netflix Apache Kafka contributor (KIP-36 Rack Aware Assignment) NetflixOSS contributor (Archaius and Ribbon) Previously Cloud

722 views • 44 slides

Real Data Real Issues what they never told me at Uni Phil Brierley WOMBAT 18 th February

Real Data Real Issues what they never told me at Uni Phil Brierley WOMBAT 18 th February 2016 Data Prep v Predictive Modelling Time wise 80% - Data Prep 5% - Model Building 50% - Explaining results Data Scientists like to over

910 views • 44 slides

Real-time PCR Data Markup Language A new standard for archiving and exchanging real-time PCR data

Real-time PCR Data Markup Language A new standard for archiving and exchanging real-time PCR data Steve Lefever On behalf of the RDML consortium qPCR = golden standard for nucleic acids quantification Dedicated and precise instruments needed

780 views • 21 slides

Modelling of the London Bus Network The Data London Open Transport API Real-time bus

Alex Gubbay Real Time Graph Modelling of the London Bus Network The Data London Open Transport API Real-time bus arrivals at every stop 130,000 data points a minute Also provided: Real-time traffic incidents Real-time

173 views • 5 slides

Big Data in Real-Time at Twitter by Nick Kallen (@nk) What is Real-Time Data? On-line

Big Data in Real-Time at Twitter by Nick Kallen (@nk) What is Real-Time Data? On-line queries for a single web request Off-line computations with very low latency Latency and throughput are equally important Not talking about

710 views • 46 slides

DATA MINING LECTURE 2 What is data? The data mining pipeline What is Data Mining? Data

DATA MINING LECTURE 2 What is data? The data mining pipeline What is Data Mining? Data mining is the use of efficient techniques for the analysis of very large collections of data and the extraction of useful and possibly unexpected

2.4k views • 94 slides

Is this normal? Finding anomalies in real-time data. Who am I? Im Theo (@postwait on Twitter)

Is this normal? Finding anomalies in real-time data. Who am I? Im Theo (@postwait on Twitter) I write a lot of code 50+ open source projects several commercial code bases I wrote Scalable Internet Architectures I sit on the ACM

327 views • 18 slides

Real time and comunications 1 Examples of use and data capture Network time Basic

Real time and comunications 1 Examples of use and data capture Network time Basic protocol for real-time traffic Protocol for the management of data flow Secure version of the protocol 2 What is real-time (time of

629 views • 33 slides

Lessons Learned Building and Operating a Serverless Data Pipeline Will Norman Introduction

Lessons Learned Building and Operating a Serverless Data Pipeline Will Norman Introduction Will Norman - Director of Engineering @ Intent FinTech and AdTech background Intent Data Science company for commerce sites Primary

703 views • 30 slides

What is Scalable Data Processing? S CALABLE DATA P ROCES S IN G IN R Michael J. Kane and Simon

What is Scalable Data Processing? S CALABLE DATA P ROCES S IN G IN R Michael J. Kane and Simon Urbanek Instructors, DataCamp In this course .. Work with data that is too large for your computer Write Scalable code Import and process data in

593 views • 34 slides

Chapter 5 Generating Linked Data in Real- time from Sensor Data Streams NIKOLAOS KONSTANTINOU

Chapter 5 Generating Linked Data in Real- time from Sensor Data Streams NIKOLAOS KONSTANTINOU DIMITRIOS-EMMANUEL SPANOS Materializing the Web of Linked Data Outline Introduction Fusion The Data layer Rule-based Reasoning Complete Example

1.02k views • 83 slides

Stream Algorithmics Albert Bifet March 2012 Data Streams Big Data & Real Time Data Streams

Stream Algorithmics Albert Bifet March 2012 Data Streams Big Data & Real Time Data Streams Data Streams Sequence is potentially infinite High amount of data: sublinear space High speed of arrival: sublinear time per example

969 views • 50 slides

Real-time Data Pipelines with Structured Streaming in Tathagata TD Das @tathadas

Real-time Data Pipelines with Structured Streaming in Tathagata TD Das @tathadas DataEngConf 2018 18 th April, San Francisco About Me Started Spark Streaming project in AMPLab, UC Berkeley Currently focused on building Structured

1.45k views • 51 slides

Building a scalable time-series database using Postgres Mike Freedman Co-founder / CTO,

Building a scalable time-series database using Postgres Mike Freedman Co-founder / CTO, Timescale mike@timescale.com https://github.com/timescale/timescaledb Time-series data is everywhere, greater volumes than ever before What DB for

689 views • 45 slides

Scalable Machine Learning 3. Data Streams Alex Smola Yahoo! Research and ANU

Scalable Machine Learning 3. Data Streams Alex Smola Yahoo! Research and ANU http://alex.smola.org/teaching/berkeley2012 Stat 260 SP 12 3. Data Streams Building realtime *Analytics at home Data Streams Data & Applications Moments

1.1k views • 92 slides

Using real-time big data for TVET policies and strategies: Case of Myanmar Hiromichi Katayama

Workshop: Can we use Big Data for Skills Anticipation and Matching? September 20, 2019 Session: Usage of big data for emerging and developing economies Using real-time big data for TVET policies and strategies: Case of Myanmar Hiromichi

353 views • 23 slides

Hitachi NEXT 2018 Building a Data Pipeline With Pentaho From Ingest to Analytics Contents

Hitachi NEXT 2018 Building a Data Pipeline With Pentaho From Ingest to Analytics Contents Page 2: Guided Demonstration: Data Source to Dashboard Page 3: Review the InputData Transformation Page 11: Review and Run the CT2000 Job Page 14:

296 views • 18 slides