Real Time Data Analytics @ Uber Ankur Bansal November 14, 2016

About Me Sr. Software Engineer, Streaming Team @ Uber ● Streaming team supports platform for real time data ○ analytics: Kafka, Samza, Flink, Pinot.. and plenty more ○ Focused on scaling Kafka at Uber’s pace Staff software Engineer @ Ebay ● Build & scale Ebay’s cloud using openstack ○ Apache Kylin: Committer, Emeritus PMC ●

Agenda Real time Use Cases ● Kafka Infrastructure Deep Dive ● Our own Development: ● ○ Rest Proxy & Clients Local Agent ○ uReplicator (Mirrormaker) ○ Chaperone (Auditing) ○ ● Operations/Tooling

Important Use Cases

Real-time Price Surging Stream Rider eyeballs KAFKA Processing SURGE MULTIPLIERS Open car information

Real-time Machine Learning - UberEats ETD

Fraud detection ● Share my ETA ● And many more ...

Apache Kafka is Uber’s Lifeline

DATA CONSUMERS Kafka ecosystem @ Uber Mobile App DATA Debugging PRODUCERS RIDER APP Real-time, Fast Analytics DRIVER APP Alerts, REAL-TIME PIPELINE Dashboards API / SERVICES DISPATCH (gps logs) Applications BATCH PIPELINE Data Science Mapping & Logistic Ad-hoc exploration Analytics Reporting

Kafka cluster stats 100s of billion Messages/day 100s TB bytes/day Multiple data centers

Kafka Infrastructure Deep Dive

Requirements Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● ● Low Latency for most use cases(<5ms ) Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● ● Reliable data replication across DC

Kafka Pipeline DataCenter-I Applications Kafka REST Regional [ProxyClient] Proxy Kafka DataCenter-III Aggregate uReplicator Kafka DataCenter-II Applications Kafka REST Regional [ProxyClient] Proxy Kafka Local Agent Secondary Kafka

Kafka Pipeline: Data Flow Aggregate Kafka Application Process Kafka Proxy Server Regional Kafka uReplicator 1 3 5 7 ProxyClient 8 4 6 2

Kafka Clusters DataCenter-I Applications Kafka REST Regional [ProxyClient] Proxy Kafka DataCenter-III Aggregate uReplicator Kafka DataCenter-II Applications Kafka REST Regional [ProxyClient] Proxy Kafka Local Agent Secondary Kafka

Kafka Clusters Use case based clusters ● Data (async, reliable) ○ Logging (High throughput) ○ Time Sensitive (Low Latency e.g. Surge, Push ○ notifications) High Value Data (At-least once, Sync e.g. Payments) ○ Secondary cluster as fallback ● Aggregate clusters for all data topics. ●

Kafka Clusters Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● ● Low Latency for most use cases(<5ms ) Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● ● Reliable data replication across DC

Kafka Rest Proxy DataCenter-I Applications Kafka REST Regional [ProxyClient] Proxy Kafka DataCenter-III Aggregate uReplicator Kafka DataCenter-II Applications Kafka REST Regional [ProxyClient] Proxy Kafka Local Agent Secondary Kafka

Why Kafka Rest Proxy ? Simplified Client API ● Multi-lang support (Java, NodeJs, Python, Golang) ● Decouple client from Kafka broker ● Thin clients = operational ease ○ Less connections to Kafka brokers ○ Future kafka upgrade ○ Enhanced Reliability ● Primary & Secondary Kafka Clusters ○

Kafka Rest Proxy: Internals

Kafka Rest Proxy: Internals Based on Confluent’s open sourced Rest Proxy ● Performance enhancements ● ○ Simple http servlets on jetty instead of Jersey Optimized for binary payloads. ○ Performance increase from 7K* to 45-50K QPS/box ○ Caching of topic metadata. ● ● Reliability improvements* Support for Fallback cluster ○ Support for multiple Producers (SLA based segregation) ○ Plan to contribute back to community ● *Based on benchmarking & analysis done in Jun ’2015

Rest Proxy: performance (1 box) End-end Latency (ms) Message rate (K/second) at single node

Kafka Clusters + Rest Proxy Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● ● Low Latency for most use cases(<5ms ) Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● ● Reliable data replication across DC

Kafka Clients DataCenter-I Applications Kafka REST Regional [ProxyClient] Proxy Kafka DataCenter-III Aggregate uReplicator Kafka DataCenter-II Applications Kafka REST Regional [ProxyClient] Proxy Kafka Local Agent Secondary Kafka

Client Libraries Support for multiple clusters. ● High Throughput ● ○ Non-blocking, async, batching <1ms produce latency for clients ○ Handles Throttling/BackOff signals from Rest Proxy ○ Topic Discovery ● ○ Discovers the kafka cluster a topic belongs Able to multiplex to different kafka clusters ○ Integration with Local Agent for critical data ●

Client Libraries What if there is network glitch / outage? Add Figure

Client Libraries Add Figure

Kafka Clusters + Rest Proxy + Clients Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● ● Low Latency for most use cases(<5ms ) Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● ● Reliable data replication across DC

Local Agent DataCenter-I Applications Kafka REST Regional [ProxyClient] Proxy Kafka DataCenter-III Aggregate uReplicator Kafka DataCenter-II Applications Kafka REST Regional [ProxyClient] Proxy Kafka Local Agent Secondary Kafka

Local Agent Local spooling in case of downstream outage/backpressure ● Backfills at the controlled rate to avoid hammering ● infrastructure recovering from outage Implementation: ● Reuses code from rest-proxy and kafka’s log module. ○ Appends all topics to same file for high throughput. ○

Local Agent Architecture Add Figure

Local Agent in Action Add Figure

Kafka Clusters + Rest Proxy + Clients + Local Agent Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● ● Low Latency for most use cases(<5ms ) Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● ● Reliable data replication across DC

uReplicator DataCenter-I Applications Kafka REST Regional [ProxyClient] Proxy Kafka DataCenter-III Aggregate uReplicator Kafka DataCenter-II Applications Kafka REST Regional [ProxyClient] Proxy Kafka Local Agent Secondary Kafka

Multi-DC data flow Traffic from DC2 Traffic from DC1 App box Dispatch http calls Mobile API Mirror Maker Kafka8 Aggregation Cluster Traffic from DC3

Mirrormaker : existing problems ● New Topic added ● New partitions added ● Mirrormaker bounced ● New mirrormaker added >> INSERT SCREENSHOT HERE << CONFIDENTIAL

uReplicator: In-house solution Helix MM Zookeeper Controller Helix Helix Helix Thread 1 Thread 1 Thread 1 Agent Agent Agent Thread N Thread N Thread N Topic-partition Topic-partition Topic-partition MM worker1 MM worker2 MM worker3

uReplicator Helix MM Zookeeper Controller Helix Helix Helix Thread 1 Thread 1 Thread 1 Agent Agent Agent Thread N Thread N Thread N Topic-partition Topic-partition Topic-partition MM worker1 MM worker2 MM worker3

Kafka Clusters + Rest Proxy + Clients + Local Agent Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● ● Low Latency for most use cases(<5ms ) Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● ● Reliable data replication across DC

uReplicator Running in production for 1+ year ● Open sourced: https://github.com/uber/uReplicator ● Blog: https://eng.uber.com/ureplicator/ ●

Chaperone - E2E Auditing

Chaperone Architecture

Chaperone : Track counts >> INSERT SCREENSHOT HERE << CONFIDENTIAL

Chaperone : Track Latency >> INSERT SCREENSHOT HERE << CONFIDENTIAL

Chaperone Running in production for 1+ year ● Planning to open source in ~2 Weeks ●

At-least Once Kafka

Why do we need it? Aggregate Kafka Application Process Kafka Proxy Server Regional Kafka uReplicator 1 3 5 7 ProxyClient 8 4 6 2 Most of infrastructure tuned for high throughput ● ○ Batching at each stage ○ Ack before produce (ack’ed != committed) ● Single node failure in any stage leads to data loss ● Need a reliable pipeline for High Value Data e.g. Payments

Real Time Data Analytics @ Uber Ankur Bansal November 14, 2016 - PowerPoint PPT Presentation

Real Time Data Analytics @ Uber Ankur Bansal November 14, 2016 About Me Sr. Software Engineer, Streaming Team @ Uber Streaming team supports platform for real time data analytics: Kafka, Samza, Flink, Pinot.. and plenty more

Scaling up Near real-time Analytics @ Uber and LinkedIn Who we are Chinmay Soman @ChinmaySoman

Real-Time in the Real World: Building a State of the Art Real-Time Analytics Platform INFORMS

GO BEYOND DATA Real-time Analytics for Application Performance Management Yury Oleynik Data

Time Predictions in Uber Eats Zi Wang@Uber QCon New York 2019 June 2019 Agenda 1. ML in Uber

Agenda 1. Capital One 2. Traditional Batch Analytics 3. The Great Paradigm Shift Real-Time

Real time Predictive Fraud Analytics using Databricks & Tableau Prasad Kona Partner Solution

Query Optimization Time: The New Bottleneck in Real-time Data Analytics IMDM 2015 Rajkumar Sen,

Data-Intensive Distributed Computing CS 451/651 (Fall 2018) Part 9: Real-Time Data Analytics (1/2)

HOW TO ACHIEVE REAL-TIME ANALYTICS ON A DATA LAKE USING GPUS Mark Brooks - Principal System

Scalable Data Analytics Pipeline for Real-Time Attack Detection; Design, Validation, and

An Open-Source Streaming Machine Learning and Real-Time Analytics Architecture Using an IoT

Replicating MySQL Data to TiDB For Near Real-Time Analytics Who Are We Jervin Real, Architect

Real-time stats for real-time problems Informing daily practice via predictive analytics

CIRUS : A Cloud Infrastructure for Real-time Ubilytics (aka ubiquitous big data analytics)

Design and Implementation of a Real-Time Cloud Analytics Platform OSCON Data 2011 David Pacheco

Predicting Real-Time Transaction Fraud Sami Niemi, PhD Barclays, Quantitative Analytics, Fraud

ARABLE Arable is a data analytics company that reduces risk throughout the agricultural supply

Quasi Real-time Data Analytics for Free Electron Lasers March 21 st 2018 OSG AHM Amedeo Perazzo

Scalable Data Analytics Pipeline for Validation of Real-Time Attack Detection Eric Badger , Phuong

Silicon Valley Big Data in Real Time: An Approach to Predictive Analytics for Alpha Generation

Automating Operational Decisions in Real-time Chris Sanden Senior Analytics Engineer About Me.

STREAM PROCESSING @ UBER DANNY YUAN @ UBER What is Uber Transportation at your fingertips

map-D map-D data refined map-D data refined map-D A GPU Database for Real-Time Big Data

Machine Learning Pipeline for Real-time Forecasting @Uber Marketplace Chong Sun, Danny Yuan