UBER RUSH AND REBUILDING UBERS DISPATCHING PLATFORM motivation - PowerPoint PPT Presentation

UBER RUSH AND REBUILDING UBER’S DISPATCHING PLATFORM

motivation CHAPTER 1 OF 8

MOTIVATION TOWARDS MICROSERVICES STUNTS AND EXPERIMENTS

UBER RUSH, DELIVERY SERVICE MOTIVATION TOWARDS MICROSERVICES

UBER RUSH REQUIREMENTS MOTIVATION TOWARDS MICROSERVICES MULTI-PICKUP BULK DELIVERIES MULTI-DISPATCH SOPHISTICATED MATCHING CAPACITY MANAGEMENT

UBER EATS REQUIREMENTS MOTIVATION TOWARDS MICROSERVICES NO PICKUP LOCATION TEMPERATURE REGULATION INVENTORY MANAGEMENT RE-SUPPLY STATIONS CHECKOUT FLOW

evolution CHAPTER 2 OF 8

MONOSERVICE TO MICROSERVICES MONOLITHIC ARCHITECTURE USER CONFIG CARS CACHE POST- DISPATCH PROCESSOR MONOSERVICE MONOSERVICE GEO- ETA SURGE CODE

1 ST GENERATION MICROSERVICES MONOSERVICE TO MICROSERVICES LOGISTICS FOUNDATIONAL PLATFORM DEPENDENCIES SERVICES SERVICES SERVICES SUPPLY GOLDETA ONEDIRECTION FLIPR DEMAND INV. MGMT ARBITER GEOCOD GEOSPATIAL RAMEN DISPATCH OPTIC CEREBRO MONOSERVICE SYNC LOCKET

2 ND GENERATION MICROSERVICES MONOSERVICE TO MICROSERVICES LOGISTICS FOUNDATIONAL PLATFORM DEPENDENCIES SERVICES SERVICES SERVICES SUPPLY GOLDETA ONEDIRECTION FLIPR DEMAND INV. MGMT ARBITER GEOCOD GEOSPATIAL RAMEN OPTIC CEREBRO DISCO SYNC LOCKET GEOFENCE DISPATCH RTTR SCAVENGER UDESTROY LUMBERGH

A MICROSERVICE GATEWAY MONOSERVICE TO MICROSERVICES LOGISTICS FOUNDATIONAL PLATFORM DEPENDENCIES SERVICES SERVICES SERVICES SUPPLY GOLDETA ONEDIRECTION FLIPR DEMAND INV. MGMT ARBITER GEOCOD GEOSPATIAL RAMEN OPTIC CEREBRO DISPATCH GATEWAY DISCO SYNC LOCKET GEOFENCE RTTR SCAVENGER UDESTROY LUMBERGH

MOTIVATION TOWARDS MICROSERVICES THE TRADE-OFFS UPGRADES ARE PAINFUL MONOSERVICE vs. TEST SUITE IS SLOW MICROSERVICE FAILURE IS CATASTROPHIC CODE IS BRITTLE DEPLOYS ARE SLOW

topologies CHAPTER 3 OF 8

MICROSERVICE LAYOUT INDEPENDENT, INDIVIDUALLY ADDRESSABLE SERVERS HOST MICROSERVICE LOAD AVERAGE SERVICE DEMAND SERVICE WORKERS

ARRANGEMENT OF MICROSERVICES MULTI-TENANT OR DEDICATED HOSTS? DEMAND SUPPLY OR DISCO OPTIC DEDICATED DEMAND MULTI-TENANT HOSTS HOST

communications and fault tolerance CHAPTER 4 OF 8

MANAGING MICROSERVICE DEPENDENCIES AUTO-GENERATED CLIENTS JSON OVER HTTP DISCO MICROSERVICE DEMAND MICROSERVICE LUMBERGH THRIFT OVER HTTP

LOAD-BALANCING MICROSERVICES WITH CLIENT-SIDE LOAD-BALANCING HOT HOST FILE RELOADING B HEALTH CHECKING A PERSISTENT CONNECTIONS B CONNECTION POOLING B RETRIES A TALKS DIRECTLY TO B

COOPERATIVE MICROSERVICE INSTANCES FROM INDEPENDENT WORKERS TO COOPERATIVE 2 32 INDEPENDENT DEMAND HOSTS COOPERATIVE AND WORKERS DEMAND WORKERS ACROSS MANY HOSTS GOSSIP WITH ONE ANOTHER AND MAINTAIN A HASH RING OF EACH WORKER

COOPERATIVE MICROSERVICE INSTANCES WITH RINGPOP @ GITHUB.COM/UBER/RINGPOP HASH WORKER ADDRESSES 2 32 > hash('10.31.1.2:9000') 53554892 > hash('10.31.8.9:9000') 1325776234 HASH APPLICATION IDS > hash('33e2dc8c-16fd-4a19-9fad-4ebfc76c66c9') 2312992577 EACH DEMAND WORKER > hash('8828169c-69c5-4b79-ae5e-6204c5f615ff') OWNS A PORTION OF THE 2640491360 KEYSPACE

RELIABLE BACKGROUND OPERATIONS WITH HASH RING TECHNOLOGY 2 10 2 32 VNODE ENTITY KEYSPACE KEYSPACE (OUTER RING) (INNER RING) FIXED DYNAMIC AND SMALLER AND LARGER

RELIABLE BACKGROUND OPERATIONS WITH HASH RING TECHNOLOGY 1 2 3 Demand A Riak DEMAND A WORKER DEMAND A WORKER DEMAND A WORKER RECEIVES DELIVERY & WRITES UUID TO VNODE CRASHES BEFORE IT INITIATES DISPATCH SET IN THE DB AND EXPIRES DISPATCH STARTS TIMER POST /jobs hash(uuid) % 1024

RELIABLE BACKGROUND OPERATIONS WITH HASH RING TECHNOLOGY 4 5 6 Riak 0 1023 DEMAND B SCANS DEMAND B DETECTS DEMAND B LOADS VNODE SET ENTIRE VNODE MEMBERSHIP CHANGE IN FROM DB AND RESTORES KEYSPACE RING BACKGROUND TIMERS for vnode in range(0, 1023) if lookup(vnode) == whoami() restore(load_uuids(vnode))

failure, monitoring and alerting CHAPTER 5 OF 8

FAILURE TESTING MICROSERVICES WITH REPEATABLE FAILURE SCENARIOS

FAULT ISOLATION IN MICROSERVICES WITH DEPLOYMENT PODS POD 1 GEOFENCE FLIPR MICROSERVICE MICROSERVICE 40.645244, -73.9449975 POD 2 SUPPLY v1 DEMAND v1 DISPATCH GATEWAY POD 2 DEMAND v2 SUPPLY v2

MICROSERVICE ALERTING WITH GRAPHITE/NAGIOS INTEGRATED THRESHOLD CHECKS PER REPO THRESHOLDS IMPORTED PYTHON BUILT AGAINST GRAPHITE ALERTS THROUGH NAGIOS

scalability and sharding CHAPTER 6 OF 8

PARTITIONING A MICROSERVICE A SCALABLE GEOSPATIAL INDEX 1 2 3 > convert(40.645, -73.944) “864c244” 864c244 READ OR > hash(“864c244”) WRITE 3747631425 > lookup(3747631425) “10.31.1.2:9000” EARTH IS BROKEN UP INTO GEOSPATIAL READS/WRITES REQUEST IS EITHER HANDLED CELLS . EACH CELL HAS AN ID. CONVERTS LAT/LNG TO CELL OR FORWARDED BY ONE OF ID . CELL ID IS THEN HASHED THE 1300 GEOSPATIAL INDEX ALONG RING . WORKERS.

LOCATION-AWARE MICROSERVICES WITH CONTEXT-SPECIFIC METADATA POST /pickup POST /pickup X-Uber-City-ID: 1 X-Uber-City-Name: New York ?lat=40.70&lng=-73.97 1 2 3 DISPATCH DISPATCH GATEWAY GATEWAY DEVICE SENDS PICKUP DISPATCH GATEWAY RESOLVES SERVICE PROCESSES REQUEST REQUEST TO DISPATCH LAT/LNG AGAINST GEOFENCE WITHIN CONTEXT OF CITY GATEWAY SERVICE

performance and diagnostics CHAPTER 7 OF 8

HIGH-PERFORMANCE MICROSERVICES WITH TCHANNEL @ GITHUB.COM/UBER/TCHANNEL PERFORMANT MULTIPLEXING STREAMING RETRIES + CIRCUIT BREAKING POWERS RINGPOP

HIGH-PERFORMANCE MICROSERVICES WITH NODESTAP @ GITHUB.COM/UBER/NODE-STAP TORCH LIVE PROCESS 1 OPEN FLAMEGRAPH IN BROWSER 2

DEBUGGING MICROSERVICES INSPECT INTERNALS WITH NODE REPL CURL REPL ENDPOINT FOR REPL PORT INSPECT THE STATE OF YOUR WORKER 1 3 TELNET INTO REPL 2

the next generation CHAPTER 8 OF 8

NEXT GENERATION MICROSERVICES A OVERLAY NETWORK FOR MICROSERVICE ROUTING REGISTER REGISTER DISCO MICROSERVICE OPTIC MICROSERVICE FORWARD SEND SEND SUPPLY MICROSERVICE DEMAND MICROSERVICE ROUTING SERVICE

THANKS! Presented by Jeff Wolski <wolski@uber.com> Uber is hiring!