The Architecture of Uber's Realtime System March 25, 2015 Amos Barreto Danny Yuan @amos_barreto @g9yuayon
What is Uber
Uber is a transportation platform
Uber connects riders to drivers
Transportation at your fingertips
What is Realtime?
It’s the brain of Uber’s logistics platform
It assigns drivers to riders Driver Riders
It balances driver & rider satisfaction
Sounds pretty simple, right?
Not Really Realtime Analytics
Not Really Realtime Analytics
Not Really Realtime Analytics
Not Really Realtime Analytics
We didn’t start with this
Instead…
The Beginning • PHP application • Transactions appended to flat files • Half of the code in Español • Lifespan: 6-9 months
But we had to evolve
So we built a distributed state machine
We tried to follow good practices
Simulator is very helpful
Extensive instrumentation
Asynchronous State Machines
Scaling Horizontally W
Scaling Horizontally W W W W W
Stateless Workers W W W W W
Stateless Workers HAProxy W W W W W
Shared States HAProxy W W W W W
Shared States HAProxy W W W W W Redis with T wemproxy
Everything can go down HAProxy W W W W W Redis with T wemproxy
Everything can go down HAProxy W W W W W Redis with Riak T wemproxy
There’s tougher problem to solve
Dispatch needs coordination • Trip states happen in partial order • Rider and driver states may need to be synchronized
Coordination is hard “Minimizing coordination, or blocking communication between concurrently executing operations, is key to maximizing scalability, availability, and high performance in database systems” Coordination Avoidance in Database Systems
Coordination is hard • Row lock? Distributed lock? Consensus protocols? • Insight: What we really need is ordered execution • Solution: Assign requests with the same user-defined key to the same stateless worker node
What about load balance? W W W W W
What about load balance? HAProxy W W W W W
What about load balance? HAProxy W W W W W
What about load balance? HAProxy W W W W W
What about load balance? Consistent Hashing to the rescue
What about load balance? Consistent Hashing to the rescue
Membership changes • A worker in a cluster can crash • A worker can join a cluster • We need fast and reliable failure detector and membership updates
Key Insights • Separate failure detection from membership updates • Do not rely on a single peer for failure detection • Membership changes via gossip-like protocols
SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol
Ringpop • Open source • Hash ring abstraction • Implements a variation of SWIM • Flap damping • Use checksums to verify correctness of ring state • Proxying capabilities
Ringpop var ringpop = new RingPop({ app: 'myapp', hostPort: 'myhost:30000' }); ringpop.bootstrap(['myhost:30001’, 'myhost2:30000']); ringpop.on('ready', function() { // do something }); var node = ringpop.lookup(‘[unique-request-id]’); if (node === ringpop.whoami()) { // process request } else { // forward request }
Ringpop Serial • Simple ringpop wrapper • Requests are queued by key • Processed serially, one at a time • Emulates transactions
Transactions
Transactions • Conflicts are possible during membership changes
Transactions • Conflicts are possible during membership changes • Need smart application level conflict resolution
Global Geospacial Index
Global Geospacial Index • High volume of location updates • Mild, but expensive, query volume • Large search space (the world)
What about crash recovery?
Sevnup • Open sourced node.js module • Ringpop extension • Key ownership hand-off • Customizable recovery & release • Pluggable persistence layer
Keys key432 key988 key654 Virtual 1 2 3 4 5 … Nodes (1024) A B Ringpop Application Cluster D C
Keys key432 key988 key654 Virtual 1 2 3 4 5 … Nodes (1024) A B Ringpop Application Cluster D C
Reliable Timers • Node.js offers in-memory timers • Use sevnup to make them reliable • Riak as persistence layer
Data Center Failure • How do we replicate trip data? • Constants updates • Writes heavy • T emporal, and minimal loss expected
Realtime Trip Replication (RTTR) Key insight: each driver application has trip data already
RTTR • A key-value store on the phone • A timeseries store for partner gps points on the phone • Piggyback on existing communication protocols • All data encrypted
Data in Realtime Realtime Analytics
Ops need realtime analytics
Ops need realtime analytics
Ops need realtime analytics
Ops need realtime analytics
Dispatch needs data for decisions Realtime Analytics
Dispatch needs data for decisions Realtime Analytics
Dispatch needs data for decisions Realtime Analytics
Dispatch needs data for decisions
Applications need real-time data
Applications need real-time data • Notification
Applications need real-time data • Notification • Marketing
Applications need real-time data • Notification • Marketing • Fraud detection
Dispatch can’t do everything
Empower But we use data to empower people
An event-based data platform state driver_arrived from_state driver_accepted timestamp 13244323342 lattitude 12.23 longitude 30.00
An event-based data platform
An event-based data platform • Reliable replication of states
An event-based data platform • Reliable replication of states • Canonical state representation
An event-based data platform • Reliable replication of states • Canonical state representation • Domain specific APIs
Reliable replication of states
Reliable replication of states
Reliable replication of states
Canonical representation of states
Canonical representation of states • Consistency matters
Canonical representation of states • Consistency matters • Normalize your events if possible. E.g., no PII
Canonical representation of states • Consistency matters • Normalize your events if possible. E.g., no PII • More generally: keep apps robust by minimizing assumptions
Recommend
More recommend
