Indexing in Distributed Actor Systems Philip Bernstein Mohammad - PowerPoint PPT Presentation

Indexing in Distributed Actor Systems Philip Bernstein Mohammad Dashti Tim Kiefer David Maier Microsoft EPFL TU Dresden Portland State Univ 8 th CIDR January 9, 2017

Stateful Object-Oriented Applications  Today’s interactive apps are built around a stateful, object -oriented middle tier  Multi-player games, IoT, social networking, mobile, telemetry  They comprise a large fraction of new app development  Naturally object-oriented, modeling real-world objects  Examples of objects  Gaming: players, games, grid positions, lobbies, player profiles, leaderboards, in-game money, and weapon caches  Social: chat rooms, messages, photos, and news items  IoT: sensors, virtual sensors (flood, break-in), buildings, vehicles, locations 2

Application Properties  Properties of these apps  Objects are active for minutes to days, sometimes forever  App manages a lot of state: millions of objects, knowledge graphs, images, videos  App does heavy computation: complex actions, render images, compute over graphs, …  Properties of the system  Scale out to large number of servers  Compute servers must scale out independently of storage servers  Geo-distributed for worldwide low-latency access 3

Middle-tier Objects Comprise a Distributed DB  Many objects outlive the processes that created them  Many (but not all) objects are persistent  Latest state is in main memory. Storage might be stale  Active objects are in-memory for fast response 4

Actor Systems  Many of these apps are implemented using actor systems  Simplifies distributed programming  Actors are objects that …  Communicate only via asynchronous message-passing  Messages are queued in the recipient's mailbox  No shared-memory state between actors  Process one message at a time  No multi-threaded execution inside an actor 5

Orleans Actor Programming Framework  Orleans is an open-source actor framework built on C#  Ensures apps are fault tolerant and scalable  https://dotnet.github.io/orleans/  Virtual actor model  Each actor has a unique location-independent ID, always valid  Actors are transparently activated on invocation  On activation, actor invokes its constructor to initialize its state (e.g., read from storage)  Actor can save state at any time (e.g., to storage)  Runtime automates fault-tolerance , load balancing, actor lifecycle, … 6

Actor-Oriented Database System (AODB)  Current distributed actor systems lack DB functionality  Persistence But users frequently ask for it (and hack it)  Vision: Actor-Oriented DB System Transactions  Indexes, queries, streams, transactions, Indexing replication, geo-distribution, views, triggers Geo-  AODB’s main distinguishing features distribution  Compatible with actor framework’s Frontend Actor AODB Cloud programming model (developer friendly) Clients Middle-Tier Plug-ins Storage  In-memory and elastically scales out to hundreds of servers  Agnostic to the storage system, e.g., cloud storage services 7

Scalable and Storage-Agnostic  Elastic scalability implies  Limited ability to co-locate functionality  Functionality must be parallelizable  Scale-out is more important than a fast path  Storage agnostic implies each DB feature  Must work for persisted and non-persisted objects  Must not require the storage system to support it  Should benefit from a storage system that does support it  Must cope with storage latency of cloud storage 8

Requirements for AODB Indexes  Statically choose indexed fields  Optional uniqueness constraints (e.g., ensure Player.Email is unique)  Index is eventually-consistent with actor and fault tolerant  Can index active actors only (e.g., offer a tournament to certain on-line players)  Can index persistent and non-persistent actors  Leverage actor storage that supports indexing  Support actor storage that does not support indexing 9

Challenges  Lookup should avoid activating actors  No type extents  No multi-actor transactions 10

Fault T olerance  Index is comprised of actors, to gain benefits of Orleans  Suppose we have an index on Player.Location 2. Update index HashIndex on Player A Player.Location 3. Update index storage 1. Update storage HashIndex on Player Storage Player.Location in Storage  Ensure recoverability after each write to storage 12

Our solution: Multi-step Fault-tolerant Workflow Local workflow queue HashIndex on Player A Player.Location HashIndex on Player Workflow queue Player.Location in Storage Storage Storage 13

Our solution: Multi-step Fault-tolerant Workflow Cont. 4.1. Check if Player has the workflow record, too Local workflow queue 1. Add update to queue 4. Batch update the index HashIndex on Player A Player.Location 6. Remove 3. Update storage 4.2. Update Batch write workflow including 5. 2. to Storage record ID workflow record ID HashIndex on Player Workflow queue Player.Location in Storage Storage Storage 14

Index Physical Representation One index-actor per server Entire index in one actor One index-actor per index bucket Player D Player D Player F Player A Player E Player E Player F Player C Player B HashIndex on HashIndex on Player.Location for actors on Server 1 Player in Redmond HashIndex on Player.Location HashIndex on Player.Location for HashIndex on actors on Server 2 Player in Bellevue Player F Player E Player C Player B Player C Player B Player A Player A Player D 15

Programming Interface: Index Definition public interface IPlayer : IIndexableGrain<PlayerProperties> { Task Move(Direction d); Task<string> GetLocation(); } public class Player : public class PlayerProperties IndexableGrain<PlayerState, PlayerProperties>, IPlayer { { public int Rank { get; set; } public Task Move(Direction d) { [Index] State.Location = public string Location { get; set; } d.GetDestination(State.Location); } return WriteStateAsync(); } public class PlayerState { public Task<string> GetLocation() public string Name { get; set; } { public int Rank { get; set; } return Task.FromResult(State.Location); public string Location { get; set; } } } } 16

Programming Interface: Index Lookup  Use LINQ to access the index IOrleansQueryable<IPlayer> activePlayersInRedmond = from player in GrainFactory.GetActiveGrains<IPlayer, PlayerProperties>() where player.Location == "Redmond" select player; //IOrleansQueryable extends IQueryable interface foreach(IPlayer player in activePlayersInRedmond) { Console.WriteLine(player.GetPrimaryKeyLong()); } 17

Performance none one-bucket perkey persilo 90 30 7 80 6 25 70 5 (kilo requests/second) (kilo requests/second) (kilo requests/second) 20 Throughput Throughput Throughput 60 4 15 3 50 10 2 40 1 5 30 20 not A-index NFT FT SM 5 10 15 20 0 1 2 3 4 indexed I-index I-index index Number of middle-tier servers Number of Indexes Index Type 18

Future Work on Indexing  Transactionally update actor and index  Range indexes  Richer materialized views  Offer indexing with other AODB features, e.g., transactions, queries, geo-dist ’ n 19

Status of Orleans’ AODB Features  Stream processing (January 2015)  Geo-distribution and multi-master replication (January 2016)  Distributed transactions (preview, this month) [MSR Technical Report]  Indexing (prototype, August 2016) 20

Acknowledgments  Sebastian Burckhardt, Sergey Bykov, Julian Dominguez, Tova Milo, Jorgen Thelin, Microsoft Studios and the Orleans community.  More at https://dotnet.github.io/orleans/ 21

22 Thank you!