RedisGears Redis in memory data processing JUNE 2019 | PIETER - PowerPoint PPT Presentation

RedisGears – Redis in memory data processing JUNE 2019 | PIETER CAILLIAU

About me • Produced in Belgium • (instanceof) SE @ TomTom • Consultant @ neo4j • Solution Architect @ Redis Labs • Product Manager @ Redis Labs • @cailliaup �2

Agenda What is Redis and Redis Enterprise 1 Stream Processing with RedisGears 2 RedisGears as a Multimodel Engine 3

Redis is Fast … �4

… Extremely Fast �5

DB-Engines Ranking �6

And you’ve been using it already �7

Redis is Extensively and Diversely Used Uses Redis for: Uses Redis for: Uses Redis for: All messages Geo search, user profiles Timeline, following Scope: 40 TB Scope: 10-20 TB Scope: 10-20 TB Uses Redis for: Uses Redis for: Repository router Local/site/global caching Scope: 10+ TB �8

Redis Top Differentiators 1 2 3 Performance Simplicity Extensibility NoSQL Benchmark Redis Data Structures Redis Modules Strings Sets Bitmaps Sorted Sets Bit field Geospatial Indexes Hashes Hyperloglog Lists Streams �9

Redis Speed differentiators OPTIMIZED ARCHITECTURE ADVANCED PROCESSING EFFICIENT OPERATION ✓ Written in C ✓ Most commands are ✓ Easy to parse networking executed with O(1) protocol ✓ Served entirely from memory complexity ✓ Pipelining for reduced ✓ Single-threaded, lock free ✓ Access to discrete elements network overhead within objects ✓ Connection pooling ✓ Reduced bandwidth/ overhead requirements �10

Redis Speed differentiators OPTIMIZED ARCHITECTURE ADVANCED PROCESSING EFFICIENT OPERATION ✓ Written in C ✓ Most commands are ✓ Easy to parse networking executed with O(1) protocol ✓ Served entirely from memory complexity ✓ Pipelining for reduced ✓ Single-threaded, lock free ✓ Access to discrete elements network overhead within objects ✓ Connection pooling ✓ Reduced bandwidth/ overhead requirements �11

Modules Extend Redis Infinitely • Create your own data types and commands • Reuse Redis’ simplicity, performance, scalability and high availability. • Can be written in C/C++/Go/Python/Rust/Zig • Leverage existing data structures • Turn Redis into a Multi-Model database https://redislabs.com/community/redis-modules-hub/ �12

Redis Modules RediSearch (GA) redisearch.io RedisBloom (GA) redisbloom.io RedisTimeSeries redistimeseries.io RedisJSON (GA) redisjson.io RedisAI redisai.io RedisGraph (GA) redisgraph.io �13

Introducing �14

Redis Enterprise DBaaS Software • Available since mid 2013 • Available since early 2015 • 8,500+ enterprise customers • 300+ enterprise customers 550K+ databases managed worldwide Customers • 6 of top Fortune 10 companies • 3 of top 4 credit card issuers • 3 of top 5 communications companies • 3 of top 5 healthcare companies �15

Cloud Providers have different incentives • DBAAS Provider • Cloud Provider • Higher margin by • Higher margin by better resource utilization – Idleness – Multi-tenancy • Cloud Lock-in – Reducing RAM – CPU utilization �16

Redis Enterprise : A Unique Primary Database RELIABLE FAST FLEXIBLE HIGHEST PERFORMANCE, BUILT-IN HIGH PERFORMANCE 
 LINEAR SCALING SEARCH HIGH AVAILABILITY WITH INSTANT 
 MULTI-MODEL FAILOVER FLEXIBLE DEPLOYMENT OPTIONS DURABILITY AT MEMORY SPEEDS (CLOUD, ON-PREM, HYBRID) ACTIVE-ACTIVE GEO DISTRIBUTION INTELLIGENT TIERED DATA ACCESS (CRDT-BASED) (RAM & FLASH MEMORY)

Redis Enterprise Cluster Uneven number of symmetric nodes Node 1 Node 2 Node N (odd number) �18

Redis Enterprise Cluster Single master database M Node 1 Node 2 Node N (odd number) �19

Redis Enterprise Cluster An HA database S M Node 1 Node 2 Node N (odd number) �20

Redis Enterprise Cluster A Clustered Database M1 M2 M3 Node 1 Node 2 Node N (odd number) �21

How do keys get assigned to partitions? �22

How do keys get assigned to partitions? �23

Redis Enterprise Cluster A Highly Available Clustered Database S3 M1 S1 M2 S2 M3 Node 1 Node 2 Node N (odd number) �24

Redis Enterprise Node Enterprise Layer Cluster Manager Zero latency proxy REST API Redis Shards Open Source Layer �25

Redis Enterprise: Shared Nothing Symmetric Architecture Data-Path and Control/Management Path Separation Cluster Node Watchdog Management Path Cluster Watchdog Redis Shards 
 Data Path & Proxies Node 1 Node 2 Node N (odd number) �26

as a datagrid �27

Microservices Architecture and Polyglot Persistence K V K V Customers Authentication Key/Value Key/Value Document Session Store API API Graph API Authentication API K V Authentication API Key/Value K V API Key/Value Fulfilment Session Store API RDBMS Catalog Customers API API API API K V Fraud Session Store Search Detection Key/Value Document RDBMS Cache Graph �28 Document Search Columnar

The Cost of Polyglot Persistence K V K V Customers Authentication Key/Value Key/Value Document Session Store API API Graph Increased application complexity � Costly communication API Application does heavy lifting in sharing data, keeping data sets in sync Authentication API K V Authentication API High operational burden � Higher cost of ownership Key/Value K V API Different databases have specialized administrative, scaling, availability Key/Value Fulfilment Session Store API requirements RDBMS Catalog Customers API Sub-optimal Resource Usage � Higher cost API API API Dedicating pods/servers for each type of database reduces deployment efficiency K V Fraud Session Store Search Detection Key/Value Document RDBMS Cache Graph Document Search Columnar

Redis Enterprise: A Multi-model Database for Microservices API API API API API API API Fulfilment Catalog Session Store Authentication Fraud Detection Customers Search Graph Search Key/Value Key/Value Key/Value Cache Document Cache RDBMS RDBMS

Built-in Pub-Sub / Streams for event synch across data stores Built-in Message Broker Fulfilment Catalog Session Store Authentication Fraud Detection Customers Search Graph Search Key/Value Key/Value Key/Value Cache Document Cache RDBMS RDBMS

What are we missing? • How to consume messages in this “built in message borker” • Given a sharded database, how can I run analytical queries? • Multi Model database – Single copy in core datatypes – Inter module communication – Component X doing translations between modules.

� 33 Introducing RedisGears

What is RedisGears? RedisGears is a Serverless engine for multi-model and cluster operations in Redis, supporting both event driven as well as batch operations

High Performance Architecture Soon Soon C - API MapReducer GearsCoordinator GearsExecuter Gears infrastructure is written in C

Scripting with RedisGears RedisGears allows to define a pipe of operations • Returning value from one operation is passed to the next operation in the pipe • Last operation returning the result to the user • First operation is called ‘reader’ - responsible for providing data – Keys reader - read keys from Redis – Stream reader - read streams from Redis – Python reader - allow to user to write his own readers in python �36

Supported Operations • Map • FlatMap • Filter • Groupby + Reduce • Aggregate • Sort • Limit • ForEach • Distinct �37

Using RedisGears – (Flat)Mapping (Flat) Reader Mapper

Using RedisGears - Filtering Filter record Reader with 1 doc

Using RedisGears - Aggregate 1 1 Count Reader Aggregator

Demo

Use Case #1 – Stream Processing Gears has a streaming API to allow to trigger gears execution on events. – Redis Stream events - Trigger an execution whenever a new data enters a steam – Redis Keys events - Trigger an operation whenever a key is touched Every sec RedisTimeSeries Redis Streams

Use Case #2 – a MultiModel Engine Because of RedisGears’ flexibility (it's actually running python) you can achieve internal module integration with it: – Read from hashes and index in RediSearch/RedisGraph – Read RedisJSON data and pass to RedisTimeSeries – … RediSearch Every update Redis Hashes RedisGraph


 
 Recipe #1 – even triggering Build a gear that creates maintains a set of all keys within redis # create the builder builder = GearsBuilder() # filter events on key:'all_keys' builder.filter(lambda x: x['key'] != 'all_keys') # add the keys to 'all_keys' set builder.map(lambda x: execute('sadd', 'all_keys', x['key'])) # register the execution on key space notification builder.register()

Recipe #2 – map reducing Build a gear that counts how often a genre is used within a set of movies # create the pipe builder. KeysOnlyReader is a performance improvement only piping the keys. builder = GearsBuilder('KeysOnlyReader') # get from each hash the genres field builder.map(lambda x: execute('hget', x, 'genres')) # filter those who do not have genres builder.filter(lambda x: x is not None) # split genres by comma builder.flatmap(lambda x: x.split(',')) # count for each genre the number of times it appears builder.countby() # start the execution builder.run('movie:*')