Intro to Redis Streams IMCSUMMIT - NOVEMBER 2019 | DAVE NIELSEN
• What is a data stream? • What are the typical challenges you face while managing data streams? • What is Redis Stream? • How Redis Stream addresses the challenges? • How to use Redis Streams? 2
Who We Are Open source. The leading in-memory database platform , supporting any high performance operational, analytics or hybrid use case. The open source home and commercial provider of Redis Enterprise technology, platform, products & services. 3
Redis Top Differentiators 1 2 3 Performance Simplicity Extensibility NoSQL Benchmark Redis Data Structures Redis Modules Strings Sets Bitmaps Sorted Sets Bit field Geospatial Indexes Hashes Hyperloglog Lists Streams 4
Data Stream Use Cases 5
What is a Data Stream? 6
Simplest form of a Data Stream Producer Consumer 7
But Often You Have M Producers and N Consumers Producer 1 Consumer 1 Producer 2 Consumer 2 Producer 3 Consumer 3 Consumer n Producer m 8
It’s a bit more complex then that 9
Data Stream Components Message Broker or Ingest Stream Output Stream Analytics Stream Processor Store the data and Collect large perform analytical IoT Transform Push the data to volume of data operations such as incoming data the consumers arriving in high predictive analytics, into one or velocity ML training, ML more formats Allow backward classification and as required by looking queries to Store the data categorization, Activity logs the consumers pull data temporarily recommendations, pattern matching, etc. Messages 10
11
Redis Supports All the Functions of a Data Stream Message Broker Ingest Stream or Output Stream Analytics Stream Processor Redis • Streams • Streams • Streams • Sorted Sets • Pub/Sub • Pub/Sub • Sorted Sets • Sets • Lists • Lists + • Geo • Spark Connector • Bitfield • ODBC/JDBC Connector + • RedisGraph • RediSearch 12
13
Understanding Redis Streams 14
Let’s look at t the the ch challen llenges es fir irst 15
How to Support a Variety of Consumers Real-time Messaging Real-time or Periodic Lookup Producer Analytics Periodic Read Data Backup Consumers 16
The Catch Up Problem Image Processor Producer Consumption Rate: 100/sec Arrival Rate: 500/sec Redis Stream Fast Slow Backlog 17
How to Address The Catch Up Problem? Image Processor Image Processor Image Processor Producer Arrival Rate: 500/sec Image Processor Redis Stream Image Processor Consumption Rate: 500/sec Scale Out New Problem: Mutual Exclusivity 18
Data Recovery from Consumer Failures Image Processor Image Processor Image Processor Producer Arrival Rate: 500/sec Image Processor Redis Stream Image Processor Consumption Rate: 500/sec Scale Out Yet Another Problem: Failure Scenarios 19
Connection Failures Image Processor Image Processor Image Processor Producer Image Processor Redis Stream Image Processor The solution must be resilient to failures 20
Here comes Redis Streams 21
What is Redis Streams? Pub/Sub Lists Sorted Sets It is like Pub/Sub, but It is like Lists, but decouples It is like Sorted Sets, but with persistence producers and consumers asynchronous + Lifecycle management of streaming data • Built-in support for timeseries data • A rich choice of options to the consumers to read streaming and static data • Super fast lookback queries powered by radix trees • Automatic eviction of data based on the upper limit • 22
1. It enables asynchronous data exchange between producers and consumers Messaging Producer Consumer 23
2. You can consume data in real-time as it arrives or lookup historical data Messaging Producer Analytics Data Backup Consumers 24
3. With consumer groups, you can scale out and avoid backlogs Image Processor Image Processor Image Processor Producer Arrival Rate: 500/sec Image Processor Redis Stream Image Processor Consumption Rate: 500/sec 25
4. Simplify data collection, processing and distribution to support complex scenarios Deep Learning-based Classification Classifier 1 Classifier 2 XREADGROUP Producer 1 XADD XACK Classifier n Producer 2 Consumer Group Producer 3 Messaging Producer m Analytics XREAD Data Backup 26 Consumers
Redis Streams Commands Quick Reference Guide https://redislabs.com/docs/getting-started-redis-streams/ 27
Commands: XADD, XREAD Asynchronous producer-consumer message transfer Messaging Producer Consumer XADD XREAD XADD mystream * name Anna XREAD COUNT 100 STREAMS mystream 0 XADD mystream * name Bert XADD mystream * name Cathy XREAD BLOCK 10000 STREAMS mystream $ 28
Commands: XRANGE, XREVRANGE Lookup historical data Messaging Producer Analytics XADD Data Backup XREAD Consumers XRANGE XREVRANGE XRANGE mystream 1518951123450-0 1518951123460-0 COUNT 10 XRANGE mystream - + COUNT 10 29
Scaling out using consumer groups Image Processor Image Processor Image Processor Producer Arrival Rate: 500/sec Redis Stream Image Processor Consumption Rate: 500/sec 30
Scaling out using consumer groups Consumer Group Image Processor Consumer 1 Image Processor Consumer 2 Producer Consumer 3 Image Processor Arrival Rate: 500/sec Redis Stream Unconsumed List Consumer n Image Processor Consumers Consumption Rate: 500/sec XADD XGROUP XREADGROUP XACK XPENDING XCLAIM 31
Command: XGROUP CREATE Create a consumer group mygroup mystream Alice App A a b c d e a b c d e Bob App B Unconsumed List XGROUP CREATE XGROUP CREATE mystream mygroup $ MKSTREAM 32
Command: XREADGROUP Read the data mygroup e d Alice App A a b c Bob App B Unconsumed List XREADGROUP XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream > XREADGROUP GROUP mygroup COUNT 2 Bob STREAMS mystream > 33
Command: XACK Consumers acknowledge that they consumed the data XACK mygroup Alice App A a b c Bob App B Unconsumed List XACK mystream mygroup 1526569411111-0 1526569411112-0 34
Command: XREADGROUP Repeat the cycle mygroup a Alice App A b c Bob App B Unconsumed List XREADGROUP XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream > 35
How to claim the data from a consumer that failed while processing the data? mygroup Alice App A c b Bob App B Unconsumed List XCLAIM mygroup c b Alice App A Bob App B Unconsumed List 36
Commands: XPENDING, XCLAIM Claim pending data from other consumers mygroup b c Alice App A Bob App B Unconsumed List XPENDING mystream mygroup - + 10 Bob XCLAIM mystream mygroup Alice 0 1526569411113-0 1526569411114-0 37
Now is the time to 38
Sample Use Case: Social Media Analytics Sentiment analyzers Consumer is x times slower than the a producer b XREADGROUP x Language classifier Redis Streams Influencer classifier Location classifier XREAD Real-time reports 39
Demo Influencer Classifier Twitter Ingest Stream Redis Streams https://github.com/redislabsdemo/IngestRedisStreams 40
Redis Clients that Support Redis Streams Java Jedis https://github.com/xetorthio/jedis https://lettuce.io/ Lettuce https://github.com/andymccurdy/redis-py Python redis-py https://github.com/luin/ioredis JavaScript ioredis https://github.com/NodeRedis/node_redis node_redis https://github.com/StackExchange/StackExchange.Redis .NET StackExchange.Redis https://github.com/gomodule/redigo Go redigo https://github.com/go-redis/redis go-redis https://github.com/redis/hiredis C Hiredis https://github.com/nrk/predis PHP predis https://github.com/phpredis/phpredis phpredis https://github.com/redis/redis-rb Ruby redis-rb 41
Questions ? ? ? ? ? ? ? ? ? ? ?
Thank you! redislabs.com 43
Recommend
More recommend
