10 Ways to Scale with Redis IMCSUMMIT - NOVEMBER 2019 | DAVE NIELSEN - PowerPoint PPT Presentation

10 Ways to Scale with Redis IMCSUMMIT - NOVEMBER 2019 | DAVE NIELSEN

In-memory Multi-model Database 2

Optionally Persistent 3

Cluster Throughput (@ 1 msec Latency) 50,420,000 41,190,000 ops/sec 30,200,000 21,120,000 11,380,000 5,020,000 3 6 12 18 24 26 # of nodes 4

Developers + Redis 5

Redis Top Differentiators 1 2 3 Performance Simplicity Extensibility NoSQL Benchmark Redis Data Structures Redis Modules Strings Sets Bitmaps Sorted Sets Bit field Geospatial Indexes Hashes Hyperloglog Lists Streams 6

1 Performance: The Most Powerful Database Highest Throughput at Lowest Latency Least Servers Needed to in High Volume of Writes Scenario Deliver 1 Million Writes/Sec 40k Servers used to achieve 1M writes/sec 500 394.42 381.31 372.31 350 Application requests/sec Latency in Milliseconds 300 400 30k 250 300 200 20k 150 200 100 10k 100 50 71.22 0 0 0 Couchbase Cassandra Datastax Redis e Redis e Cassandra Couchbase $2,226,216 $371,040 $14,832 ANNUAL COST Applica � on Latency (msec) 150X 25X COST COMPARED e TO REDIS Benchmarks performed by Avalon Consulting Group Benchmarks published in the Google blog 7

Simplicity: Data Structures - Redis’ Building Blocks 2 “REDIS IS FULL OF DATA STRUCTURES!”

2 Simplicity: Redis Data Structures – ’Lego’ Building Blocks Strings Sets "I'm a Plain Text String!” { A , B , C , D , E } Bitmaps Sorted Sets 0011010101100111001010 { A: 0.1, B: 0.3, C: 100 } Bit field Key Geospatial Indexes {23334}{112345569}{766538} { A: (51.5, 0.12), B: (32.1, 34.7) } Hashes Hyperloglog { A: “foo”, B: “bar”, C: “baz” } 00110101 11001110 Streams Lists à {id1=time1.seq1(A:“xyz”, B:“cdf”), [ A → B → C → D → E ] d2=time2.seq2(D:“abc”, )} à = ”Retrieve the e-mail address of the user with the highest ZREVRANGE 07242015_2300 0 0 bid in an auction that started on July 24th at 11:00pm PST” 9

3 Extensibility: Modules Extend Redis Infinitely • Add-ons that use a Redis API to seamlessly support additional use cases and data structures. • Enjoy Redis’ simplicity, super high performance, infinite scalability and high availability. • Any C/C++/Go program can become a Module and run on Redis. • Leverage existing data structures or introduce new ones. • Can be used by anyone; Redis Enterprise Modules are tested and certified by Redis Labs. Model database • Turn Redis into a Mu Multi-Mo 10

3 Extensibility: Modules Extend Redis Infinitely • Re RediSearch • Redis-Rating • Re RedisTimerseries • Redis-Cuckoofilter • Re ReJSON • Cthulhu • Re Rebloom • Redis Snowflake • Re RedisGraph • redis-roaring • Neural-Redis • Session Gate • Redis-Cell • ReDe • Redis-Tdigest • TopK • Redis-ML • countminsketch 11

Deep Dive

Real Time User Session Real Time Data High Speed Job & Queue Analytics Store Ingest Transactions Management Time Series Data Complex Notifications Distributed Lock Caching Statistical Analysis Very Large Data Sets Geospatial Data Streaming Data Machine Learning Search 13

Manage Session Stores w/ Redis Hash

A user session store is… • An chunk of data that is connected to one “user” of a service – ”user” can be a simple visitor – or proper user with an account • Often persisted between client and server by a token in a cookie* – Cookie is given by server, stored by browser – Client sends that cookie back to the server on subsequent requests – Server associates that token with data • Often the most frequently used data by that user – Data that is specific to the user – Data that is required for rendering or common use • Often ephemeral and duplicated

Session Storage Uses Cases Traditional Intelligent • Username • Traditional + • Preferences • Notifications • Name • Past behavior – content surfacing • “Stateful” data – analytical information – personalization

In a simple world Internet Server Database

Good problems Internet Server Database Traffic Grows… Struggles

Good solution performance restored Internet Server Database Session storage on the server

More good problems Struggling Internet Server Database Session storage on the server

Problematic Solutions Load balanced Internet Server Database Session storage Session storage on the server on the server

Multiple Servers + On-server Sessions? Server #1 – Hello Robin! Robin Server Database

Multiple Servers + On-server Sessions? Server #3 – Hello ???? Robin Server Database

Better solution Redis Session Storage Load balanced Internet Server Database

User Session Store • Th The Prob oblem Why Redis Rocks Hashes are perfect for this! • Maintain session state across • HSET lets you save session • multiple servers variables as key/value pairs Multiple session variables • HGET to retrieve values • HINCRBY to increment any • High speed/low latency required • field within the hash structure

Redis Hashes Example - https://redis.io/commands#hash hash key: usersession:1 HMSET usersession:1 userid 8754 name dave ip 10:20:104:31 hits 1 userid 8754 name dave HMGET usersession:1 userid name ip hits ip 10:20:104:31 HINCRBY usersession:1 hits 1 hits 1 lastpage home HSET usersession:1 lastpage “home” HGET usersession:1 lastpage HDEL usersession:1 lastpage EXPIRE usersession:1 10 or DEL usersession:1 Hashes store a mapping of keys to values – like a dictionary or associative array – but faster

Managing Queues w/ Redis Lists

Managing Queues of Work • Th The Prob oblem Why Redis Rocks Lists are perfect for this! • Tasks need to be worked on asynch • LPUSH, RPUSH add values at • to reduce block/wait times beginning or end of queue Lots of items to be worked on • RPOPLPUSH – pops an item • Assign items to worker process and • from one queue and pushes it remove from queue at the same time to another queue Similar to buffering high speed data- • ingestion

Redis Lists Example - https://redis.io/commands#list LPUSH queue1 orange LPUSH adds values to head of list LPUSH queue1 green LPUSH queue1 blue RPUSH queue1 red blue green orange .. .. red RPUSH adds value to tail of list

Redis Lists Example - https://redis.io/commands#list LPUSH queue1 orange LPUSH queue1 green LPUSH queue1 blue RPUSH queue1 red blue green orange .. .. red RPOPLPUSH queue1 queue2 RPOPLPUSH pops a value from one list and pushes it to another list LLEN queue1 LINDEX queue1 0 LRANGE queue1 0 2

Managing Tags w/ Redis Sets

Managing Tags Example Why Redis Rocks • Th The Prob oblem Sets are unique collections of strings • Loads of tags • SADD to add tags to each article • Find items with particular tags • SISMEMBER to check if an article has • a given tag High speed/low latency required • SMEMBERS to get all the tags for an • Also used for: article SINTER to find which articles are • Recommending Similar Purchases • tagged with tag1, tag2 and tag77 Recommending Similar Posts •

Redis Sets Example – https://redis.io/commands#set article 1 SADD article:1 tag:1 tag:22 tag:24 tag:28 SADD tag:1 article:1 tag1 tag22 tag24 tag28 SADD tag:1 article:3 SADD tag:2 article:22 SADD tag:2 article:14 tag 1 SADD tag:2 article:3 article 1 article 3 …. SISMEMBER article:1 tag:1 tag 2 SMEMBERS article:1 article 3 article 14 article 22 .. SINTER tag:1 tag:2

Managing Leaderboards w/ Redis Sorted Sets

Leaderboard with Sorted Sets Example • Th The Prob oblem Why Redis Rocks Sorted Sets are perfect! • MANY users playing a game or • Automatically keeps list of • collecting points users sorted by score Display real-time leaderboard. • ZADD to add/update • ZRANGE, ZREVRANGE to get • Who is your nearest competition • user Disk-based DB is too slow • ZRANK will get any users • rank instantaneously

Redis Sorted Sets - https://redis.io/commands#sorted_set ZADD game:1 10000 id:1 44000 id:3 ZADD game:1 21000 id:2 ZADD game:1 34000 id:3 ZADD game:1 35000 id:4 35000 id:4 ZADD game:1 44000 id:3 34000 id:3 + 10000 id:3 or ZINCRBY game:1 10000 id:3 ZREVRANGE game:1 0 0 21000 id:2 ZREVRANGE game:1 0 1 WITHSCORES 10000 id:1

Searching within a Geospatial Index

Search within a Geographic Area Example • Th The Prob oblem Why Redis Rocks GEOADD to add an item • MANY moving items within a • Redis Geo uses Sorted Sets • geographical area so related Z-commands such Display real-time locations. • as ZRANGE & ZREM are useful too What is the nearest item NOW • GEODIST to find distance • SQL Queries are too slow • between to points GEORADIUS to find all points • within an area