466382733 Scaling Saved Searches Serving real time push-notifications for millions saved searches
Who are we?
ebay kleinanzeigen ≠ ebay
What are we?
ads = classified ads
some numbers
22M ads live!
18M searches/day
466382733 Saved Searches Serving real time push-notifications for millions saved searches
700k new ads/day 8M saved searches
48.000.000.000 theoretical matches a day!
p r o c e s s i t !
W hat?
H ow?
* * 0/1 * * ?
r e a l t i m e ?
s c a l a b l e ?
C a n w e d o b e t t e r ?
2 0 1 5
src=https://www.esciencecenter.nl/img/main/logo-elastic.png
Percolator Traditionally you design documents based on your data, store them into an index, and then define queries via the search API in order to retrieve these documents. The percolator works in the opposite direction. First you store queries into an index and then, via the percolate API, you define documents in order to retrieve these queries. src=https://www.elastic.co/guide/en/elasticsearch/reference/current/search-percolate.html
H o w m a n y p u s h e s p e r d a y ?
~3x
H ow?
700k new ads/day
match all?
a s k s e a r c h
h o w m a n y r e s u l t s ?
c r e a t e b u c k e t s
0 - 100: RT 101 - 1000: 1h 1001 - 10000: 2h > 10000: 6h
...
l i f e t i m e o f a s e a r c h
s l e e p ... Z Z Z Z Z Z
Z Z Z Z Z Z
S e t u p
S e t u p
cloud
2 data centers
2 data centers 10 data + 3 master
2 data centers 10 data + 3 master
replication x1 shards x80
SOLVED ES5
s k i p o n o v e r l o a d
e l a s t i c f a s t o n i n d e x i n g
f i l t e r s l e e p i n g s e a r c h e s
m e t a d a t a
filter:{ “next_pushdate”: [* TO NOW]}
o n l y 3 0 % s e a r c h e s a r e o n l i n e
d e s k t o p
a v o i d d b - r e a d p e r s e a r c h
h a s h p e r s e a r c h
Recommend
More recommend
