How Elasticsearch powers the Guardians newsroom shay banon @kimchy - - PowerPoint PPT Presentation

▶

Apr 04, 2024 172 likes •514 views

How Elasticsearch powers the Guardians newsroom shay banon @kimchy graham tackley @tackers creator, co-founder and cto director of architecture elasticsearch guardian news and media created in 1936 ... to secure the financial and

SLIDE 1

How Elasticsearch powers the Guardian’s newsroom

graham tackley ■ @tackers

director of architecture guardian news and media

shay banon ■ @kimchy

creator, co-founder and cto elasticsearch

SLIDE 2

SLIDE 3

SLIDE 4

“created in 1936 ... to secure the financial and editorial independence of the Guardian in perpetuity”

SLIDE 5

SLIDE 6

ur in-house real-time traffic tool

SLIDE 7

SLIDE 8

my desktop workstation

production apaches

something htmly

?

SLIDE 9

ssh $SERVER "nice tail -f /apache2/logs/guardian-access_log"

SLIDE 10

my desktop workstation

2 x production apaches publisher ssh “tail” zeromq

x

SEO dashboard

SLIDE 11

SLIDE 12

SLIDE 13

SLIDE 14

my desktop workstation

x

SLIDE 15

Javascript in browser

SNS SQS

hidden pixel

Dashboard Tracker

SLIDE 16

SLIDE 17

SLIDE 18

Javascript in browser

Tracker SNS SQS

hidden pixel

SQS Dashboard Serf elasticsearch Dashboard

SLIDE 19

SLIDE 20

1 2 * m 3 . x l a r g e

i n a n a u t

c a l i n g g r

p ( w i t h m a n u a l s c a l i n g ) i n s t a n c e s t

e ( S S D )

https://github.com/guardian/status-app

SLIDE 21

SLIDE 22

{ "dt": "2014-03-03T02:01:48.026Z", "url": "http://www.theguardian.com/film/2014/mar/03/oscars-2014-winners-list", "queryString": "", "host": "www.theguardian.com", "path": "/film/2014/mar/03/oscars-2014-winners-list", "section": "film", "platform": "r2", "userAgent": { "type": "Browser", "family": "Safari 5.1.9", "os": "OS X 10.6.8", "device": "Personal computer" }, "documentReferrer": "http://www.theguardian.com/world", "browser": { "id": "gA6RUFLhWNQvWdt0rW4r78Fg", "isNew": false }, "referringHost": "theguardian.com", "referringPath": "/world", "isContent": true, "contentPublicationDate": "2014-03-03", "countryCode": "US", "countryName": "United States", "location": { "lonlat": [-73.4409, 41.2094] } }

⇠filter ⇠filter ⇠count per minute

SLIDE 23

{ "query" : { "filtered" : { "query" : { "match_all" : { } }, "filter" : { "term" : { "path" : "/film/2014/mar/03/oscars-2014-winners-list" } } } }, …

SLIDE 24

… "facets": { "Reddit": { "date_histogram": { "field": "dt", "interval": "1m" }, "facet_filter": { "term": { "referringHost": "reddit.com" } } }, "Facebook": { "date_histogram": { "field": "dt", "interval": "1m" }, "facet_filter": { "term": { "referringHost": "facebook.com" } } }, "Google": { "date_histogram": { "field": "dt", "interval": "1m" }, "facet_filter": { "or": { "filters": [ { "prefix": { "referringHost": "www.google." } }, { "prefix": { "referringHost": "news.google." } } ] } } } } }

SLIDE 25

SLIDE 26

/graph/breakdown?section=commentisfree

SLIDE 27

?section=commentisfree

phan.StandardFilters
phan.StandardFiltersToElasticsearch
rg.elasticsearch.index.

query.FilterBuilder

SLIDE 28

{ "query" : { "filtered" : { "query" : { "match_all" : { } }, "filter" : { "term" : { "path" : "/film/2014/mar/03/oscars-2014-winners-list" } } } }, …

SLIDE 29

"filter": { "and": { "filters": [ { "range": { "dt": { "from": "2014-03-03T00:00:00.000Z", "to": "2014-03-03T22:30:59.999Z", "include_lower": true, "include_upper": false } } }, { "not": { "filter": { "term": { "countryCode": "GNM" } } } }, { "not": { "filter": { "term": { "userAgent.type": "Robot" } } } }, { "filter": { "terms": { "section": [ "commentisfree" ] }} } ] } }

SLIDE 30

SLIDE 31

SLIDE 32

thank you

graham tackley ■ @tackers

director of architecture guardian news and media

shay banon ■ @kimchy

creator, co-founder and cto elasticsearch