Building a relevance platform with Couchbase and Elasticsearch - PowerPoint PPT Presentation

OneHippo @ Goto Building a relevance platform with Couchbase and Elasticsearch @jreijn | Hippo #gotoams, June 18 follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto About me • Architect @ Hippo • DevOps guy • Blogger @ http://blog.jeroenreijn.com follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto About Hippo follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Relevance? OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto “The capability of a search engine or function to retrieve data appropriate OneHippo @ Goto to a user's needs.” http://www.thefreedictionary.com/relevance follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto How we deliver relevant content OneHippo @ Goto @Hippo follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Registration Visitor - entity making HTTP requests Collector - records data about a visitor or his behavior Example: location collector (GeoIPCollector) Targeting Data - all data about a specific visitor Example: IP address is located in Amsterdam follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Matching Characteristic - a type of fact about visitors Example: "comes from a city", "experiences a type of weather" Target Group - the specification of a Characteristic Example: "comes from a European city", "comes from Amsterdam" Persona - one or more target groups that describe a certain type of visitor Example: "Jim, the European urban consumer", "Alice, the Pet owner" follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto What do we store? Request log Targeting data Statistics Averages, e.g. how many visitors became which persona follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Real-time analysis follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Architecture OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto (X)HTML JSON XML App server Hippo Delivery Tier Hippo Repository RDBMS follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Request Delivery Tier URL Matching Fetch content Compose output Response follow the Hippo trail Request dinsdag 18 juni 13

OneHippo @ Goto Request Delivery Tier URL Matching Targeting Data Collection Scoring Fetch content Compose output Response follow the Hippo trail Request dinsdag 18 juni 13

OneHippo @ Goto Scaling OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Scaling out App server App server Hippo Delivery Tier Hippo Delivery Tier Hippo Repository Hippo Repository RDBMS follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Scaling out App server App server Delivery Tier Delivery Tier Targeting Datastore Repository Repository RDBMS follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto What kind of ‘storage’? OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Distributed Cache? follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto We have a winner! follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Requirements OneHippo @ Goto change! follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto NoSQL to the rescue OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Suitable types • Key-value store • Document database follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Assessment Criteria Maturity Data model Scalability Replication Performance Reliability Caching model Query model Support Consistency model Monitoring follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Selection Criteria • Performance! • Scalability • Schema flexibility • Simplicity • Monitoring • Support follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Performance !! OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Scalability OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Schema flexibility OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Request log document { "visitorId": "7a1c7e75-8539-40", "pageUrl": "http://localhost:8080/site/news", "pathInfo": "/news", "remoteAddr": "127.0.0.1", "referer": "http://localhost:8080/site/", "timestamp": 1371419505909, "collectorData": { "geo": { "country": "", "city": "", "latitude": 0, "longitude": 0 }, "returningvisitor": false, "channel": "English Website" }, "personaIdScores": [], "globalPersonaIdScores": [] } follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Visitor document { "geo": { "collectorId": "geo", "city": "", "country": "", "latitude": 0, "longitude": 0 }, "channel": { "collectorId": "channel", "channels": [ "English Website" ], "lastVisitedChannel": "English Website" } } follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Simplicity OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Monitoring OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Support OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Couchbase OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Why Couchbase? • Drop-in replacement for memcached • Read/Write-through cache • High throughput • Easy scalability • Schema flexibility • Low latency follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Couchbase • Open Source • Document-oriented • Easy Scalable • Consistent High Performance follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Performance • Object managed cache • Write Queue to disk • Avoids Cold Cache follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Easy scalable • Auto sharding • Cross cluster replication (XDCR) • Master - Master replication follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Flexible data model • Native JSON support • Incremental Map Reduce • Gives power to the developer follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto How we run OneHippo @ Goto Couchbase @Hippo follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Load Balancer Hippo Delivery Tier Couchbase cluster • Request log data • Targeting data Database cluster • Statistics data follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Query capabilities • Querying via views • Secondary indexes via views • Views based on Map - Reduce • Lacks some advanced query capabilities follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Elasticsearch • Apache Lucene • Designed to be distributed • Schema free • Apache 2 licensed • RESTful API follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Added value of ES • Full text search • Faceted search • Geo spatial search • All in (near) real-time follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Replicating to ES Hippo Delivery Tier Java API Write Read Couchbase Server Cluster Elasticsearch Server Cluster Couchbase ES XDCR Transport plugin follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Demo time! OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto What’s Next? OneHippo @ Goto follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Advanced analytics follow the Hippo trail dinsdag 18 juni 13

OneHippo @ Goto Thank you! Questions? OneHippo @ Goto j.reijn@onehippo.com @jreijn ps. We’re hiring! follow the Hippo trail dinsdag 18 juni 13