One Year with Sagemaker Our experience of enhancing SaaS products - PowerPoint PPT Presentation

One Year with Sagemaker Our experience of enhancing SaaS products based on custom Deep Learning Dennis Weyland – econda GmbH | September 9, 2019 Community Day 2019 Sponsors

Dennis Weyland Big Data & Artificial Intelligence » Realtime applications » Cloud Infrastructure » Technical Business Developement » Since 2018 at econda » Data Engineer Personalization Team econda GmbH E-Mail: weyland@econda.de

AGENDA 1. Starting Point: Perfect Salesperson 3. The Architecture » Who visits the shop? » Requirements » What do I know about this person? » AWS Service used » How to increase revenue? » Latencies 2. The Deep Learning Model 4. One Year later: What has changed? » The Combined Model » Model » The technology behind it ( Demo ) » Use Cases » What was the uplift? » Architecture » Feature Wishlist

WHO WHO visits my shop/app/portal? WHAT WHAT do I know about this person? HOW HOW can I generate more revenue?

IDENTITY + DEVICE MANAGEMENT A central ID graph is used to collect and link IDs of different type from various source systems. This graph then allows to connect different databases – Online and Offline. Customer ID (Web shop) ONSITE ONSI TE Web tracking External ID OFFLINE OFFLI NE Recipient ID CRM 1st party cookie ERP User ID ESP Mobile ID MOBILE Mobile app tracking Email hash OFFSITE OFFSIT Adserver networks Adserver ID

REALTIME PROFILE Each realtime profile contains data from different channels. The different types of data support different use cases. DIGITAL PROFILE DATA » Data from digital channels (e.g. Web tracking, apps) » Online customer journey with all relevant touchpoints Cross device data » TARGETING DATA Data for future personalization » CRM PROFILE DATA » Bridge to campaign and BI systems » Data from CRM and ERP systems » Personal record of customers Offline customer journey with all » relevant touchpoints

Usage of Artificial Intelligence A trained model is able to predict in real time. Those Prediction can be used for next best offers, categories or target groups. NEXT BEST OFFERS DEEP LEARNING MODELL Viewed Products Shopping Basket Bought Products Search Terms Target Groups Affinities Gender Age Favorite Brands

The Combined Model older than Classic today Viewed Products Shopping Basket Void Combined Bought Products Search Terms Target Groups Affinities Gender Age Live from Favorite Brands today

The Combined Model Classic Void Live » Interactions are from before » Interactions are from today » For unknown users today » Chronological order is » Use campaign information » Chronological order does important » Use weekday and hour not matter » The more recent, the more » Before the first product » Recos are slow to change relevant interaction » Reflects general interest » One click completely change Recos Combined » Current curiosity » Input is output of other models » Combines the best of all

LIVE DEMO

A/B TESTING The added value of using new algorithms has to evaluated based on A/B tests with real customers. A B ARTIFICIAL INTELLIGENCE ARTIFICIAL INTELLIGENCE Usage of profile data Usage of profile data » » Deep Learning Deep Learning » » VS STATISTICAL PERSONALIZATION STATISTICAL PERSONALIZATION » Usage of profile data » Usage of profile data » Collaborative filtering, business rules » Collaborative filtering, business rules STATIC CONTENT STATIC CONTENT » One size fits all » One size fits all » Top sellers, manually selected content » Top sellers, manually selected content

A/B TEST – PERSONALIZED MAILINGS A B ARTIFICIAL INTELLIGENCE ARTIFICIAL INTELLIGENCE Usage of profile data Usage of profile data » » Deep Learning Deep Learning » » VS STATISTICAL PERSONALIZATION STATISTICAL PERSONALIZATION » Usage of profile data » Usage of profile data » Collaborative filtering, business rules » Collaborative filtering, business rules STATIC CONTENT STATIC CONTENT » One size fits all » One size fits all » Top sellers, manually selected content » Top sellers, manually selected content

A/B TEST – PERSONALIZED MAILINGS NBOs Manually selected (Realtime profile + recommendations Deep Learning) VS

EINSATZ VON KÜNSTLICHER INTELLIGENZ A/B-Test Ergebnisse aus A/B-Tests in Mailings zeigen das enorme Potential von Deep Learning in den Push-Kanälen. Mailing to customers (6-digit number of recipients) A: Widget with manually B: Widget mit Deep NBOs Redaktionelle (Echtzeitprofile + Deep selected recommendations Learning NBOs Empfehlungen Learning) VS 22% ??? Increase of revenue (%, last touch):

A/B TEST – PERSONALIZED ENTRY PAGE A B ARTIFICIAL INTELLIGENCE ARTIFICIAL INTELLIGENCE Usage of profile data Usage of profile data » » Deep Learning Deep Learning » » VS STATISTICAL PERSONALIZATION STATISTICAL PERSONALIZATION » Usage of profile data » Usage of profile data » Collaborative filtering, business rules » Collaborative filtering, business rules STATIC CONTENT STATIC CONTENT » One size fits all » One size fits all » Top sellers, manually selected content » Top sellers, manually selected content

EINSATZ VON KÜNSTLICHER INTELLIGENZ A/B-Test Trainierte Modelle können in Echtzeit ausgewertet werden um zum Beispiel Next Best Offers, Affinitäten oder Kunden-Scores zu berechnen. Widget on main entry page of Web shop NEXT BEST OFFERS DEEP LEARNING MODELL A: Widget based on statistical B: Widget based on recommendations Deep Learning NBOs (e.g. bought together) Gesehene Produkte Warenkorb Gekaufte Produkte 12% Suchbegriffe ??? Increase of revenue (%):

The Architecture: What do we want to accomplish? > THE SERVICE MUST SCALE UP PER CLIENT Each of our client need its own training and inference service. > TRAINING SHOULD BE DONE AUTOMATICALLY ON A DAILY BASIS The training should be triggered on a daily basis and be able to collect the necessary data from our econda API. > TRAINING SHOULD BE FAST AND COST EFFICIENT As a SaaS our products should be as cheap as possible for our clients. Despite that, the training should be still fast enough to start in the middle of the night and finish before morning. > INFERENCE SERVICE WHICH SHOULD BE ROBUST , FAST AND COST EFFICIENT Inference Service must be 24h available per day. Like training it should be as cheap as possible while allowing for predictions times <100ms.

Which aws services did we chose? The architecture is designed to perform an automated training on a daily basis for each of our clients and deploy the trained model as a live inference server. Training Storage Inference Secret Lambda Manager ECS ECR Real Time Personalisation Service econda Sagemaker API S3 Lambda Sagemaker

How to integrate the services? The architecture is designed to perform an automated training on a daily basis for each of our clients and deploy the trained model as a live inference server. 65 ms client’s aws econda webshop 20 ms webshop customer Rule Set Editor Application Load Balancer Personalisation ECS Service 95 ms

Year

The Model has changed New Input New Output Void NBO Categories » Includes classic categories, » Not every user is recognized brand, color . . . » Show new products of your » No more Topsellers favorite brand » Show banner depending on the category of interest

New use cases: Not only Widgets and Banners Offsite Onsite Personalized Search Display Marketing » Target groups » Change order of Search results based on relevance » Use NBO categories to determine Target Group » Can either be done on products (rearraning) or categories (boosting) » Use NBO Products to give » Real time scenario individual Recommendations » Near Real Time scenario ~10 min

One Year with Sagemaker: Lessons Learned > ARCHITECTURE DID NOT CHANGE Because we didn’t have any reason to change it. Up till now our lambdas did a good job of scaling our clients and the onboarding of clients is fast. > SAGEMAKER TRAINING JOBS ARE VERY STABLE Besides very few “Instance not available” Errors. The Service works without any worries, > LOOK OUT FOR MANAGED SPOT TRAINING Managed Spot training seems like a good way to reduce costs if you can wait longer for your training jobs to complete and if your algorithm supports checkpointing. > LAMBDA YOUR WAY AROUND AWS Lambdas work well with Sagemaker. If a desired feature is missing, often you can use Lambdas to implement them yourself.

Our Wishlist for the Future > SCHEDULED TRAINING JOBS Instead of triggering lambdas on a daily basis, scheduled training jobs which automatically start every day would be nice. They also could be grouped for better visibility. > ALARM WHEN TRAINING JOB FAILED Although there are alarms for a specific training job, there is no metric for all or a group of training jobs. > MAKE INFERENCE SECTION WORK WITH ECS/EKS Although Sagemaker use Container for Inference internally, the feature lacks the variety of ECS/EKS. Why not link them together? > TYPE HINTS FOR BOTO3 (PYTHON3) With type hints writing lambdas would be much easier. Boto3 has complex Parameters and Return types. Not having to look each one every time up would be a huge plus.

QUESTIONS? QUESTIONS? Dennis Weyland <weyland@econda.de> www.econda.de