Beat Googles bidder using ML Dolead RD team Arnaud Fouchet DOLEAD - PowerPoint PPT Presentation

Beat Google’s bidder using ML Dolead RD team Arnaud Fouchet DOLEAD @ Py.Paris –NOV 2018 WWW.DOLEAD.COM

Paid search basics 2 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics 3 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Click 4 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Click Visit Landing page 5 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Click Visit Landing page Subscription (victory) Not interested (defeat) 6 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Typed: “BTS a distance” Impression Will he click ? Click Will he subscribe ? Subscription New user 7 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Typed: “BTS a distance” Impression Will he click ? Click Will he subscribe ? Subscription 100$ question : how much should I pay the ad ? New user 8 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Typed: “BTS a distance” Impression Google’s problem Will he click ? Click Will he subscribe ? Subscription 100$ question : how much should I pay the ad ? New user 9 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Typed: “BTS a distance” Impression Google’s problem Will he click ? Click Will he subscribe ? Subscription 10$ question : how much should I pay the click ? New user 10 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics How much should I pay the click ? 1. Find revenue by click 11 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics How much should I pay the click ? 1. Find revenue by click 2. Pay less than that 12 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics How much should I pay the click ? 1. Find revenue by click (using Machine learning: search query → revenue by click) 2. Pay less than that 13 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Typed: “BTS a distance” Impression Google’s problem Will he click ? Click Google’s bidder Will he subscribe ? Subscription 10$ question : how much should I pay the click ? New user 14 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Typed: “BTS a distance” Impression Google’s problem Will he click ? Click Google’s bidder Will he subscribe ? Subscription 1$ for 2 questions : Do I accept giving Google all power ? New user Can we do better ? 15 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Typed: “BTS a distance” Impression Google’s problem Will he click ? Click Google’s bidder Will he convert ? Conversion 1$ for 2 questions : Do I accept giving Google all power ? New user Can we do better ? CPA: Cost Per Acquisition (Cost of a conversion) 16 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Basic ML steps Identify learnable outcome (revenue by click) ● Transform input ( user’s search query ) into features (X) ● Identify relevant loss function ● Try and optimize models ● 17 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Our ML research 18 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Our ML research 19 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Our ML research 20 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Consensus Log RF GBDT ... Reg 21 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Our ML research 22 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Neural networks 23 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Typed: “BTS a distance” Impression Google’s problem Will he click ? Click Will he convert ? Conversion 10$ question : how much should I pay the click ? New user 24 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Our ML research 25 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Paid search basics Typed: “BTS a distance” Impression Google’s problem Will he click ? Click Will he convert ? Conversion 10$ question : how much should I pay the click ? New user 26 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Our ML research 27 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Our ML research 28 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Consensus RF GBDT NN ... Better 29 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Consensus RF GBDT NN ... Better Weighted Consensus b x a x RF c x NN ... GBDT 30 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Consensus RF GBDT NN ... Better Weighted Consensus b x a x RF c x NN ... GBDT Learned model f RF GBDT NN ... 31 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Stacked models x GBDT NN RF ... x’ Learning problem on (x’, y) 32 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Stacked models x ➝ data leakage problem GBDT NN RF ... Train x’ Learning problem on (x’, y) 33 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Stacked models x ➝ data leakage problem DEVIL GBDT NN RF Train x’ Learning problem on (x’, y) 34 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Measure Stacked Models Perf Train Train Campaign ... Campaign 1 Campaign 2 Predictions Predictions GBDT GBDT NN NN RF RF Test Test ... ... 35 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Measure Stacked Models Perf Train Train Campaign ... Campaign 1 Campaign 2 Predictions Predictions GBDT GBDT NN NN RF RF Test Test ... ... Test ! Campaign T+ ... Train Train Campaign T+1 Campaign T+2 Predictions Predictions GBDT GBDT NN RF NN RF Test Test ... ... 36 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Our ML research 37 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Bidder comparison: context A/B test Experiment 1: 50% bids Google Campaign Experiment 2: 50% bids Dolead 38 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Bidder comparison: context A/B test 50 campaigns 4 weeks Goal : maximize conversions while respecting target CPA Experiment 1: 50% bids Google Campaign Experiment 2: 50% bids Dolead 39 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Bidder comparison: % Below target CPA Raw results : - Google 33% - Dolead 63% Distribution CPA(Dolead) / target CPA(Google) / target CPA CPA 40 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Bidder comparison: Conversions & CPA Conversions : In 67% of campaigns, Google had more convs Distribution Convs(Google) / Convs(Dolead) 41 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Bidder comparison: Conversions & CPA Conversions : CPA : In 67% of campaigns, Google had more convs In 83% of campaigns, Dolead was cheaper Distribution Distribution Convs(Google) / Convs(Dolead) CPA(Google) / CPA(Dolead) 42 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Bidder comparison: Conversions & CPA Conversions : CPA : In 67% of campaigns, Google had more convs In 83% of campaigns, Dolead was cheaper Distribution Distribution Google’s margin: 9.8K Dolead’s margin: 13.3K (~+35%) Convs(Google) / Convs(Dolead) CPA(Dolead) / CPA(Google) 43 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM

Thanks for your attention ! PyApero 13 décember 2018 www.pyapero.com DOLEAD @ Py.Paris –NOV 2018 WWW.DOLEAD.COM