Twitter Cortex Twitter Timelines Quality ML at Twitter: A Deep Dive into Twitter’s Timeline Cibele Montez Halasz, Twitter Cortex Satanjeev “Bano” Banerjee, Twitter Timelines Quality O’Reilly AI Conference April 2019
Agenda 1 ML at Twitter 2 Timelines Product 3 Modeling and Serving Pipeline 4 Model 5 Timelines on Tensorflow 6 Lessons Learned and Conclusions 2
ML at Twitter 3
ML at Twitter Personalized Search 4
ML at Twitter Recommendations 5
ML at Twitter Health 6
ML at Twitter Ads 7
ML at Twitter ML Data ML Platform Tools Observability Cortex ML Core ML Feature Environment (TF Management based platform) ML Extended ML Modeling 8
Timelines Product 9
Timelines Product Twitter “Timeline” Tweets shown to users when they log in and ask for Tweets in the Home section of the app Similar to Facebook News Feed, Instagram Home, etc. 10
Timelines Product ‘Reverse Chron’ Timeline 1m 1 minute old 5m 5 minutes old 9m 9 minutes old 13m 13 minutes old 29m 29 minutes old 11
Timelines Product Problems of Reverse Chron - The most recent Tweet may not be the most interesting Tweet - The most interesting Tweets for you may be - From hours ago - From people who tweet infrequently - From people you don’t even follow - Some Tweets may be bad: e.g. abusive, or near-duplicate content 12
Timelines Product Goal of Timelines Product Show each user the Tweets they are most likely to be interested in 13
Timelines Product User interests —> User engagement Engagement == Likes / replies / retweets / ... Timelines Product’s Goal: Show users Tweets that they are more likely to engage with 14
Timelines Product Relevance Ranked Timeline 29m 29 minutes old 5m 5 minutes old 13m 13 minutes old 1m 1 minutes old 9m 9 minutes old 15
Timelines Product ML problem: Given a Tweet and a user, predict the user’s probability of engaging with that Tweet 16
Modeling and Serving Pipeline 17
Modeling and Serving Pipeline Optional caption copy here CLIENT 18
Modeling and Serving Pipeline CLIENT request Tweets TWEET SERVICE 19
Modeling and Serving Pipeline CLIENT request Tweets fetch Tweets request TWEET CANDIDATE SERVICE TWEETS 20
Modeling and Serving Pipeline CLIENT request Tweets fetch Tweets request TWEET CANDIDATE SERVICE TWEETS Tweets 21
Modeling and Serving Pipeline CLIENT request Tweets fetch Tweets request TWEET CANDIDATE SERVICE TWEETS Tweets t s e u q e r s t e e w T e r o c s PREDICTION SERVICE 22
Modeling and Serving Pipeline CLIENT request Tweets fetch Tweets request TWEET CANDIDATE SERVICE TWEETS Tweets t s e u Tweets with scores q e r s t e e w T e r o c s PREDICTION SERVICE 23
Modeling and Serving Pipeline CLIENT request Tweets fetch Tweets request TWEET CANDIDATE SERVICE TWEETS Tweets sort, threshold, return Tweets to user t s e u Tweets with scores q e r s t e e w T e r o c s PREDICTION SERVICE 24
Modeling and Serving Pipeline CLIENT request Tweets fetch Tweets request TWEET CANDIDATE SERVICE TWEETS Tweets sort, threshold, return Tweets to user t s e u Tweets with scores q e r s t e e w T e r o c save as future s training examples PREDICTION SERVICE TRAINING DATA 25
Modeling and Serving Pipeline CLIENT request Tweets fetch Tweets request TWEET CANDIDATE SERVICE TWEETS Tweets sort, threshold, return Tweets to user t s e u Tweets with scores q e r s t e e w T e r o c save as future s training examples PREDICTION SERVICE TRAINING DATA 26
Modeling and Serving Pipeline CLIENT request Tweets fetch Tweets request TWEET CANDIDATE SERVICE TWEETS Tweets sort, threshold, return Tweets to user t s e u Tweets with scores q e r s t e e w T e r o c save as future s training examples PREDICTION SERVICE TRAINING DATA 27
Modeling and Serving Pipeline CLIENT request Tweets fetch Tweets request TWEET CANDIDATE SERVICE TWEETS Tweets sort, threshold, return Tweets to user t s e u Tweets with scores q e r s t e e w T e r o c save as future s training examples PREDICTION SERVICE TRAINING DATA save engagements as labels 28
Modeling and Serving Pipeline CLIENT request Tweets fetch Tweets request TWEET CANDIDATE SERVICE TWEETS Tweets sort, threshold, return Tweets to user t s e u Tweets with scores q e r s t e e w T e r o c save as future s training examples PREDICTION SERVICE retrain engagement TRAINING DATA save engagements models as labels 29
Modeling and Serving Pipeline 126 M Total Daily Monetizable Daily Active Users Source: Twitter 2018 Q4 Earnings Report 30
Modeling and Serving Pipeline Modeling Engagements 31
Modeling and Serving Pipeline Modeling Engagements 32
Modeling and Serving Pipeline Modeling Engagements 33
Modeling and Serving Pipeline Modeling Engagements 34
Modeling and Serving Pipeline Modeling Engagements 35
Modeling and Serving Pipeline Modeling Engagements 36
Modeling and Serving Pipeline Modeling Engagements 37
Modeling and Serving Pipeline Combining Separate Engagement Scores per Tweet Currently we combine the models with learned weights ➔ score = x × P(like) + y × P(retweet) + z × P(reply) + … Additional score adjustments ➔ Photo penalty ◆ Author diversity penalty ◆ Producer-side value boost ◆ Lots of experiments to tune these parameters ➔ 38
Modeling and Serving Pipeline Predictive Features for Timelines Ranking Tweet features ➔ # User’s network features ➔ User historical features ➔ Request features ➔ Text features ➔ 39
Model 40
The Model Timelines Model 1 2 3 4 Percentile Sparse Layer Dense MLP Isotonic Discretizer Calibration 41
The Model Timelines Model: Percentile Discretization Discretized feature values Raw feature values 42
The Model Timelines Model 1 2 3 4 Percentile Sparse Layer Dense MLP Isotonic Discretizer Calibration 43
The Model Timelines Model: Sparse Linear Layer K 2 ... 1 V1 V2 V Vn-2 Vn-1 44
The Model Timelines Model 1 2 3 4 Percentile Sparse Layer Dense MLP Isotonic Discretizer Calibration 45
The Model Timelines Model 46
The Model Timelines Model 1 2 3 4 Percentile Sparse Layer Dense MLP Probability Discretizer Calibration 47
The Model Timelines Model: Skewed Dataset Very few positive examples of likes, replies, retweets vs negative ● examples (Tweets that were shown to the user, but the user did not like, reply, retweet, etc). Solution: ● Heavily downsample negative examples ○ Train without example weights ○ Then apply probability calibration in the end ○ 48
The Model Sparse Linear Layer: Probability Calibration Sparse Linear Layer: Probability Calibration Goal: Bring predicted CTR back in line with data CTR 49
The Model Sparse Linear Layer: Probability Calibration predictions 0.8, 0.5, 0.6, 0.7, 0.8 MLP weights (multilayer 1, 2 , 1, 2, 3 perceptron) labels 1, 0 , 1, 1, 1 50
The Model Sparse Linear Layer: Probability Calibration predictions sorted predictions 0.8, 0.5, 0.6, 0.7, 0.8 0.5, 0.6, 0.7, 0.8, 0.8 MLP weights sorted weights (multilayer 1, 2 , 1, 2, 3 2 , 1, 2, 3, 1 perceptron) labels sorted labels 1, 0 , 1, 1, 1 0 , 1, 1, 1, 1 51
The Model Sparse Linear Layer: Probability Calibration predictions sorted predictions 0.8, 0.5, 0.6, 0.7, 0.8 0.5, 0.6, 0.7, 0.8, 0.8 MLP weights sorted weights (multilayer 1, 2 , 1, 2, 3 2 , 1, 2, 3, 1 perceptron) labels sorted labels 1, 0 , 1, 1, 1 0 , 1, 1, 1, 1 52
The Model Sparse Linear Layer: Probability Calibration predictions sorted predictions 0.8, 0.5, 0.6, 0.7, 0.8 0.5, 0.6, 0.7, 0.8, 0.8 MLP weights sorted weights (multilayer 1, 2 , 1, 2, 3 2 , 1, 2, 3, 1 perceptron) labels sorted labels 1, 0 , 1, 1, 1 0 , 1, 1, 1, 1 binning 53
The Model Sparse Linear Layer: Probability Calibration predictions sorted predictions 0.8, 0.5, 0.6, 0.7, 0.8 0.5, 0.6, 0.7, 0.8, 0.8 MLP weights sorted weights (multilayer 1, 2 , 1, 2, 3 2 , 1, 2, 3, 1 perceptron) labels sorted labels 1, 0 , 1, 1, 1 0 , 1, 1, 1, 1 Compute average binning prediction per bin 54
The Model Sparse Linear Layer: Probability Calibration: Closer Look Source: Scikit-learn 55
Timelines on Tensorflow 56
Timelines on Tensorflow Timelines Model and Tensorflow Hub Probability MLP (multilayer Calibration perceptron) 57
Timelines on Tensorflow Tensorflow: Profiler 58
Timelines on Tensorflow Tensorflow: Tensorflow Model Analysis 59 Source: Tensorflow Medium
Timelines on Tensorflow Tensorflow: Tensorboard 60
Lessons Learned, Conclusions, Next Steps 61
Recommend
More recommend
