Bootstrapping Food Preferences Through an Adaptive Visual Interface - PowerPoint PPT Presentation

Bootstrapping Food Preferences Through an Adaptive Visual Interface Longqi Yang , Yin Cui, Fan Zhang, JP Pollak, Serge Belongie, Deborah Estrin

MOTIVATION Food preferences learning is important!

Health and Life Obesity 113M HBP 50M Unflavored Healthy diet recommendations are of NO 15M Diabetes Benefit! *Number of Americans Living with Diet-and Inactivity-Related Diseases

Social Media and Commerce Personalized diet profile is the Key to user experience!

Our Vision Clinicians Social Network Personalize Treatment Plan Content personalization Online Groceries Nutritionists Healthy recommendations Customers Targeting Personalized Diet Profile Recipes Restaurants Customized dishes Food environment at home

OUR SOLUTION An adaptive visual interface

Exploration, 2 iters Exploration-exploitation: <15 iters Diet Start Profile 10 food items Pairwise Comparison

ü Efficient: completed within a minute. ü Visual interface: low cognitive load, personalized and legible. ü Preference Elicitation: NO history required, NO ratings. ü Deep understanding of food images. ü Novel Online Learning Framework.

System Design Online Learning Visual User Interface …… Online Learning framework (LE + EE) Yuc uck Yuc uck online # Iterations Explo lora rati tion on (2 2 iter eratio tions) Explo lora rati tion on – Explo loita itati tion on (>2) Ø What images to present to the user? Backend Online Learning User Us er Pref efer eren ence Food od alread eady expl plor ored ed Ø How to update users’ preferences? Food Similarity Embedding Users have close preferences for similar items Food Similarity Embedding offline 1000 dim 200 dim Ø Feature representation that can reflect similarities Pre-trai Pr aine ned d Deep ep Siamese se Ne Networ ork Ingr gred edient ents Image raw pixels Metadata Food Items Harvesting Food Items Harvesting Ingredients Ø Food images and metadata. Nutrients ……

System Design: offline Visual User Interface …… Yuc uck Yuc uck online # Iterations Explo lora rati tion on (2 2 iter eratio tions) Explo lora rati tion on – Explo loita itati tion on (>2) Backend Online Learning User Us er Pref efer eren ence Food od alread eady expl plor ored ed Food Items Harvesting Food Similarity Embedding offline 1000 dim 200 dim Ø 12,000 food items from Yummly API. Pr Pre-trai aine ned d Deep ep Siamese se Ne Networ ork Ingr gred edient ents Ø Images + Metadata (ingredients, nutrients etc.) Image raw pixels Metadata Food Items Harvesting Ø Outliers filtering, 10,028 items were used. Ingredients Nutrients ……

System Design: offline Food Similarity Embedding Visual User Interface …… Representation: 1000 dim visual + 200 dim ingredients Yuc uck Yuc uck online # Iterations Explo lora rati tion on (2 2 iter eratio tions) Explo lora rati tion on – Explo loita itati tion on (>2) 1000 dim visual feature from Food-CNN Backend Online Learning Us User er Pref efer eren ence Food od alread eady expl plor ored ed Image 2 Contrastive Loss A (CNN) x f(x) Food Similarity Embedding Image 1 offline 1000 dim 200 dim B (CNN) Pre-trai Pr aine ned d Deep ep Siamese se Ne Networ ork Ingr gred edient ents y f(y) Image raw pixels Metadata Food Items Harvesting Ingredients Nutrients ……

System Design: offline Food Similarity Embedding Visual User Interface …… Representation: 1000 dim visual + 200 dim ingredients Yuc uck Yuc uck online # Iterations Explo lora rati tion on (2 2 iter eratio tions) Explo lora rati tion on – Explo loita itati tion on (>2) 1000 dim visual feature from Food-CNN Backend Online Learning Us User er Pref efer eren ence Food od alread eady expl plor ored ed 𝒎 = 𝟐 , − ≈ 0 𝓜 = 𝟐 𝟑𝒎𝑬 𝟑 + 𝟐 (𝟏, 𝒏 − 𝑬) 𝟑 𝟑 𝟐 − 𝒎 𝐧𝐛𝐲 Food Similarity Embedding offline 1000 dim 200 dim − > 𝑛 , Pr Pre-trai aine ned d Deep ep Siamese se Ne Networ ork Ingr gred edient ents 𝒎 = 𝟏 Image raw pixels Metadata Food Items Harvesting Pairs/Labels were sampled from Food-101 dataset Ingredients Nutrients ……

System Design: offline Food Similarity Embedding Visual User Interface …… Representation: 1000 dim visual + 200 dim ingredients Yuc uck Yuc uck online # Iterations Explo lora rati tion on (2 2 iter eratio tions) Explo lora rati tion on – Explo loita itati tion on (>2) 200 dim ingredients feature Backend Online Learning User Us er Pref efer eren ence Food od alread eady expl plor ored ed Ø Lemmatization and preprocessing . Ø Filtering: Top 200 ingredients. Ø Feature vector: 0-1 vector denotes the existence of the ingredient. Food Similarity Embedding Visual and ingredients feature vectors are normalized offline 1000 dim 200 dim separately with 𝒎 𝟐 norm Pr Pre-trai aine ned d Deep ep Siamese se Ne Networ ork Ingr gred edient ents Image raw pixels Metadata Food Items Harvesting Ingredients Nutrients ……

System Design: online Online Learning Visual User Interface …… Food preferences representation: Yuc uck Yuc uck online # Iterations Explo lora rati tion on (2 2 iter eratio tions) Explo lora rati tion on – Explo loita itati tion on (>2) Backend Online Learning t t t t t 𝒒 = 𝑞 8 , 𝑞 9 , … ,𝑞 𝒯 <𝑞 = = 1 Us User er Pref efer eren ence Food od alread eady expl plor ored ed = Distribution of preferences over all food items in 𝒯 𝒒 ? :updated preference vector after iteration t Two tasks at each iteration t: Food Similarity Embedding offline 1000 dim 200 dim Ø User state update: update 𝒒 ? based on the items Pr Pre-trai aine ned d Deep ep Siamese se Ne Networ ork Ingr gred edient ents Image raw pixels Metadata presented and user’s choices at iteration t-1 . Food Items Harvesting Ø Images selection: Select a set of images to show at Ingredients Nutrients iteration t . ……

System Design: online Online Learning Ø User state update: update 𝒒 ? based on the items presented and user’s choices at iteration t-1 . Image Labeling Users’ selections Images selected Label “+1” Label “-1” Images not selected Images not presented Label “0”

System Design: online Online Learning Ø User state update: update 𝒒 ? based on the items presented and user’s choices at iteration t-1 . Label propagation with regularized optimization 𝒯 𝒯 H H min < 𝜕 =E 𝑧 = − 𝑣 + < 1 − 𝜕 =E 𝑣 E − 𝑧 E E 𝒗 EI9,EJ= EI9,EJ= Fitting Smoothness L abel Propagation and E xponentiated Gradient Algorithm ( LE ) 𝒯 QRS NO P K9 HU V 𝒈 XP K𝒈 XY QRS ? 𝑣 E = < 𝜕 =E 𝑧 = ?K9 ×𝑓 T P 𝑞 = 𝑞 = 𝜕 =E = 𝑓 =I9

System Design: online Online Learning Ø Images selection: Select a set of images to show at iteration t . E xploration and E xploration-exploitation Algorithm ( EE ) Exploration (Ten images): 𝑢 ≤ 2 K-means++ Exploration-exploitation (Two images): 𝑢 > 2 One Item that user “prefer” (with high value of 𝑞 ) The other item that user hasn’t explored.

System Design: online Online Learning images user state selection update

EXPERIMENTS AND USER STUDY Evaluation, findings and evidence

Experiments: embedding Clustering performance of Food-CNN (Tested on Food-101 dataset ). Ø K -neighbors of each test image, calculate the precision-recall for each K 10 0 )RRG-C11 (PAP: 0.216) Alex1eW (PAP: 0.051) PreFLsLRn(LRgarLWhPLF 6Fale) 6I)T+BR: (PAP: 0.019) 5anGRP Guess (PAP: 0.01) 10 -1 10 -2 0.0 0.2 0.4 0.6 0.8 1.0 5eFall

Experiments: user study Ø 227 anonymous users. Ø Two factors were controlled in the study. 1 st . Algorithm: L abel Propagation and Exploration and Exploration- E xponentiated Gradient ( LE ) exploitation ( EE ) Online Perceptron ( OP ) Random Selection ( RS ) 2 nd . Number of iterations: 5/10/15

Experiments: user study Ø Algorithm to test: LE+EE Ø Trials: 1 /3 Exploration Exploration-exploitation PlateClick (10 iters) Testing (10 iters) One image from top 1% of preference value. ( unexplored ) The other image from bottom 1% of preference value. ( unexplored )

Experiments: user study Ø Algorithm to test: LE+EE Ø Trials: 2 /3 Exploration Exploration-exploitation PlateClick (5 iters) Testing (10 iters) One image from top 1% of preference value. ( unexplored ) The other image from bottom 1% of preference value. ( unexplored )

Experiments: user study Ø Algorithm to test: LE+EE Ø Trials: 3 /3 Exploration Exploration-exploitation PlateClick (15 iters) Testing (10 iters) One image from top 1% of preference value. ( unexplored ) The other image from bottom 1% of preference value. ( unexplored )

Experiments: user study Prediction accuracy under different algorithms and number of iterations ** *** * *** ** ** * *

Experiments: user study Cumulative distribution of prediction accuracy for LE+EE algorithm 1.0 LE+EE:5 CuPulDtLve DLstrLEutLon LE+EE:10 0.8 LE+EE:15 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0 PredLctLon AccurDcy

Conclusions and Future work Ø Engine for food preferences learning. Ø Applicable to general human-in-the-loop problems.