Beyond Classification: La Latent User Interests Profiling from Visual Contents Analysis Lon Longqi Ya Yang , Cheng-Kang (Andy) Hsieh, Deborah Estrin
Communicati Co tions On Online P Purchases So Social Network
Ou Our inte terests ts are manifeste ted online … Po Posted/Shared Contents Pe People Connected/Followed It Items s Purch chase sed
Pr Preferences learning using small data Sh Shared Images On Online Posts Pr Preference Pr Profile Pe Personal Im Image Pr Private Ga Gallery Communication Co Ne News Se Search Engine Di Dietary En Entertainment
Te Text/label-ce centric c ap approac ach is widely studied River restaurant tourism landscape Classification/Labeling/Image-to-text Topic Modeling Structure Prediction An Animal Tr Travel Ar Art
Bu But preferences are sometimes not just about text... User A Travel Tr Im Images User B Intra-ca In categorica cal variance ce: Hard to to captu ture with th te text/ t/label!
Res esea earch ques estion Im Images’ predictive power for users’ preferences be beyond d labe bels Task 1: Pairwise Comparison Task 2: Prediction
Pa Pairwise Comparison Us User A …... IMG 1 IMG 2 IMG n Di Discr scriminative Sa Same Po Power of images La Label IMG 1 IMG 2 …... IMG m Us User B
Pr Prediction: Consistency of Pr Preferences Predict/Retrieve Pr Us User 1 …... …... IMG 1 IMG 2 IMG n Sa Same La Label …... …... User N Us IMG 1 IMG 2 IMG m Timeline Ti
Da Datase set Tr Travel bo boards ds ❶ ≥ 100 pins ❷ ∃ pins after June 2014 5,790 5, 790 Tr Travel bo boards ds 1,800 1, 800 3, 3,990 990 Background corpus Ba Anal An alysis
User Modeling and Image Representation Us Pretrained Siamese Network dim 205 410 dim … … dim 205 * Pretrained Places CNN User Profile Pretrained cluster 200 dim pins centers (200) * B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. “Learning Deep Features for Scene Recognition using Places Database.” Advances in Neural Information Processing Systems 27 (NIPS), 2014
Us User Modeling and Image Representation Image 2 Contrastive Loss A (CNN) x f(x) Image 1 B (CNN) y f(y) 𝒎 = 𝟐 , − ≈ 0 𝓜 = 𝟐 𝟑𝒎𝑬 𝟑 + 𝟐 (𝟏, 𝒏 − 𝑬) 𝟑 𝟑 𝟐 − 𝒎 𝐧𝐛𝐲 − > 𝑛 , 𝒎 = 𝟏
Pa Pairwise Comparison Us User A …... IMG 1 IMG 2 IMG n Travel Tr Images Im …... IMG 1 IMG 2 IMG m User B Us Effects of ba Ef backgr ground d di distribu bution!
Pa Pairwise Comparison Do Document 1 “a “and” ” 10% “f “fatuous” ” 0.001% 1% 1% Do Document 2 “and” “a ” 11% “f “fatuous” ” 1.001% Ba Background “a “and” ” 11% “f “fatuous” ” 0.001%
Pa Pairwise Comparison Ba Background Us User B Us User A co corpus :;< :;< ) 𝜏 > (𝜀 9 𝜀 9 Log Log-od odds-ra ratio Un Uncertainty :;< 𝜀 9 :;< = 𝑨 9 :;< ) 𝜏 > (𝜀 9
Pa Pairwise Comparison Confidence)Level:) 99% Confidence)Level:) 95% Fo For all user pairs among 3,990 boards 𝑩;𝑪 𝐧𝐛𝐲 𝒜 𝒍
Pr Prediction 10~ 10~50 50 pins for train 50 50 pins for test …... …... User 1 Us IMG 1 IMG 2 IMG 51 IMG 100 …... …... Us User N IMG 1 IMG 2 IMG 51 IMG 100 Sampled 100 pins Sa Timeline Ti
Pr Prediction 𝑶 𝑵𝑺𝑺 = 𝟐 𝟐 𝑶G 𝒔𝒃𝒐𝒍 𝒋 𝒋L𝟐
Co Conclusion Sh Shared Images On Online Posts Preference Pr Pr Profile Personal Im Pe Image Private Pr Ga Gallery Communication Co Sm Small data fueled preferences learning – wh what can we we do next? v Utilities of images beyond text/labels. v Multi-modal data fusion v End-to-end learning
Fo For more information ht http:// ://www www.cs.cornell.edu/~ /~yl ylongqi ht http:// ://sm smalldata.io/ ylongqi@c @cs.cornell.edu @yl ylongqi
Recommend
More recommend
