Analyzing large multimedia collections in an urban context MSc. VU - PDF document

12-7-2016 Analyzing large multimedia collections in an urban context MSc. VU computer Science Amsterdam Data Science PhD: UvA Informatics Institute Now: 0.8fte Informatics Institute Marcel Worring 0.2fte Amsterdam Business School Marcel Worring Associate Director Amsterdam Data Science Marcel Worring Stevan Rudinac, Jan Zahalka, Dennis Koelma Joost Boonzajer Flaes, Jorrit van den Berg Informatics Institute, Amsterdam Data Science Amsterdam Data Science Objective and Subjective data Data Sources Author .,. josemanuelerre (Flickr) .,. Jose ´ Manuel R´ ıos V aliente Tags .,. cyclist .,. bike Open Data .,. street Open data Comments Structured data Geo location .,. “I love Amsterdam! Numeric data Temporal data .,. Amsterdam, Netherlands great photo!” .,. “Great compostion, Exif beautiful B&W!!” Geographic data .,. Camera: Nikon N60 .,. “Estupendo B&N, bella Image data .,. Focal length: 55 mm imagen.” Textual data .,. Exposure time: 1/200 Unstructured data . . . .,. Flash: off 1

12-7-2016 Data Sources Objective and Subjective data .,. “Koningsdag, or ‘King’s Day ,’ is one of the principal holidays of the Netherlands. . . ” Open Data Open data + Content Analysis .,. In this case, the image says more than the text Photo: quantz @ Flickr Professional Recommender Systems WHAT DOES IS BRING? Recommender system for tourists Touristic Routing 11 2

12-7-2016 City Sentiment City Marketing Analytics Ranking of data Result Some query defines starting point and order Best ALGORITHMS Worse An image/video/text collection For Social Media Concept detection • The Ranking can be based on Visual examples – The objective content of the comments Positive negative – The subjective content of the comments – The objective visual content – The subjective visual content Unknown images Learn model Score of presence -> ranking – ……… • Or any combination of the above 3

12-7-2016 Requires annotation to learn Animals Zebu Lions Lemurs People What do we learn? The new trend: Deep learning 14,197,122 images, 21841 synsets indexed Start with raw pixels, learn all parameters 1200 trained visual concept detectors for adjective-noun pairs Krishevsky NIPS 2012 The learned filters The layered network Convolution + pooling + fully connected layers + output layers 60.000.000 parameters to learn But what do all these layers do? Krishevsky NIPS 2012 Zeiler and Fergus 4

12-7-2016 Zeiler and Fergus Visualizing deep networks Visualizing deep networks Visualizing deep networks Visualizing deep networks State-of-the-art: GoogleNet Text Analysis Latent Dirichlet Allocation and growing …… Latent Dirichlet Allocation D. Blei, 2003 Makes image search keyword driven 5

12-7-2016 Latent Dirichlet Allocation Latent Dirichlet Allocation .,. Generative model , discovers topics and scores them .,. 100 topics are enough to sufficiently cover entire Wikipedia .,. Input : Raw text .,. Output : T opic scores per document 0.054*mexico + 0.049*forest + 0.024*argentina + 0.022*islands + ...+ 0.014*aires We treat comments or sets tags as documents D. Blei, 2003 The task .,. Venue recommendation — suggesting places of interest ( venues ) based on user preferences VENUE RECOMMENDER .,. The classic approach is collaborative filtering utilizing the user-item matrix City Melange Characteristics Data Gathering .,. City Melange — a venue explorer utilizing multimedia analytics techniques Q(venue name,geo) .,. Content-based — based solely on the content of venue-related social media .,. Multimodal — combining content from images and the associated text .,. Interactive — user preferences are modelled on the fly Venue information Images, metadata as you explore the city Venue images User data .,. Cross Platform — integrates data from diverse social platforms 6

12-7-2016 Data Analysis Data Analysis Content Content Features V F V V Images Images ConvNet T F T T T ags T ags LDA V C V C Comments U Comments U . . . . . . Venues Users Venues Users Data Analysis Data Analysis Processed data Processed data V Visual venue V Visual venue V T V T Content Features Clustering Content Features Clustering topics topics U Visual user V T topics V F V F V V Images Images ConvNet ConvNet T F T F T T T ags T ags LDA LDA Comments U Comments V C V C U . . . . . . Venues Users Venues Users Data Analysis Data Analysis Processed data Processed data V Visual venue V Visual venue Clustering V T Clustering V T Content Features Content Features topics topics U U Visual user Visual user V T V T topics topics V F V F V V V V T ext venue T ext venue Images Images ConvNet T ConvNet T T T topics topics U T T F T T F T ext user T T topics T ags LDA T ags LDA Comments V C Comments V C U U . . . . . . Venues Users Venues Users 7

12-7-2016 Data Analysis Processed data V Visual venue Clustering V T Content Features topics U Visual user V T topics V V F V T ext venue Images ConvNet T T topics U T F T T ext user T T topics T ags LDA User-venue Comments V C U U V .,. ACM Multimedia Grand Challenge 2014 1st Prize . . . matrix Venues Users .,. newyorkermelange.com Interactive Recommendation Interactive Recommendation User-venue User-venue Users U Users U U V U V matrix matrix Venue topics Venue topics V V V V V T , T T V T , T T User topics User topics U U U U V T , T T V T , T T Grid Interactive Recommendation Interactive Recommendation User-venue User-venue Users U U V Users U U V matrix matrix Venue topics Positives Venue topics Positives V V + + V V + + V T , T T V T , T T V T , T T V T , T T Rel. Rel. venues venues User topics User topics U U U U V T , T T V T , T T Grid Grid User ranking 8

12-7-2016 Interactive Recommendation Interactive Recommendation User-venue User-venue Users U U V Users U U V matrix matrix Venue topics Positives Venue topics Positives + + + + V V V V V T , T T V T , T T V T , T T V T , T T Rel. Rel. Suggested venues venues users Linear U S SVM User User topics User topics Rand. Rand. ranking V − , T − V − , T − U U U U V T , T T sample V T , T T sample T T T T Negatives Negatives Grid User ranking Grid User ranking Interactive Recommendation Interactive Recommendation User-venue User-venue Users U Users U U V U V matrix matrix Venue topics Venue topics Positives Positives + + + + V V V V V T , T T V T , T T V T , T T V T , T T Rel. Suggested Rel. Suggested Suggested users users venues venues venues Linear Linear V S U S U S SVM SVM Venue User User User topics User topics Rand. Rand. ranking ranking ranking V − , T − V − , T − U U U U V T , T T sample V T , T T sample T T T T Negatives Negatives User ranking Venue ranking User ranking Venue ranking Grid Grid Interactive Recommendation Interactive Recommendation User-venue User-venue Users U Users U U V U V matrix matrix Venue topics Positives Venue topics Positives V V + + V V + + V T , T T V T , T T V T , T T V T , T T Rel. Suggested Suggested Rel. Suggested Suggested venues users venues Suggestions venues users venues Suggestions Linear U S V S ( U S ,V S ) Linear U S V S ( U S ,V S ) SVM SVM User Venue User Venue User topics User topics Rand. Rand. ranking ranking ranking ranking V − , T − V − , T − U U U U V T , T T sample V T , T T sample T T T T Negatives Negatives Grid User ranking Venue ranking Grid User ranking Venue ranking Map 9

12-7-2016 Interactive Recommendation Recommender system for tourists User-venue Users U U V matrix Venue topics Positives + + V V V T , T T V T , T T Rel. Suggested Suggested venues users venues Suggestions Linear ( U S ,V S ) U S V S SVM User Venue User topics Rand. ranking ranking V − , T − U U V T , T T sample T T Relevance Negatives indication Grid User ranking Venue ranking Map 56 Evaluation Evaluation .,. 621 fine-grained venue types (Japanese restaurant, 1. Can we recommend the right type of venue ? skate park. . . ) 2. Can we recommend mainstream venues to mainstream .,. 100 artificial actors , use 75% of the data to seed Melange tourists and specialized venues to afficionados ? .,. Perform 10 interaction rounds Methods Compared Data Collection • .,. City Melange .., Visual modality only • ext modality only • .., T .,. New Y ork — 1.07M images and associated text from .., Multimedia (vis + txt) Foursquare, Flickr, and Picasa • .,. Amsterdam — 56K images and associated text from • .,. Recommender baselines Foursquare and Flickr .., WRMF — Weighted regularized matrix factorization • .., BPRMF — Bayesian personalized ranking matrix • factorization • .,. Popularity ranking ( PopRank ) — most visited venues according to Foursquare 10