Inferring Restaurant Styles by Mining Crowd Sourced Photos from - PowerPoint PPT Presentation

@IEEE BigData 2016 Inferring Restaurant Styles by Mining Crowd Sourced Photos from User-Review Websites Haofu Liao, Yuncheng Li, Tianran Hu and Jiebo Luo Department of Computer Science UNIVERSITY of ROCHESTER

Part I - The Problem UNIVERSITY of ROCHESTER

Restaurant Style Classification Business meetings Romantic Restaurant Photos UNIVERSITY of ROCHESTER

Multi-Instance Multi-Label Learning (MIML) Instance object object Instance Label Instance Label ... Instance (a) Traditional Supervised (b) Multi-instance learning learning Instance Label Label object object Label Instance Label Instance ... ... ... Instance Label Label (d) Multi-instance multi-label learning (c) Multi-label learning UNIVERSITY of ROCHESTER

Multi-Instance Multi-Label Learning (MIML) ● Each restaurant (object) is described by a set of photos (instances) and associated with several class restaurant style tags (label). Business Photo 1 Restaurant ● Conventionally, MIML is based on the meetings assumption that there exists a “key” Special Photo 2 instance that contributes the object’s Occasion ... ... class labels. Photo N Romantic ● For restaurant style classification, such assumption is not guaranteed. For example, just one picture of a delicate dish does not mean the restaurant itself is romantic UNIVERSITY of ROCHESTER

Part II - Solution UNIVERSITY of ROCHESTER

Proposed Architecture Label Multi-Label Pseudo Initialization CNN Tagging SVM Business meetings SVM Restaurant Multi-Label ... Profiling CNN Romantic SVM UNIVERSITY of ROCHESTER

Overall Architecture Label Multi-Label Pseudo Initialization CNN Tagging SVM Business meetings SVM Restaurant Multi-Lacbel ... Profiling CNN Romantic SVM UNIVERSITY of ROCHESTER

Multi-Label CNN CONV4 ● Sigmoid Cross Entropy Loss POOL3 CONV5 CONV3 POOL5 AlexNet POOL2 FC6 CONV2 FC7 CONV1 FC8 Sigmoid Data Cross Entropy Label Loss UNIVERSITY of ROCHESTER

Overall Architecture Label Multi-Label Pseudo Initialization CNN Tagging SVM Business meetings SVM Restaurant Multi-Label ... Profiling CNN Romantic SVM UNIVERSITY of ROCHESTER

Pseudo Tagging Photo 1 Photo 2 Photo 3 Photo 4 Photo 5 Photo 6 Photo 7 Photo 8 Images Scores of tag k 7.1 -5.4 0.9 -1.1 9.5 -3.7 3.2 -0.2 Images from restaurants with tag k Images from restaurants without tag k UNIVERSITY of ROCHESTER

Pseudo Tagging Steps: 1. Order images according their scores Photo 5 Photo 1 Photo 7 Photo 3 Photo 8 Photo 4 Photo 6 Photo 2 9.5 7.1 3.2 0.9 -0.2 -1.1 -3.7 -5.4 Ordered UNIVERSITY of ROCHESTER

Pseudo Tagging Steps: 1. Order images according their scores Photo 5 Photo 1 Photo 1 Photo 7 Photo 3 Photo 8 Photo 4 Photo 6 Photo 2 2. Add tag k to the image that has the highest score among the images that do not have tag k. 9.5 7.1 3.2 0.9 -0.2 -1.1 -3.7 -5.4 Ordered UNIVERSITY of ROCHESTER

Pseudo Tagging Steps: 1. Order images according their scores Photo 5 Photo 1 Photo 1 Photo 7 Photo 3 Photo 8 Photo 4 Photo 6 Photo 2 Photo 2 2. Add tag k to the image that has the highest score among the images that do not have tag k. 3. Drop tag k from the image that has 9.5 7.1 3.2 0.9 -0.2 -1.1 -3.7 -5.4 the lowest score among the images that have tag k. Ordered UNIVERSITY of ROCHESTER

Pseudo Tagging Steps: 1. Order images according their scores Photo 5 Photo 1 Photo 1 Photo 7 Photo 3 Photo 8 Photo 4 Photo 6 Photo 2 Photo 2 2. Add tag k to the image that has the highest score among the images that do not have tag k. 3. Drop tag k from the image that has 9.5 7.1 3.2 0.9 -0.2 -1.1 -3.7 -5.4 the lowest score among the images that have tag k. Ordered UNIVERSITY of ROCHESTER

Pseudo Tagging Steps: 1. Order images according their scores Photo 5 Photo 1 Photo 1 Photo 7 Photo 3 Photo 8 Photo 4 Photo 6 Photo 2 Photo 2 2. Add tag k to the image that has the highest score among the images that do not have tag k. 3. Drop tag k from the image that has 9.5 7.1 3.2 0.9 -0.2 -1.1 -3.7 -5.4 the lowest score among the images that have tag k. Ordered 4. Repeat step 3 & step 4 until the scores of images reach a predefined threshold UNIVERSITY of ROCHESTER

Overall Architecture Label Multi-Label Pseudo Initialization CNN Tagging SVM Business meetings SVM Restaurant Multi-Label ... Profiling CNN Romantic SVM UNIVERSITY of ROCHESTER

Restaurant Profiling Dining on Multi-Label CNN a budget Features SVM UNIVERSITY of ROCHESTER

Part III - Experimental Results UNIVERSITY of ROCHESTER

Top Scored Images UNIVERSITY of ROCHESTER

Distribution of Image Scores 1st Round Multi-Label CNN 2nd Round Multi-Label CNN UNIVERSITY of ROCHESTER

Overall Performance UNIVERSITY of ROCHESTER

F-Measure @m UNIVERSITY of ROCHESTER

Part IV - Questions? UNIVERSITY of ROCHESTER

Thank You! UNIVERSITY of ROCHESTER