CERTH at MediaEval Placing Task 2013 Giorgos Kordopatis 1 , Symeon - PowerPoint PPT Presentation

CERTH at MediaEval Placing Task 2013 Giorgos Kordopatis 1 , Symeon Papadopoulos 2 , Eleftherios Spyromitros-Xioufis 2 , Andreas L. Symeonidis 2 and Yiannis Kompatsiaris 2 1 Electrical Engineering Dept., Aristotle University of Thessaloniki, Greece 2 Information Technologies Institute, CERTH, Thessaloniki, Greece

Summary of our participation • 2 tag-based runs – Build upon the method of (Van Laere et al., ICMR ‘11) and make use of a two-level LDA scheme to filter non-(geo)-discriminative tags • 2 visual runs – A nearest neighbor scheme using VLAD+SURF features (Spyromitros- Xioufis et al., WIAMIS ‘12) and an efficient indexing method for very fast retrieval • 1 hybrid run – Uses a simple fallback scheme to combine a tag-based and a visual run • No external gazetteers or Internet data! 2

Placing using tags – Training (1/3) Training Prediction Spatial Clustering Global LDA Assignment Similarity Filtering & local LDAs & BoEW in Areas Search • Removal of noisy and irrelevant tags, e.g. “ geolat =… ” 3

Placing using tags – Training (2/3) Training Prediction Spatial Clustering Global LDA Assignment Similarity Filtering & local LDAs & BoEW in Areas Search • 𝑙 -means clustering is applied on the lat-long values of the training images to create 𝑙 =5000 cluster-areas • A local topic model is created for the images of each area using LDA (100/20) 4

Placing using tags – Training (3/3) Training Prediction Spatial Clustering Global LDA Assignment Similarity Filtering & local LDAs & BoEW in Areas Search • A global topic model is created for all training images using LDA (500/50) • For each global topic, a topic-area histogram is computed by counting its frequency per area • Top terms of the highest entropy topics are added in the bag-of-excluded words ( BoEW ) 5

Placing using tags – Prediction (1/2) Training Prediction Spatial Clustering Global LDA Assignment Similarity Filtering & local LDAs & BoEW in Areas Search • Noisy or BoEW tags are filtered and Jaccard similarity between the image’s tags and each local topic is computed • Each test image 𝑈 𝑗 is assigned to the most similar area: – Area containing the highest similarity topic ( tmax ) – Area with the highest average (across topics) similarity ( tmean ) 6

Placing using tags – Prediction (2/2) Training Prediction Spatial Clustering Global LDA Assignment Similarity Filtering & local LDAs & BoEW in Areas Search • Having assigned the image to an area: – 𝑙 most similar images within the area are identified – their center of gravity is used as location estimate 7

Placing using visual features • We extract optimized VLAD+SURF vectors (Jegou and Chum, ECCV 2012) for all training images • Search efficiently with Product Quantization (Jegou et al., TPAMI 2011) – 14 bytes per image! – 13 hours for querying the index with the full test set! • 1 st variant – vnn : Assign the location of the most similar image • 2 nd variant – vclust : Top 𝑙 =20 nearest neighbors are considered and an incremental spatial clustering algorithm is applied 8

Runs and Results Measure tmax vnn tmean vclust hyb acc(1km) 10.26 0.60 7.82 0.76 10.37 acc(10km) 23.53 0.99 19.86 1.16 23.70 Median error 651 6715 1028 6691 681 • Tag-based runs: tmax and tmean • Visual runs: vnn and vclust • Hybrid run: hyb , if there is at least one clean tag, tmax is used, else vnn is used 9

Future work • A more thorough analysis on the different sources of error for the proposed scheme – # of geographical areas – # of topics and terms per topic used in the local and the global LDAs – type and quality of visual features • Extension to include additional metadata and external resources, e.g. the author of an image as an indicator of the image location 10

The end This work was supported by the SocialSensor FP7 project 11