Predictive Video Retrieval A Matter of Trust Bouke Huurnink - PowerPoint PPT Presentation

Predictive Video Retrieval A Matter of Trust Bouke Huurnink MediaMill

The Team Bouke Huurnink Michiel van Liempt Jiyin He Richard van Balen Koen van de Sande FeiYan Ork de Rooij Muhammad Tahir Cees Snoek Krystian Mikolajczyk Maarten de Rijke Josef Kittler Jan van Gemert Jan-Mark Geusebroek Jasper Uijlings Theo Gevers Xirong Li Marcel Worring Ivo Everts Arnold Smeulders Vladimir Nedovic Dennis Koelma

Come see our interactive search demo 0.20 0.20 0.15 0.15 0.10 0.10 0.05 0.05 0 0 UvA Now with (inter)active learning! Presented by Ork de Rooij

Why predictive video retrieval? • Video retrieval is multichannel problem: • Speech • Detectors • Examples • Observations • Speech works for named entity topics • Detectors work when closely related to topic • Examples can also work pretty we ll • We want to exploit this knowledge

Idea • Predict which type of search - retrieval channel - we can trust for a topic • Rerank results from this channel with secondary result information

Outline • System description • Result overview • Analysis • Conclusion

Our predictive system Information Need Predict Result Lists Find shots of pieces Trusted Trusted of paper with writing, Channel typing, or printing, Results filling more than half - Detector results Retrieval Channels of the frame area. Speech Secondary Reranking Search Results - Example results Detector Search Secondary Results Example - Speech results Search Final Results

Our predictive system Information Need Predict Result Lists Find shots of pieces Trusted Trusted of paper with writing, Channel typing, or printing, Results filling more than half - Detector results Retrieval Channels of the frame area. Speech Secondary Reranking Search Results - Example results Detector Search Secondary Results Example - Speech results Search Final Results

Our predictive system Information Need Predict Result Lists Find shots of pieces Trusted Trusted of paper with writing, Channel typing, or printing, Results filling more than half - Detector results Retrieval Channels of the frame area. Speech Distribute ASR and MT over shot neighbourhood, Secondary Reranking Search then retrieval using language modelling approach Results - Example results Detector Content based selection from 57 learned concepts, Search followed by unweighted score-based fusion Secondary Results Example Pseudo active-learning, with positive examples from topic - Speech results Search and 100 random negative examples from collection Final Results

Our predictive system Information Need Predict Result Lists Find shots of pieces Trusted Trusted of paper with writing, Channel typing, or printing, Results filling more than half - Detector results Retrieval Channels of the frame area. Speech Secondary Reranking Search Results - Example results Detector Search Secondary Results Example - Speech results Search Final Results

Our predictive system Information Need Predict Named entity? Trust speech results Result Lists Find shots of pieces Trusted Detector match? Trust detector results Trusted of paper with writing, Else...trust example results Channel typing, or printing, Results filling more than half - Detector results Retrieval Channels of the frame area. Speech Secondary Reranking Search Results - Example results Detector Search Secondary Results Example - Speech results Search Final Results

Our predictive system Information Need Predict Result Lists Find shots of pieces Trusted Trusted of paper with writing, Channel typing, or printing, Results filling more than half - Detector results Retrieval Channels of the frame area. Speech Secondary Reranking Search Results - Example results Detector Search Secondary Results Example - Speech results Search Final Results

Our predictive system Information Need Predict Result Lists Find shots of pieces Trusted Trusted of paper with writing, Channel typing, or printing, Results filling more than half - Detector results Retrieval Channels of the frame area. Speech Secondary Reranking Search Results - Example results Detector Truncate result lists to top 1000 Search Eliminate all results not in trusted list Secondary Combine results with (weighted) Borda fusion Results Example - Speech results Search Final Results

Query-class vs Prediction Query-class Prediction Query class determines Query features determine retrieval strategy retrieval strategy Focus on assigning query- Focus on identifying trusted class dependent weights retrieval channel

Runs • Speech channel only UvA-MM-6 • Detector channel only UvA-MM-5 • Example channel only supplementary • Predictive reranking UvA-MM-4 • Predictive weighted reranking UvA-MM-3

Overall Automatic Search Performance Predictive Predictive weighted reranking mean inferred average precision 0.07 reranking 0.06 0.05 0.04 0.03 0.02 0.01 0 All runs Example channel Speech channel Detector channel

Overall Automatic Search Performance Predictive Predictive weighted reranking mean inferred average precision 0.07 reranking 0.06 0.05 0.04 0.03 0.02 0.01 0 All runs Example channel Speech channel Detector channel Predictive reranking outperforms individual channels

Overall Automatic Search Performance Predictive Predictive weighted reranking mean inferred average precision 0.07 reranking 0.06 0.05 0.04 0.03 0.02 0.01 0 All runs Example channel Speech channel Detector channel Predictive reranking Weighting did not have big outperforms individual channels influence

General findings • 20 topics > 0.05 inferred average precision • 1 speech topic • 11 detector topics • 8 example topics • Accurately predicted 15 of 20 topics

A closer look inferred average precision 0 0.1 0.2 0.3 0.4 0.5 person opening door a bridge people with trees and plants face filling over half the frame paper with writing people with a body of water a map vehicle moving away people looking in microscope person watching television people in a kitchen a crowd of people outdoors a classroom scene an airplane exterior a plant that is the main object a street scene at night people at table with computer Predictive w. reranking people in white lab coats Detector channel ships or boats in the water Example channel man talking to camera indoors Speech channel

A closer look inferred average precision 0 0.1 0.2 0.3 0.4 0.5 person opening door a bridge people with trees and plants face filling over half the frame paper with writing people with a body of water a map vehicle moving away people looking in microscope person watching television people in a kitchen A lot of variance a crowd of people outdoors between channels a classroom scene an airplane exterior a plant that is the main object a street scene at night people at table with computer Predictive w. reranking people in white lab coats Detector channel ships or boats in the water Example channel man talking to camera indoors Speech channel

When prediction worked inferred average precision 0 0.1 0.2 0.3 0.4 0.5 person opening door a bridge Only people with trees and plants trusted channel and reranked paper with writing performance shown a map people looking in microscope people in a kitchen a crowd of people outdoors a classroom scene an airplane exterior a plant that is the main object a street scene at night people at table with computer Predictive w. reranking people in white lab coats Detector channel Example channel man talking to camera indoors Speech channel

When prediction worked inferred average precision 0 0.1 0.2 0.3 0.4 0.5 person opening door a bridge Only people with trees and plants trusted channel and reranked paper with writing performance shown a map people looking in microscope people in a kitchen Predictive reranking often close a crowd of people outdoors a classroom scene to or better than trusted channel an airplane exterior a plant that is the main object a street scene at night people at table with computer Predictive w. reranking people in white lab coats Detector channel Example channel man talking to camera indoors Speech channel

When prediction didn’t work inferred average precision 0 0.1 0.2 0.3 0.4 0.5 Only trusted channel face filling over half the frame and reranked performance people with a body of water shown vehicle moving away person watching television Predictive w. reranking Detector channel ships or boats in the water Example channel man talking to camera indoors Speech channel

When prediction didn’t work inferred average precision 0 0.1 0.2 0.3 0.4 0.5 Only trusted channel face filling over half the frame and reranked performance people with a body of water shown vehicle moving away person watching television Predictive reranking boosts trusted channel results Predictive w. reranking Detector channel ships or boats in the water Example channel man talking to camera indoors Speech channel

Conclusions

Conclusions Predictive retrieval works, even with simple reranking

Conclusions Predictive retrieval works, even with simple reranking Incorrect predictions have limited impact