Consuming videos with the ForkBrowser Consuming videos with the - PowerPoint PPT Presentation

Consuming videos with the ForkBrowser Consuming videos with the ForkBrowser Ork de Rooij, Cees G.M. Snoek, Marcel Worring Ork de Rooij, Cees G.M. Snoek, Marcel Worring

2 Consuming Videos With The ForkBrowser 2 The Problem: Search in video The Problem: Search in video What is search?: enter query… What is search?: enter query…  browse results browse results  Query by keyword Query by region Automated selection Query by example Concept suggestion Leads to: Query by concept RANKING Query component ‘mixer’ Language filter … and search

3 Consuming Videos With The ForkBrowser 3 Mixing dimensions: CrossBrowser Mixing dimensions: CrossBrowser -Easy to understand - Time is very important - Very fast browsing time results Combination of query results and time Combination of query results and time

4 Consuming Videos With The ForkBrowser 4 Mixing dimensions: RotorBrowser Mixing dimensions: RotorBrowser - Doesn’t require the user to think of options. - Allows dataset exporation time - Browsing ‘outside query’ very important for some topics -Query screen visited only -once Initial query result Allows combinations based on query on demand Allows combinations based on query on demand

5 Consuming Videos With The ForkBrowser 5 Mixing dimensions Mixing dimensions Easy to understand Easy to understand Time is very important Time is very important Very fast browsing Very fast browsing Doesn’t require the user to think of options. Allows dataset exporation Browsing ‘outside query’ very important for some types of query Limits visits to a “query screen” Typically a user wants to explore a dataset fast and easy, without difficult query screens. A hybrid between both browsers is required

6 Consuming Videos With The ForkBrowser 6 The ForkBrowser The ForkBrowser results any query method any query method time history Combination of fixed set of query methods Combination of fixed set of query methods

7 Consuming Videos With The ForkBrowser 7 The ForkBrowser The ForkBrowser Combines Combines  Fast browsing through results Fast browsing through results  Assignable dimensions, e.g. Assignable dimensions, e.g. Visual similarity Visual similarity semantic similarity semantic similarity  User doesn’t have to revisit query screen User doesn’t have to revisit query screen  Animations on demand Animations on demand

8 Consuming Videos With The ForkBrowser 8 The ForkBrowser The ForkBrowser

9 Consuming Videos With The ForkBrowser 9 Experiments Experiments TRECVID 2007 Interactive Search TRECVID 2007 Interactive Search We compare: We compare: Run with CrossBrowser (UVA_MM_1) Run with CrossBrowser (UVA_MM_1)  Run with ForkBrowser (UVA_MM_2) Run with ForkBrowser (UVA_MM_2)  Evaluation metrics try to minimalize effect of comparing expert users Evaluation metrics try to minimalize effect of comparing expert users  Set up: Set up: Seed: Seed:  Automatic search results Automatic search results Query by concept, keyword and example Query by concept, keyword and example Extra ‘tines’ in ForkBrowser: Extra ‘tines’ in ForkBrowser:  Weibull and Gabor visual similarity features Weibull and Gabor visual similarity features What do we want to know? What do we want to know? Is browsing using multiple dimensions useful? Is browsing using multiple dimensions useful?  Does a fixed layout lead to faster browsing and better results? Does a fixed layout lead to faster browsing and better results? 

10 Consuming Videos With The ForkBrowser 10 Is browsing multiple dimensions useful? Is browsing multiple dimensions useful?  Evaluate effectiveness of having multiple Evaluate effectiveness of having multiple dimensions dimensions

11 Consuming Videos With The ForkBrowser 11 Query method usage per topic Query method usage per topic 0205 0212 0209 0213 0214 train in motion boat moving past 3+ people at table woman talking large crowd train in motion Helped by: Helped by: Helped by: Helped by: Helped by: Helped by: animation animation animation concept visual similarity concept time time time time concept Different topics visual similarity have different search strategies

12 Consuming Videos With The ForkBrowser 12 Unique results per browser Unique results per browser 300 250 200 150 100 50 0 0197 0198 0199 0200 0201 0202 0203 0204 0205 0206 0207 0208 0209 0210 0211 0212 0213 0214 0215 0216 0217 0218 0219 0220 Graph shows the number of (correct) shots retrieved Grey: shots found by both browser Both browsers find Red: shots found only by the CrossBrowser different results Green: shots found only by the ForkBrowser

13 Consuming Videos With The ForkBrowser 13 Movement vs Average Precision Movement vs Average Precision Movement to Average Precision ratio 0219 0219 1.0 Clear difference between 0.8 # of movement actions required 0205 0.6 0218 0218 0214 0214 0200 0.4 0212 A P v e a g e e c is io n r r 0215 0205 0208 0215 0211 0206 0209 0213 0203 0211 0207 0208 0220 0207 0217 0.2 0202 0206 0213 0216 0212 0216 0204 0204 0201 0197 0220 0200 0209 0217 0203 0199 0201 0198 0199 0197 0198 0202 0210 0210 0.0 0 1000 2000 3000 4000 5000 6000 7000 # of moves

14 Consuming Videos With The ForkBrowser 14 Conclusions Conclusions Evaluation: Evaluation:  Different combinations of query dimensions are beneficial Different combinations of query dimensions are beneficial for individual topics for individual topics  ForkBrowser requires less interaction steps from the user ForkBrowser requires less interaction steps from the user for the same average precision for the same average precision  Both browsers find different unique results Both browsers find different unique results

15 Consuming Videos With The ForkBrowser 15 Any questions? Any questions?