FXPAL Interactive Search for TRECVID 2004 John Adcock, Matthew - PowerPoint PPT Presentation

FXPAL Interactive Search for TRECVID 2004 John Adcock, Matthew Cooper, Andreas Girgensohn, Lynn Wilcox

Overview • First time doing search – 2 nd year of participation overall • Emphasis on interface elements – Rich visualization of search results – Quick and easy exploration of results • Straightforward search engine – Text search over ASR transcripts • Literal search with Lucene • Fuzzy search with LSS – Keyframe search by image similarity • Color correlograms 2 FX Palo alto Laboratory Inc @ trecvid 2004

Preprocessing Unit of search retrieval is a “story”, but we couldn’t don’t have reference story segmentation for the test set • Group reference shots into “stories” – Bootstrap an LSS with common shot boundaries and ASR – use similarity-matrix method to find “story” boundaries • Given new story boundaries – Generate text indices for story and shots – Generate story-based LSS for search 3 FX Palo alto Laboratory Inc @ trecvid 2004

Preprocessing Common ASR Shot Ref Lucene Index (shots) Lucene Index Bootstrap (stories) LSS (shots) LS Index Similarity (shots) Segmentation LS Index (stories) Story Segments 4 FX Palo alto Laboratory Inc @ trecvid 2004

Search Engine • User specifies combination of: – Text query • Literal query using Lucene or fuzzy query using LSS – Image examples • Any keyframe in the interface can be dragged onto the image example area – Text/image weighting is static and equal – Max image similarity of shot propagated to story – Text similarity of story propagated to shot • Averaged with shot-based text similarity 5 FX Palo alto Laboratory Inc @ trecvid 2004

Search Engine Searcher option Lucene Search Query text LSS Search Ranked Combine Stories Image Color Correlogram Query Images Search 6 FX Palo alto Laboratory Inc @ trecvid 2004

Interface Elements • Stories summarized in keyframe “quads” • Navigate through stories to video timeline/shots • Transparent icon overlays – Visited: grayed – Relevant: green – Irrelevant:red • Query-relevance shown with size and color • Hotkeys for most actions • Multi-select and drag and drop 7 FX Palo alto Laboratory Inc @ trecvid 2004

Media player Relevant shots area Query results area and zoom area Selected story Gray visited Video timeline overlay Expanded shots area Text query box Image query box Excluded overlay Trecvid topic Included overlay text Text search type Trecvid topic images 8 FX Palo alto Laboratory Inc @ trecvid 2004

Story Summary Quads • Query-dependent story summary – Use 4 highest scoring shots in the story – Allocate space proportional to score Story thumbnail Shot thumbnails 9 FX Palo alto Laboratory Inc @ trecvid 2004

Building on searches • Find similar • Add related – Use shot/story text – Auto re-query with for search existing results 10 FX Palo alto Laboratory Inc @ trecvid 2004

Expanded Story / Timeline Browsing • Selecting a story expands the video at that point – Clickable video timeline with relevancy shading – Clickable story quad timeline – Shot thumbs marked with relevancy – Overlay on shots marked (non)relevant – Mouse-overs zoom in the media player and tool-tip shows relevancy context – Double clicks play video in the media player 11 FX Palo alto Laboratory Inc @ trecvid 2004

Experiments • 6 searchers answering 12 topics each in latin square – Pairs of orthogonal users grouped together • Each topic answered 3 times – Searchers include 2 primary developers • 1 ended up in best and 1 in worst performing group • Each of the 3 complete searcher runs goes through 3 “systems” or methods for filling out the shot list yielding 9 total submissions 12 FX Palo alto Laboratory Inc @ trecvid 2004

System Types • Type 1: re-issue user queries and weight results of each query by precision against the user-labeled shots • Type 2: take text from all relevant shots and issue a single new LSS-based text query • Type 3: take text from each relevant shot in turn for LSS-based query and apply query ranking as in system type 1 Shots marked as not-relevant excluded from system results Every system type preceded by bracketing the user- retrieved shots 13 FX Palo alto Laboratory Inc @ trecvid 2004

Submissions User IDed Shots + Bracketed Shots + System1 System2 System3 (Weighted) (LSA1) (LSA2) 14 FX Palo alto Laboratory Inc @ trecvid 2004

Results • Ranks 3-6, 9-13 in overall MAP – Strongly user dependent (user groups clump together) – Post-processing methods perform nearly same 0.4 FXPal submissions 0.35 Other contributors 0.3 0.25 MAP 0.2 0.15 0.1 2 3 1 0.05 0 I_A_1_AL_2_5 I_A_1_AL_1_4 I_A_1_AL_3_6 I_A_1_AL_1_7 I_A_1_AL_2_8 I_A_1_AL_3_9 I_A_1_AL_1_1 I_A_1_AL_2_2 I_A_1_AL_3_3 Submissions 15 FX Palo alto Laboratory Inc @ trecvid 2004

User vs. System System Summary 0.35 0.3 0.25 WEIGHTED 0.2 MAP LSA1 0.15 LSA2 Bracketed 0.1 None 0.05 0 Group 1 Group 2 Group 3 User Group 16 FX Palo alto Laboratory Inc @ trecvid 2004

MAP 0.05 0.15 0.25 0.35 0.1 0.2 0.3 0.4 0 I_A_1_AL_2_5 User vs. System in Overall I_A_1_AL_1_4 I_A_1_AL_3_6 fxpal_2_bracketed I_A_1_AL_1_7 I_A_1_AL_2_8 I_A_1_AL_3_9 I_A_1_AL_1_1 I_A_1_AL_2_2 I_A_1_AL_3_3 fxpal_2_users fxpal_1_bracketed fxpal_3_bracketed FX Palo alto Laboratory Inc @ trecvid 2004 fxpal_3_users fxpal_1_users Submission Complete submission Other contributors User selected only With bracketing 17

MAP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0 1 people on steps or stairs Performance by Question pedestrians and vehicles bicycles rolling people moving a stretcher umbrellas Overall max FXPal average Overall median fingers striking keyboard buildings on fire FX Palo alto Laboratory Inc @ trecvid 2004 handheld weapon firing golf ball into the hole Bill Clinton tennis player contacting ball horses in motion people and dogs wheelchairs signs at a protest zooming in US Capitol dome Benjamin Netanyahu buildings with flood waters Henry Hyde Saddam Hussein. Sam Donaldson hockey rink Boris Yeltsin 18

Directions • More sophisticated: – Story segmentation – Image similarity / video features • Simplify user interface for non power-users and more typical search and re-use tasks • Handle multiple simultaneous media streams – Presentation slides – Multi-camera capture 19 FX Palo alto Laboratory Inc @ trecvid 2004