TRECVID-2006: Search Task Alan Smeaton Dublin City University & Tzveta Ianeva NIST
Search Task Definition Goal: promote progress in content-based retrieval from digital video via open, metrics-based evaluation; Given a test collection, a topic and a common shot boundary reference, return a ranked list of at most 1,000 shots which best satisfy the need; NIST created more topics asking for general (vs. specific) NIST created 10 of 24 topics to ask for video of events – encouraging exploration beyond one-keyframe-per-shot Videos were viewed by NIST personnel, notes taken on content, and candidates emerging were chosen; TRECVID 2006 2
Search Task Definition Per-search measures: average precision, elapsed time Per-run measure: mean average precision (MAP) Interactive search participants were asked to have their subjects complete pre, post-topic and post- search questionnaires; Each result for a topic can come from only 1 user search; same searcher does not need to be used for all topics. TRECVID 2006 3
Search Task Definition Bing Xiang, John Makhoul, and Ralph Weischedel at BBN for providing MT/ASR Christian Petersohn (Fraunhofer Institute) for master shot reference DCU team for formatting and selecting keyframes MediaMill team for 101 features baseline results donation CMU and IBM for 449 LSCOM features annotations TRECVID 2006 4
Data characteristics TRECVid 2006 data is again (deliberately) text- noisy with video from English language, Arabic & Chinese broadcasts; 32.2% of the test video comes from programs not represented in the development data Text is derived from speech recognition and then machine translation, thus poorer quality than with English-only sources but ASR/MT from “state-of- the-art” GALE system. TRECVID 2006 5
2006: Search task participants (26, up from 20) AT AT&T T Lab abs – – Re Resea earch ch US USA Be Beiji jing g Jia iaoto tong g U. China Ch na Bi Bilke kent t U. Tu Turke key Ca Carne negie ie Me Mello lon U U. USA US Ch Chine nese e U. . of f Hon ong K Kong ng Ch China na Ci City y Uni niver ersit ity o of H Hong ng Ko Kong China Ch na CLIPS CL PS-IM IMAG Franc Fr nce Co Colum umbia ia U. U. US USA Du Dubli lin C City ty U. U. Ir Irela land d Fu Fudan an U. U. Ch China na FX FX Pa Palo o Alt lto L Labo borat atory ry In Inc US USA He Helsi sinki ki U. U. of of Te Techn hnolo logy Fi Finla land IB IBM T T. J J. W Wats tson n Res esear arch h Cen enter er US USA Im Imper erial al Co Colle lege e Lon ondon on / / Joh ohns s Hop opkin ins U U. UK, U UK USA TRECVID 2006 6
2006: Search task participants (continued) NU NUS / / I2 I2R Si Singa gapor ore Media Me iamil ill / / U. U. of of Am Amste terda dam Ne Nethe herla lands ds RM RMIT T U. . Sch chool ol of of CS CS&IT IT Au Austr trali lia Ts Tsing nghua ua U. U. Ch China na U. U. of of Ce Centr tral l Flo lorid ida US USA U. U. of of Gl Glasg sgow w / U U. o of S Shef effie ield UK UK U. of U. of Io Iowa US USA U. U. of of Ox Oxfor ord UK UK U. Re U. Rey J Juan an Ca Carlo los Spain Sp in Zh Zheji jiang ng U. U. Ch China na CO COST2 T292 2 (ww www.c .cost st292 92.or org) Fr Franc nce, , Net ether erlan ands, s, UK UK, I Irel eland nd, G Gree eece, e, Tu Turke key, , Se Serbi bia a and d Mon onten enegr gro, , Slo lovak akia K- K-Spa pace e (ks kspac ace.q .qmul ul.ne net) UK, G UK Germ rmany ny, A Aust stria ia, G Gree eece, e, Irela Ir land, d, Ne Nethe herla lands ds, F Fran ance, e, Sw Switz tzerl rland nd, C Czec echia ia TRECVID 2006 7
Search Types: Automatic, Manual and Interactive Number of runs: 76 automatic 11 manually assisted 36 interactive TRECVID 2006 8
Everybody likes to search automatically, dislikes manually 100% 80% Interactive 60% Manual Fully automatic 40% 20% 0% 2004 2005 2006 TRECVID 2006 9
24 Topics [ number of image, video examples and relevant found] 173. Find shots with a view of one or more tall buildings (more than 4 stories) and the top story visible [3, 4, 142] 174. Find shots with one or more people leaving or entering a vehicle [0, 10, 675] 175. Find shots with one or more soldiers, police, or guards escorting a prisoner [0, 4, 204] 176. Find shots of a daytime demonstration or protest with at least part of one building visible [4, 4, 111] 177. Find shots of US Vice President Dick Cheney [3, 3, 393] 178. Find shots of Saddam Hussein with at least one other person's face at least partially visible [8, 0, 99] 179. Find shots of multiple people in uniform and in formation [3, 5, 191] 180. Find shots of US President George W. Bush, Jr. walking [0, 5, 197] TRECVID 2006 10
24 Topics [ number of image, video examples and relevant found] 181. Find shots of one or more soldiers or police with one or more weapons and military vehicles [2, 6, 128] 182. Find shots of water with one or more boats or ships [3, 5, 307] 183. Find shots with one or more emergency vehicles in motion (e.g., ambulance, police car, fire truck, etc.) [0, 4, 299] 184. Find shots of one or more people seated at a computer with display visible [3, 4, 440] 185. Find shots of one or more people reading a newspaper [3, 4, 201] 186. Find shots of a natural scene - with, for example, fields, trees, sky, lake, mountain, rocks, rivers, beach, ocean, grass, sunset, waterfall, animals, or people; but no buildings, no roads, no vehicles [2, 4, 523] 187. Find shots of one or more helicopters in flight [0, 6, 119] TRECVID 2006 11
24 Topics [ number of image, video examples and relevant found] 188. Find shots of something burning with flames visible [3, 5, 375] 189. Find shots of a group including at least four people dressed in suits, seated, and with at least one flag [3, 5, 446] 190. Find shots of at least one person and at least 10 books [3, 5, 295] 191. Find shots containing at least one adult person and at least one child [3, 6, 775] 192. Find shots of a greeting by at least one kiss on the cheek [0, 5, 98] 193. Find shots of one or more smokestacks, chimneys, or cooling towers with smoke or vapor coming out [3, 2, 60] 194. Find shots of Condoleezza Rice [3, 7, 122] 195. Find shots of one or more soccer goalposts [3, 4, 333] 196. Find shots of scenes with snow [3, 6, 692] TRECVID 2006 12
Some statistics 2006: Number of shots in test collection: 79.484 79.484 ~9.1% ~9.1% relevant shots found: 7.225 7.225 2005 Number of shots in test collection: 45.765 45.765 ~18.3% relevant shots found: 8.395 ~18.3% 8.395 2004 Number of shots in test collection: 33.367 33.367 ~5.4% ~5.4% relevant shots found: 1.800 1.800 2003 Number of shots in test collection: 32.318 32.318 ~6.5% relevant shots found: 2.114 ~6.5% 2.114 TRECVID 2006 13
2006: 20 sites contributed one or more unique, relevant shots 70 Number of unique, relevant shots 60 50 40 30 20 10 0 G U a s m . g t g . 2 n e d y . M a . e U U U U n M o c d w o r n A n 9 r g a e B o l u o n 2 d a o a i w o d g M C o r r k f I p T p o a t K a n l u I r n l x o o i I S s o C e a d o h l f a B - O g a n F o t g S g K i u O L g s n s j i h n P n F f n l . e J C a m e o c a U a i o h I i S g L r l u g e s G A H . t Z n C e U T T n J f i l . o f e U j l f y o o i o C e e . C / U . R B . U f U m o l l . a a e U a i . n i s k U e r o e n e T p i i n t s l a i m i h l m N e I C H a / i d R e 2 M I TRECVID 2006 14
2006: Rel shots contrib. uniquely per topic by team 9 28 14 8 12 19 6 5 11 16 8 5 7 7 1 4 10 2 4 6 3 1 5 4 2 2 1 8 2 2 2 1 2 3 2 Number of 7 3 1 2 2 1 1 2 1 1 2 6 1 4 unique 1 2 2 2 1 1 2 1 2 1 1 1 true 2 3 1 4 1 2 7 1 1 1 1 2 1 1 1 3 1 2 5 1 shots 1 1 1 1 1 1 1 3 1 1 2 2 1 9 1 3 1 1 2 1 5 1 2 1 1 9 1 1 ( 1 1 3 1 2 1 3 1 1 3 0 9 ( 3 1 1 6 1 ) 0 1 8 1 1 1 ( 1 ) 9 7 g 3 1 n 1 7 t 8 n 1 o 1 1 2 3 ( 5 t e G 7 4 a k A 1 ) i U l 1 4 J i M 1 M 8 B g ( g 6 I C n . 5 1 n _ U 1 ) S o 1 i 2 j 1 i P K n 8 e 9 9 a ( I 2 n B L g 3 2 d o ) T e 1 C n u S d c 0 ( o F d 1 8 O n a H 2 1 p r Topic (total relevant) o 1 C o y L s 9 ) f f g 1 o . K x U e o 9 O 7 l M ( . g ) U o a 9 1 e f B n u a e l o 1 2 l h h I d o . s . c g i U 7 8 e C U r ( a e n o n w 7 1 ) T i l w i l a s F s 9 h o o 1 f T o C i g r o l I m 7 1 e a l ( s r . f 3 p r o a a . 5 ) U t a U C d m n l 9 G . 1 i U r e e g 3 I k n e 7 C r ( n t n a u s a 3 2 ) Group i u p s f m o i 0 J j a l e e A . g 4 H U y h ( e n . Z 1 ) U i R S 4 / . U f 2 m o ) a . e U 186, 191, 196 have 500+ T l a l i n m o a i t i a d N e M / R 2 I TRECVID 2006 15
Recommend
More recommend
