TREC Video Retrieval Evaluation TRECVID 2012 Paul Over* Alan Smeaton (Dublin City University) George Awad# Wessel Kraaij (TNO, Radboud University Nijmegen) Lori Buckland* Darrin Dimmick* Georges Quénot (Laboratoire d’Informatique de Grenoble) Jonathan Fiscus** Stephanie Strassel+ David Joy** Amanda Morris+ Greg Sanders** et al Barbara Shaw** Martial Michel^ * Retrieval Group / ** Multimodal Information Group Information Access Division Information Technology Laboratory NIST ^ Systems Plus + Linguistic Data Consortium Rockville, MD # Dakota Consulting, Inc Silver Spring, MD
What is TRECVID? Workshop series (2001 – present) http://trecvid.nist.gov to promote research/progress in content-based video analysis/exploitation Foundation for large-scale laboratory testing Forum for the exchange of research ideas discussion of approaches – what works, what doesn’t , and why. Focus: content-based approaches search / detection / summarization / segmentation / … Aims for realistic system tasks and test collections unfiltered data focus on relatively high-level functionality (e.g. interactive search) measurement against human abilities Provides data, tasks, and uniform, appropriate scoring procedures 2 TRECVID 2012
TRECVID’s Evolution … 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 4500 4000 Flickr 3500 New development or test data 3000 as added HAVIC Data: 2500 (hours) Internet Archive Creative Commons 2000 1500 Airport English Sound & Surveillance 1000 TV News vision BBC rushes TV news 500 0 Ad hoc search ■■■■■■■■■■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ Features/semantic indexing ■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ Stories ■■■■■■■■■■■■■ Camera motion ■■ BBC rushes - - - - ■■■■■■■■■■■■■■■■ Summaries ■■■■■■■■■ Copy detection - - - - - - - - - - - - ■■■■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■■■■ Tasks: Surveillance events (SED) - - - - - - - - ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ Known-item search (KIS) - - - - - - - - - - - - - - - - ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ Instance search pilot (INS) - - - - - - - - - - - - - - - ■■■■■■■■■ ■■■■■■■■ ■■■■■■■■ Multimedia event detection (MED) - - - - - - - - - - - - - ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ Multimedia event recounting (MER) - - - - - - - - - - - - - - - - - - - ■■■■■■■■ 500 Rough 400 notebook 300 paper 200 author 100 count: 0 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 3 TRECVID 2012
TRECVID 2012 Tasks and Data Internet Archive – Flickr Airport HAVIC - Internet Creative Commons surveillance multimedia (IACC) 91 test video clips 50 hours, test video 3722 hours, test video 291 hours test data,video, title, divided into 74,958 keywords, description files 100 hours, dev video 1429 hours, dev video 687 hours dev video, metadata Known-item search Instance search Surveillance event Multimedia (361 text-only ad hoc test pilot detection event detection queries) (21 ad hoc image (interactive) (20 pre-specified & test queries) (7 known test 5 ad hoc complex events, choose 3) test events) Semantic indexing Multimedia (346 known test concepts, event recounting auto-assigned, submitted; pilot 46 evaluated; 10 concept- (5 events) pairs evaluated) 4 4
TV2012 Finishers PicSOM INS KIS --- --- --- SIN Aalto U. AXES INS KIS MED --- --- --- Acess to Audiovisual Archives: www.axes-project.eu ATTLabs INS --- --- --- --- *** AT&T Labs Research BUPT.MCPRL INS KIS *** --- SED *** Beijing U. of Posts and Telecommunications Bilkent INS --- --- --- --- *** Bilkent U. RETINA Vision and Learning Group INF --- --- *** *** SED SIN Carnegie Mellon U. CEALIST INS --- *** --- --- SIN CEA VIREO INS *** MED MER --- SIN City U. of Hong Kong CU *** --- MED MER --- --- Columbia U. PRISMA-Orand INS --- --- --- --- --- Department of Computer Science, U. of Chile. U_Tokushima INS --- --- --- --- *** U. of Tokushima DCU_IAD INS KIS MED MER --- --- Dublin City U., IAD dcu_savasa --- --- --- --- SED --- Dublin City U., CLARITY ECL_Liris *** *** *** --- --- SIN Ecole Centrale de Lyon, Universit de Lyon EURECOM --- --- *** --- --- SIN EURECOM - Multimedia Communications VideoSense --- --- --- --- --- SIN EURECOM VideoSense Consortium Brno *** --- --- --- SED *** Faculty of Information Technology, Brno U. of Technology FIU_UM --- --- --- --- --- SIN Florida International U. U. of Miami FTRDBJ INS --- --- --- --- SIN France Telecom Orange Labs (Beijing) MADM INS --- --- --- --- --- German Research Center for Artificial Intelligence IBM *** *** MED MER SED SIN IBM T. J. Watson Research Center ITI_CERTH *** KIS MED MER --- SIN Informatics and Telematics Institute, Greece Quaero *** --- *** --- --- SIN INRIA, IRIT, LIG, U. Karlsruhe ARTEMIS.Ubimedia INS --- --- --- --- --- Institut TELECOM; TELECOM SudParis; France Alcatel-Lucent BJTU_SED --- --- --- --- SED --- Inst. of Information Science,Beijing Jiaotong U. ECNU --- --- MED --- --- SIN Inst. of Computer Applications, East China Normal U. PKU_ICST INS KIS *** --- --- *** Inst. of Computer Science and Technology, Peking U. PKU_OS --- --- --- --- SED --- Inst. of Computer Science and Technology, Peking U. (OS) IRIM INS --- *** --- --- SIN Indexation et Recherche d'Information Multimédia GDR-ISIS JRS.VUT INS --- *** --- --- SIN JOANNEUM RESEARCH FmbH Vienna U. of Technology KBVR --- KIS *** --- --- --- KB Video Retrieval 5 TRECVID 2012
TV2012 Finishers GENIE --- --- MED MER --- --- Kitware Inc. ITEC_KLU --- KIS --- --- --- --- Klagenfurt U. kobe_muroran --- --- *** --- --- SIN Kobe U., Muroran Institute of Technology NII INS KIS MED --- *** SIN National Institute of Informatics NHKSTRL *** *** *** --- *** SIN NHK Science and Technical Research Labs NTT_NII INS --- MED *** --- --- NTT Communication Science Labs, Nat. Inst. of Informatics ntt --- --- --- --- --- SIN NTT Cyber Space Labs School of Software, Dalian U. IMP INS --- MED --- --- --- Osaka Prefecture U. PKU_NEC --- --- --- --- SED --- Peking U. and NEC Labs China SAIVT --- --- --- --- SED --- Queensland U. of Technology BBNVISER --- --- MED MER --- --- Raytheon BBN Technologies RMIT INS --- --- --- --- --- RMIT U. School of CS&IT IRC_Fuzhou --- *** --- --- --- SIN School of Mathematics and Computer Science Fuzhou U. SJTU_BCMI --- --- --- --- SED --- Shanghai Jiaotong U. Aurora --- --- MED MER --- --- SRI International Sarnoff Aurora SESAME --- --- MED MER --- --- SRI International SESAME stanford *** --- --- --- --- SIN Stanford U. MediaCCNY --- --- --- --- SED --- The City College of New York Media Team TNOM3 INS --- --- --- --- --- TNO TokyoTechCanon --- --- MED *** --- SIN Tokyo Institute of Technology and Canon GIM --- --- --- --- --- SIN U. of Extremadura MediaMill INS --- MED MER --- SIN U. of Amsterdam sheffield_harbin INS --- --- --- --- --- U. of Sheffield UCSB_UCR_VCG INS --- --- --- SED --- U. of California, Santa Barbara UEC *** *** MED --- --- SIN U. of Electro-Communications VIVA_uOttawa --- --- --- --- SED --- U. of Ottawa 6 TRECVID 2012
TV2012 Finishers Unique finishing teams 1 Groups Task Task 2 Finished code name Surveillance 19 13 SED 14 event detection Semantic 25 SIN indexing Known-item 9 KIS search Instance search 21 24 INS pilot Multimedia event Asia Europe 17 MED detection North America South America Multimedia event Australia 10 MER recounting 7 TRECVID 2012
Support The running of TRECVID 2012 has been funded directly by: National Institute of Standards and Technology (NIST) Intelligence Advanced Research Projects Activity (IARPA) TRECVID is only possible because of the additional efforts of many individuals and groups around the world. TRECVID 2012 8
Recommend
More recommend
