TREC Video Retrieval Evaluation TRECVID 2019 Ian Soboroff* Alan - PowerPoint PPT Presentation

TREC Video Retrieval Evaluation TRECVID 2019 Ian Soboroff* Alan Smeaton, Yvette Graham (Dublin City University, Ireland) George Awad#* Wessel Kraaij (TNO, Leiden University, Netherland) Asad Butt^* Georges Quénot (Laboratoire d’Informatique de Grenoble, France) Keith Curtis* Stephanie Strassel+ Angela Ellis* Darrin Dimmick* Jonathan Fiscus** Afzal Godil** Yooyoung Lee** Andrew Delgado** Eliot Godard** Lukas Diduch** * Retrieval Group / ** Multimodal Information Group Information Access Division, Information Technology Laboratory, NIST + Linguistic Data Consortium # Georgetown University ^ John Hopkins University TRECVID 2019 1

What is TRECVID ? • Workshop series (2001 – present) à http://trecvid.nist.gov • Started as a track in the TREC (Text REtrieval Conference) evaluation benchmark. • Became an independent evaluation benchmark since 2003. • Focus: content-based video analysis, retrieval, detection, etc • Provides data, tasks, and uniform, appropriate scoring procedures • Aims for realistic system tasks and test collections: • unfiltered data • focus on relatively high-level functionality (e.g. interactive search) • measurement against human abilities • Forum for the • exchange of research ideas and for • the discussion of research methodology – what works, what doesn’t , and why TRECVID 2019 2

TRECVID Datasets overview Security cameras (since 2008) Social media (since 2016) HAVIC Soap opera (since 2013) Video Sharing (since 2019) TRECVID 2019 3

Notebook unique author count by years 500 400 300 200 100 0 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 TRECVID 2019 4

TRECVID Bibliography Partial bibliography of peer-reviewed journal and conference papers (mostly from ACM Digital Library and IEEE Explorer ) based on TRECVID resources 140 120 119 100 97 97 94 87 Paper count 80 81 75 72 67 60 59 54 52 40 42 39 31 20 18 1 10 0 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 http://www-nlpir.nist.gov/projects/trecvid/trecvid.bibliography.txt TRECVID 2019 5

TRECVID 2019 Tasks and Data Overview Ad-hoc Instance Video to Text Activities in Video Search Search Description Extended Videos Vimeo Creative Commons BBC Eastenders Twitter Vines + VIRAT Collection* (V3C1 subset) Flicker Videos 1000 h test video data divided 464 h divided into 471,527 1044 Vine test videos URLs 10 h video surveillance into ~1 M shots, title, keywords, shots, transcripts 1010 Flicker videos dataset description ~2000 hours dev 1 episode dev video and any ~5700 Vine dev URLs & Detection and video, metadata external web resources corresponding descriptions Localization of activities and objects 30 ad-hoc textual queries 30 ad-hoc image/video Match video with the evaluated Person + Action description 18 activities evaluated queries evaluated & 20 progress queries Generate automatic video (fixed 2019-2021) 20 progress queries description (fixed 2019-2021) * New dataset TRECVID 2019 6

TV2019 Finishers Groups Task Task Unique finishing teams Finished code name 1 Activities in 3 7 ActEV Extended Video Ad-hoc Video 3 10 AVS Search Instance search 6 INS 15 Video to Text 10 VTT 5 Description Asia Europe North America Australia South America TRECVID 2019 7

TV2019 Finishers : 27 teams out of 39 EURECOM --- VTT ----- AVS Eur EURECOM FDU --- VTT ----- --- Asia Fudan University KU_ISPL --- VTT ----- --- Asia Korea University MUDSML --- *** ActEv *** Aus Monash University PicSOM --- VTT ----- --- Eur Aalto University PKU_ICST INS *** ----- --- Asia Peking University SIRET --- --- ----- AVS Eur Charles University UTS_ISA --- VTT ----- --- Aus Centre for Artificial Intelligence, University of Technology Sydney Insight_DCU --- VTT ----- --- Eur Insight Dublin City University WasedaMeiseiSoftbank --- *** ----- AVS Asia Waseda University; Meisei University; SoftBank Corporation BUPT_MCPRL INS --- ActEv --- Asia Beijing University of Posts and Telecommunications KsLab --- VTT ----- --- Asia Nagaoka University of Technology RUC_AIM3 --- VTT ----- --- Asia Renmin University of China RUCMM --- VTT ----- AVS Asia Renmin University of China; Zhejiang Gongshang University VIREO --- --- ActEv AVS Asia City University of Hong Kong WHU_NERCMS INS --- ----- --- Asia National Engineering Research Center for Multimedia Software UCF --- --- ActEv --- Nam University of Central Florida FraunhoferIOSB --- --- ActEv --- Eur Fraunhofer IOSB and Karlsruhe Institute of Technology (KIT) MKLab --- --- ActEv --- Asia Information Technologies Institute, Centre for Research and Technology Hella --- : Didn’t plan to participate *** : Planned to submit but didn’t TRECVID 2019 8

TV2019 Finishers : 27 teams out of 39 FIU_UM --- --- ----- AVS NAm Florida International University; University of Miami ATL --- *** ----- AVS Asia Alibaba group, ZheJiang University Inf INS *** ***** AVS NAm+Asia+Aus Monash University; Renmin University; Shandong University IMFD_IMPRESEE --- VTT ----- AVS NAm+SAm Millennium Institute Foundational Research on Data (IMFD); Chile Impresee Inc ORAND S.A. Chile kindai_kobe --- --- ***** AVS Asia Department of Informatics, Kindai University; Graduate School of System, Informatics, Kobe University NTTandCQUPT --- --- ActEv --- Asia NTT Media Intelligence Laboratories; Chongqing University of Posts and Telecommunications NII_Hitachi_UIT INS *** ActEv *** Asia National Institute of Informatics, Japan; Hitachi, Ltd University of Information Technology, VNU-HCM HSMW_TUC INS --- ***** --- Eur University of Applied Sciences Mittweida; Chemnitz University of Technology JRS *** --- ----- *** Eur JOANNEUM RESEARCH MediaMill --- *** ***** *** Eur University of Amsterdam IOACAS *** --- ----- --- Asia University of Chinese Academy of Sciences D_A777 *** --- ----- *** Asia Malla Reddy College of Engineering Technology, Department of Electronics and communication Engineering --- : Didn’t plan to participate *** : Planned to submit but didn’t TRECVID 2019 9

TV2019 Finishers : 27 teams out of 39 Arete --- *** ***** --- NAm Scientific Computing Data Analytics Image Processing and Computer Vision GDGCV --- *** ----- --- Asia G D Goenka University MAGUS_ITAI.Wing --- *** ----- --- Asia Nanjing University ITAI TokyoTech_AIST --- --- ***** *** Asia Tokyo Institute of Technology, National Institute of Advanced Industrial Science and Technology TeamCRN *** --- ***** *** NAm+Asia Microsoft Research Singapore; Management University, University of Washington USF --- --- ***** --- NAm University of South Florida, USF MIAOTEAM *** --- ----- *** Aus University of Technology Sydney MET --- --- ----- *** Asia Sun Yet-sen University --- : Didn’t plan to participate *** : Planned to submit but didn’t TRECVID 2019 10

Support The running of TRECVID 2019 has been funded directly by: § National Institute of Standards and Technology (NIST) § Intelligence Advanced Research Projects Activity (IARPA) § Defense Advanced Research Projects Agency (DARPA) TRECVID is only possible because of the additional efforts of many individuals and groups around the world. TRECVID 2019 11

Additional resources and contributions § Luca Rossetto and Heiko Schuldt (University of Basel) § Provided the V3C dataset, master shot reference and metadata § Georges Quénot § provided the master shot reference for the IACC.3 video § The LIMSI Spoken Language Processing Group and Vocapia Research provided ASR for the IACC.2-3 videos and HAVIC data § Koichi Shinoda of the TokyoTechCanon team agreed to host a copy of the IACC.2 data § Noel O'Connor and Kevin McGuinness at Dublin City University along with Robin Aly at the University of Twente worked with NIST and Andy O’Dwyer plus Rob Cooper at the BBC to make the BBC EastEnders video available for use in TRECVID § Alan Smeaton and Yvette Graham (DCU, Ir) for supporting the video-to-text Direct Assessment annotations TRECVID 2019 12

Agenda: Day 1 § Arranged by task § Time for discussion of approaches & evaluation § Tuesday § Intros, thanks, etc. § Ad hoc Video Search (AVS) task § Lunch § Instance Search (INS) task § Poster Session § Workshop dinner TRECVID 2019 13

Agenda: Day 2 § Wednesday § Activities in Extended Video (ActEv) task § Lunch § Video to Text (VTT) task § TRECVID planning § Workshop close TRECVID 2019 14

Miscellany • If you have registered for the workshop dinner and will not be using your ticket, please turn it in at the registration desk during the break this morning or at lunch so someone else who wants to attend can use it. • Breakfast, morning and afternoon coffee breaks: Green foyer (Next to the registration desk) • Wifi: NIST-Guest The online workshop proceeding can be found at : • http://www-nlpir.nist.gov/projects/tv2019/tv19.workshop.notebook/ • Also offline version is available on the USB flash drive TRECVID 2019 15