TREC 2003 Video Retrieval Evaluation Overview Coordinators: Alan - PowerPoint PPT Presentation

Frame-recall & -precision for GTs �� !�� "� �� #$$� �� #%��&�� '��(�� )�*"��'�� %�� %��+� �� %,��

So, who did what ? The approaches….

24 Participating Groups Shots Stories Features Search ��-�� .��/0(��1��-��2 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X �� !��"�"� ��# CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X ��$%��&'� (%��)%� �� Fudan Univ. (China) X X X X �$��&' ��*�� +�*)�,�)�'!��! '�-.� FX-Pal (US) X IBM Research (US) X X X X ��/�'!��! '�-��!"��)%�!�'!��! '�-��!"� Imperial College London (UK) X X X Indiana University (US) X �$$)-��&�� &�)".��&�!��,'!�# Institut Eurecom (FR) X KDDI (JP) X X �&' �*'�� '!"'��)��'�!��!"�' � KU Leuven (BE) X Mediamill/U Amsterdam (NL) X ��))�/�"�,-��"��"'!*��!"��'!�� National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X *��'!�"��'!��'�!# RMIT University (Aus) X StreamSage (US) X Univ. of Bremen (D) X Univ. of Central Florida (US) X X X Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

Recall and precision for cuts (zoomed) �� !�� "� �� #$$� �� #%��&�� '��(�� )�*"��'�� %�� %��+� �� %,��

Gradual Transitions �� !�� "� �� #$$� �� #%��&�� '��(�� )�*"��'�� %�� %��+� �� %,��

24 Participating Groups Shots Stories Features Search ��*��32 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X 0� �"��!�'��*��"'��!�� /'�&��'�!� CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X ��$�! ��'�!�/&'�&�% � ��$�'��)��)�/�� Dublin City University (Irl) X X ��$��$�� !"�"'��"��'�!�� Fudan Univ. (China) X X X X FX-Pal (US) X "' �)�� # IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X 1�� % �"�'!��!"�� Institut Eurecom (FR) X KDDI (JP) X X /'�&�)'��)��"'�'��'�!# KU Leuven (BE) X Mediamill/U Amsterdam (NL) X 2) ��'!�)%"� �"'��"��'�!�� National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X �)� &� # RMIT University (Aus) X StreamSage (US) X Univ. of Bremen (D) X Univ. of Central Florida (US) X X X Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search �� %��&��-02 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X ��% �"��10��$$��&�,� �"��!� CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X ��$��' �!�% '!*�)%�'!�!�� Dublin City University (Irl) X X "'��!��!"��)�%��&' ��*�� Fudan Univ. (China) X X X X FX-Pal (US) X '�')��'�-# IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X 2"�$�'��&�� &�)"'!* Institut Eurecom (FR) X KDDI (JP) X X ��'�!��)� &� # KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X 3� �� &�"� ��4'!*��,)��4�� Ramon Llull Univ. (ES) X RMIT University (Aus) X ��"��'!��/&��&��&�-��"� .��) �� StreamSage (US) X "' �)�� # Univ. of Bremen (D) X Univ. of Central Florida (US) X X X Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search �!*��2 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X 5��&��$%�� )�� '�')��'�-� CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X �*�'! ��))�'!��/'!"�/��$� ��!"��%�%�� Dublin City University (Irl) X X �� .�� /�))�� '�')��'�-� Fudan Univ. (China) X X X X FX-Pal (US) X ,��/��!�$� ��6��%�%�� # IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X 3�!�� '�')��'�-��'��!"� Institut Eurecom (FR) X KDDI (JP) X X ��'!��&��' �'� ��&' ��'�� KU Leuven (BE) X '!"'��%� ��!"�3� # Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X �!�)%"� ��)��/�-��"%�� RMIT University (Aus) X StreamSage (US) X ��$%��'�!�� # Univ. of Bremen (D) X Univ. of Central Florida (US) X X X 7�� !��'�!��))�/# Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search �"�(��.2 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X 8 �"�10��%��'"�� - �� Fudan Univ. (China) X X X X FX-Pal (US) X 7�� !��'�!��))�/ IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X Univ. of Bremen (D) X Univ. of Central Florida (US) X X X Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search �'4��(��/�(�� 2 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X ��)�%��&' ��*�� '�')��'�-��"9��!�� CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X �� /'�&��! ��!�� '�')��'�-� Dublin City University (Irl) X X �&�� &�)"# Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X 1�� !"� &�/'!*��"�� Imperial College London (UK) X X X Indiana University (US) X 7�� &�� &�)"��'� # Institut Eurecom (FR) X KDDI (JP) X X 3��"�$��!�� '�$)��$$��&# KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X Univ. of Bremen (D) X Univ. of Central Florida (US) X X X Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search #$$�2 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X 5��%� .�$��$�� &��!��"�"�:7�3�� CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X ��)��&'*&�'!�� Dublin City University (Irl) X X "'��!�� % '!*��'�!�� &�!� Fudan Univ. (China) X X X X FX-Pal (US) X "��"��)'4�)-�� !"�� IBM Research (US) X X X X Imperial College London (UK) X X X )%�'!�!��!"��&��'!�!��"'��!�� # Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X 5��"' �)�� .�"��*��"%�)��&�!*'!*� KU Leuven (BE) X ��'��% '!*��'�'�-�"��# Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X 1$��'�'��"��'�!�)��4'!*��/'$� .��!"� RMIT University (Aus) X StreamSage (US) X ��)� &� # Univ. of Bremen (D) X Univ. of Central Florida (US) X X X 0��% ��'��$�� !��"�"� ��.� Univ. of Iowa (US) X X Univ. of Kansas (US) X ��)��'��!�7�# Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search #%(��&��2 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X 2"�$�'��&�� &�)"'!* �!��&��*�� CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X '!��! '�-�"'��!�� ,��/��!��"9��!�� Dublin City University (Irl) X X �� # Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X �!�)%"� ��'�!��$�! ��'�!�/&'�&� Imperial College London (UK) X X X Indiana University (US) X ��$%�� !��'!��! ��'�!� Institut Eurecom (FR) X KDDI (JP) X X ,��/��!��! ��%�'�� # KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X Univ. of Bremen (D) X Univ. of Central Florida (US) X X X Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search ��'��(�� %��&��-02 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X 3)�,�)��)�%��&' ��*��"'��!�� CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X �� %��"' ��!�'!%'�-�' �% �"��"�� Dublin City University (Irl) X X �%� # Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X 5��3� .��&�"��%!��)'!�� Imperial College London (UK) X X X Indiana University (US) X ��)�%��'��'�!��'��*� �� &�� Institut Eurecom (FR) X KDDI (JP) X X "%��'�!��&��3�.�/'�&� $��'�'�� KU Leuven (BE) X ��!��'!*��,9�� "%�'!*��&�� Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X 3��/&'�&��!�"' ��&' # Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X Univ. of Bremen (D) X Univ. of Central Florida (US) X X X Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search ��(%��&��-02 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X ��*��3� # CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X 8 '!*��'!*�/'!"�/��;��<�� .� Fudan Univ. (China) X X X X % ��%��!�� =0��*�'! ��))�'!� FX-Pal (US) X IBM Research (US) X X X X �&��/'!"�/�/'�&��:>��%!"� Imperial College London (UK) X X X Indiana University (US) X �%��!��# Institut Eurecom (FR) X KDDI (JP) X X 0� �"��!�� '�')��'�-��!"� KU Leuven (BE) X Mediamill/U Amsterdam (NL) X �"�$�'��&�� &�)"'!*# National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X 2��'!��!��!��# StreamSage (US) X Univ. of Bremen (D) X Univ. of Central Florida (US) X X X Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search %��&��-0(�5("��'��2 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X ��,'!��'�!��$$��&� ? CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X ��&�!*� �'!�'��*��)%�'!�!��# Fudan Univ. (China) X X X X FX-Pal (US) X �*��-�)��)�&' ��*��"'��!�� # IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X �55��%��'�!# Institut Eurecom (FR) X KDDI (JP) X X ��,'!�".�/'�&��"�$�'��&�� &�)"'!*# KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X Univ. of Bremen (D) X Univ. of Central Florida (US) X X X Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search %��&��-0(�5(��-��(�� 2 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X ��)�%��&' ��*��'!�� '�!�� CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X /'�&� %,� ��$)'!*��'"��$ # Dublin City University (Irl) X X Fudan Univ. (China) X X X X �&' �*'�� $$��'��)��'�!�� &�� FX-Pal (US) X IBM Research (US) X X X X ,�%!" .��))�/�"�,-��'!��*��'!�"�� Imperial College London (UK) X X X Indiana University (US) X ��$��' �!�% '!*��,'!��)�%�� Institut Eurecom (FR) X KDDI (JP) X X &' ��*��# KU Leuven (BE) X Mediamill/U Amsterdam (NL) X 7� ��$�� '!*��"��,�%$��&�!*� � National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X '!�'))%�'!��'�!�;��)� &� <# RMIT University (Aus) X StreamSage (US) X 2) ��"��'!�"��! '�'�!��-$� # Univ. of Bremen (D) X Univ. of Central Florida (US) X X X Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search %��&��-0(�5(��+�2 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X ��$��' �!��"9��!�� ,� �"��! CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X ��,'!�*)�,�)��)�%��&' ��*�� Fudan Univ. (China) X X X X FX-Pal (US) X ��$'��)��&%�,!�')�� &%�,!�')� IBM Research (US) X X X X Imperial College London (UK) X X X ,� �"��!�$'��)@$'��) Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X �1�,�) �')��'!*��!"�"��"��"*�� KU Leuven (BE) X "'��!�� Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X �!"��&�!�0��)��!��!"��'�&��'��$��"%�� RMIT University (Aus) X StreamSage (US) X ��,'!��'�! ��&� �# Univ. of Bremen (D) X Univ. of Central Florida (US) X X X 7�� !��'�!��))�/ Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

24 Participating Groups Shots Stories Features Search %��&��-0(�5(#��2 Accenture Technology Laboratories (US) X X Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X ��"��') ��')�,)��&' ��'�� Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X Univ. of Bremen (D) X Univ. of Central Florida (US) X X X Univ. of Iowa (US) X X Univ. of Kansas (US) X Univ. of North Carolina (US) X Univ. Oulu/VTT (FI) X X ��

Observations o Most techniques are based on frame-frame comparisons, some with sliding windows; o Comparisons are based on colour and on luminance, mostly; o Some use adaptive thresholding, some don’t; o Most operate on decoded video stream; o Some have special treatment of motion during GTs, of flashes, of camera wipes; o Performances are getting better; ��

Story segmentation and news typing o Identify the individual news items in a news show o New task in TRECVID, has been studied in ASR/IR community (TDT) o Hope to show the gain of using video features 1. Segmentation task n Identify story boundaries in CNN and ABC news shows n Ground truth based on TDT 2 annotations n Evaluation based on precision & recall, boundaries have to be within +/- 5 seconds interval around ground truth boundaries 2. News classification task n Annotate stories as either news or non-news n Evaluation based on percentage of correctly identified news story footage ��

8 Participating Groups Dublin City University (Irl) Fudan Univ. (China) IBM Research (US) KDDI (JP) National Univ. Singapore (Sing.) StreamSage (US) Univ. of Central Florida (US) Univ. of Iowa (US) ��

Story segmentation: recall and precision by condition � �(6� ��7�� 6� ��7�� 7�� (��(�� (8-.��

Story segmentation: recall and precision by system and condition (1-4) � $�% Conditions: �� 1: V+A 4 4 2 �� 2 4 1 2 2: V+A+ASR �"� 1 2 1 3: ASR �� #$$� 3 3 4: Other �� 1 �� 9%� �� 3 3 1 �-��'��/� �� 2 4 %�� 1 �� 2 %(��+� 3 1 3 4 �� 3 3 1 4 3 3 3 3 �� 3 ��

Segmentation, within system (F) 1 0,9 0,8 0,7 0,6 F (beta=1) AV AV+ASR 0,5 ASR 0,4 0,3 0,2 0,1 0 NUS IBM Fudan Iowa DCU StreamSage ��

Story classsification: news recall and precision by condition � �(6� ��7�� (6� ��7�� 7�� (��(�� (8-.�� 3��(:��(��+�: ��

Story classsification: news recall and precision by condition - zoomed � �(6� ��7�� 6� ��7�� 7�� (�� (��(�� (8-.�� 3��(:��(��+�: ��

Story classifcation: news recall and precision by system � �� "� �� 9%� �� %�� %(��+� �� 3��(:��(��+�: � � ��

Story classifcation: news recall and precision by system and condition (1-4) zoomed � 2 �� 2 Conditions: �� 1 1 2 1 4 4 1: V+A 4 �� "� 2: V+A+ASR 1 2 1 3: ASR 1 �� 2 1 2 4: Other 9%� 2 1 2 2 �� 3 3 %�� 3 �� %(��+� 2 3 2 3 3 3 1 �� 3��(:��(��+�: ��

Classification, within system (F) 1 0,95 0,9 0,85 AV F value AV+ASR 0,8 ASR OTHER 0,75 0,7 0,65 0,6 NUS IBM Fudan Iowa ��

3��%$�&��")'!� �� %��&��-0 1�*��!��'�! $�1��(��-0(%��&��-0(;��< A2!�&��"��'�!�,� �"��!��)% ��'!*� �� %��&�(;�.��< �!"�&�%�' �'� �"�(��.(;%�< #$$�(;=�<( A��'�)�"��'�!�,� �"��!�B 9�-��(%��&�(��/�4��(;��/�< A21�� *��!��'�!�% '!*��'�!�� -��'��/� ;%�<( ��')'!* %��&�(�5(��-��(�� ((;%�<( A�%)��,� �"��!"�:��!� �)� '�'�� %��&�(�5(��+�(;%�<(((((((((((((((((((((((((((( ��/ ��)� '�'��'�! A3::@:��!� % '!*��% '�.� ��'�)��!"� $��&�$��$��'�!�� %��

3��%$�&��")'!� #$$� 1�*��!��'�! $�1��(��-0(%��&��-0(;��< �� 2))� &�� )� '�'�"�� 2��CD�.� �� %��&�(;�.��< ��7D��D::��2E.�% '!*��%"'�� "�(��.(;%�< 6��'�!�'!��! '�-.��)�� Ł 1�:�� #$$�(;=�<( 1%, �(%�!�)-��%)��,� �"� 9�-��(%��&�(��/�4��(;��/�< �*��!��'�!� �-��'��/� ;%�<( �� '��)� '�'��'�!��,�%!"��'� .� %��&�(�5(��-��(�� ((;%�<( % '!*��&��%�� /�� &�� ,�� %��&�(�5(��+�(;%�<(((((((((((((((((((((((((((( �!"��&��,�%!"��-��!"'"��1�: �)� '�'��'�! 1�:��F1��F1.��F1�:�1�� !"�:�1��F1

3��%$�&��")'!� �-��'��/� ;>($�%< $�1��(��-0(%��&��-0(;��< 21��!)-� �*��!��'�!��%! �� %��&�(;�.��< �&��&�" ? �"�(��.(;%�< �� )��'��)��&�'!'!*��"��'!��$'��))-� #$$�(;=�<( ��&��!�� *��!� 9�-��(%��&�(��/�4��(;��/�< �-��'��/� ;%�<( �� '�!��')'!* %��&�(�5(��-��(�� ((;%�<( �� 8 ��&�" ��!"��$')'!*� %��&�(�5(��+�(;%�<(((((((((((((((((((((((((((( ��)' ��%��$&�� &��!!�%!�� $'��'!��"%��'�!��)� %��

3��%$�&��")'!� %��&��-0(�5(��-��(�� $�1��(��-0(%��&��-0(;��< ��,'!�"�1�*��!��'�!��!"� �� %��&�(;�.��< �)� '�'��'�!? �"�(��.(;%�< �� 1��-�,�%!"��'� ��4�"�,-�,)�!4� #$$�(;=�<( �� 9�-��(%��&�(��/�4��(;��/�< �� E�!*� ��-� Ł !�/ .� &�� -� Ł �-��'��/� ;%�<( !�!�!�/ %��&�(�5(��-��(�� ((;%�<( �� :��*��"9��!��!�!�!�/ � ��'� %��&�(�5(��+�(;%�<(((((((((((((((((((((((((((( ��!�)% '�!?� ��-�)�!*�&�' �� !*� ��%��!�/ ��)� '�'��'�! ��

3��%$�&��")'!� $�1��(��-0(%��&��-0 $�1��(��-0(%��&��-0(;��< �"�(��. �� %��&�(;�.��< 9�-��(%��&��-0(��/�4�� "�(��.(;%�< %��&��-0(�5(��+� #$$�(;=�<( 9�-��(%��&�(��/�4��(;��/�< $�� !��'�! ��))�/+� �-��'��/� ;%�<( %��&�(�5(��-��(�� ((;%�<( %��&�(�5(��+�(;%�<(((((((((((((((((((((((((((( ��

Observations o Video provides strong clues for story segmentation and even more for classification, best runs are either type 1 or 2 o AV runs generally have a higher precision o Combination of AV and ASR gives a small gain for segmentation o Most approaches are generic o Are the combination methods optimal? o Are the ASR segmentation runs state of the art? ��

FE Task definition o Goal: Build benchmark for detection methods of high-level features o Secondary goal: feature-indexing can help search and navigation o New: common feature annotation n Helps (a.o.) to standardize training resources across sites n Category A: sites work with just the common development data and common annotations n Category B: sites work with just the common development data and any annotation set n Category C: other ��

FE evaluation o Each feature is assumed to be binary: absent or present for each shot o Find shots that contain a certain feature, rank them according to confidence measure, submit the top 2000 o Submissions are pooled o Evaluate performance quality by measuring the average precision of each feature detection method ��

10 Participating Groups Accenture Technology Laboratories (US) Carnegie Mellon Univ. (US) CLIPS-IMAG (FR) CWI Amsterdam / Univ. of Twente (NL) Fudan Univ. (China) IBM Research (US) Imperial College London (UK) Institut Eurecom (FR) Univ. of Central Florida (US) Univ. Oulu/VTT (FI) ��

17 Features 11. Indoors 12. News subject face – not a news show person 13. People – at least three humans 14. Building – walled structure with roof 15. Road 16. Vegetation – living vegetation in its natural env. 17. Animal 18. Female speech – woman speaking (visible, audible) 19. Car/truck/bus – exterior of .. ��

17 Features 20. Aircraft 21. News subject monologue – uninterrupted 22. Non-studio setting 23. Sporting event 24. Weather news 25. Zoom in 26. Physical violence – between people / objects 27. Madeleine Albright – visible ��

Who worked on which features e c h t s n n c w e e e l v e o e o n n e i p o v n e o X s i r c o g d l i e n a t n s a g m u n a e i t h l c r f e n i a l f t o t t m o a a s t i l e r s i s a s p m s d o o m r w g w e o y d n r o c d l p i i r h e a e W o e u n e r e o e a n S P v i Z P o a N N p b F a N 3��%$ �� c i r 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 Accenture Technology Laboratories (US) X X X Carnegie Mellon Univ. (US) X X X X X X X X X X X X X X X X X CLIPS-IMAG (FR) X CWI Amsterdam / Univ. of Twente (NL) X X X X X X X X X X X X X X Fudan Univ. (China) X X X X X X X X X X X X X X X X X IBM Research (US) X X X X X X X X X X X X X X X X X Imperial College London (UK) X Institut Eurecom (FR) X X X X X X X X X X X X X X X Univ. of Central Florida (US) X X Univ. Oulu/VTT (FI) X X X X X X X X X X X X X X X 6 6 6 6 6 7 6 6 6 6 4 7 6 8 3 6 6 ��

AvgP by feature (all runs) Middle half of the data Median ��

AvgP by feature (top 10 runs) �� 1 �� 2 �&��/�(4�� 3 �� 4 5 �� 6 7 �� 8 9 �� 10 Median �� Median -> � �� -��

TREC 2003 Video Retrieval Evaluation Overview Coordinators: Alan - PowerPoint PPT Presentation

TREC 2003 Video Retrieval Evaluation Overview Coordinators: Alan Smeaton Centre for Digital Video Processing Dublin City University Wessel Kraaij Department of Data Interpretation Information Systems Division TNO TPD NIST: Paul Over

TREC 2005 Video Retrieval Evaluation Introductions Paul Over* Wessel Kraaij (TNO ICT) Tzveta

TREC 2006 Video Retrieval Evaluation Introductions Paul Over* Wessel Kraaij (TNO ICT) Tzveta

TREC 2003 Tracks A Tale of Two Evaluat ions Retrieval in a domain Genome Novelty Answers,

TREC Video Retrieval Evaluation TRECVID 2019 Ian Soboroff* Alan Smeaton, Yvette Graham

TREC Video Retrieval Evaluation TRECVID 2018 George Awad#* Alan Smeaton, Yvette Graham

9. Evaluation Outline 9.1. Cranfield Paradigm & TREC 9.2. Non-Traditional Measures 9.3.

Overview of TREC 2014 Ellen Voorhees Text REtrieval Conference (TREC) TREC 2014 Track

Search Evaluation at Grooveshark Yoni Teitelbaum 2013-07-02 Traditional Evaluation: TREC Image

Text REtrieval Conference (TREC) TREC TRACKS Crowdsourcing Personal Blog, Microblog documents

Text REtrieval Conference (TREC) Question Answering Tasks and Evaluation Methods Hoa Trang Dang

Overview of TREC 2013 Ellen Voorhees Text REtrieval Conference (TREC) Back to our roots, writ

What Weve Learned from Users Evaluation, session 11 CS6200: Information Retrieval Users vs.

What is TRECVID? Workshop series (2001 present) http://trecvid.nist.gov to promote

TREC, TAC, takeoffs, tacks, tasks, and titillations for 2009 Ian Soboroff, NIST

Video Retrieval using Speech and Image Information Alexander G. Hauptmann, Rong Jin, and Tobun D.

Russian Information Retrieval Evaluation Seminar (ROMIP) (ROMIP) http://romip.ru/en/ Igor

Retrieval by Content Image Retrieval Image Retrieval Problem Large Image and video data sets

Merging Results from Multiple Sources in Video Retrieval Wei-Hao Lin Language Technologies

Regional Trec - September 27, 2015 - Cadogan Farms TREC Workshop April 2015 Regional TREC

TRECVID 2017 George Awad#* Alan Smeaton (Dublin City University) Ian Soboroff *

From video information retrieval to hypervideo management Olivier Aubert Yannick Pri

TRECVID 2015 Paul Over^ George Awad# Alan Smeaton (Dublin City University) Ian Soboroff *

MCG-ICT-CAS TRECVID 2008 Automatic Video 2008 Automatic Video Retrieval System Retrieval System

TRECVID 2012 Paul Over* Alan Smeaton (Dublin City University) George Awad# Wessel Kraaij

TREC 2003 Video Retrieval Evaluation Overview Coordinators: Alan - PowerPoint PPT Presentation

TREC 2003 Video Retrieval Evaluation Overview Coordinators: Alan Smeaton Centre for Digital Video Processing Dublin City University Wessel Kraaij Department of Data Interpretation Information Systems Division TNO TPD NIST: Paul Over

TREC 2005 Video Retrieval Evaluation Introductions Paul Over* Wessel Kraaij (TNO ICT) Tzveta

TREC 2006 Video Retrieval Evaluation Introductions Paul Over* Wessel Kraaij (TNO ICT) Tzveta

TREC 2003 Tracks A Tale of Two Evaluat ions Retrieval in a domain Genome Novelty Answers,

TREC Video Retrieval Evaluation TRECVID 2019 Ian Soboroff* Alan Smeaton, Yvette Graham

TREC Video Retrieval Evaluation TRECVID 2018 George Awad#* Alan Smeaton, Yvette Graham

9. Evaluation Outline 9.1. Cranfield Paradigm &amp; TREC 9.2. Non-Traditional Measures 9.3.

Overview of TREC 2014 Ellen Voorhees Text REtrieval Conference (TREC) TREC 2014 Track

Search Evaluation at Grooveshark Yoni Teitelbaum 2013-07-02 Traditional Evaluation: TREC Image

Text REtrieval Conference (TREC) TREC TRACKS Crowdsourcing Personal Blog, Microblog documents

Text REtrieval Conference (TREC) Question Answering Tasks and Evaluation Methods Hoa Trang Dang

Overview of TREC 2013 Ellen Voorhees Text REtrieval Conference (TREC) Back to our roots, writ

What Weve Learned from Users Evaluation, session 11 CS6200: Information Retrieval Users vs.

What is TRECVID? Workshop series (2001 present) http://trecvid.nist.gov to promote

TREC, TAC, takeoffs, tacks, tasks, and titillations for 2009 Ian Soboroff, NIST

Video Retrieval using Speech and Image Information Alexander G. Hauptmann, Rong Jin, and Tobun D.

Russian Information Retrieval Evaluation Seminar (ROMIP) (ROMIP) http://romip.ru/en/ Igor

Retrieval by Content Image Retrieval Image Retrieval Problem Large Image and video data sets

Merging Results from Multiple Sources in Video Retrieval Wei-Hao Lin Language Technologies

Regional Trec - September 27, 2015 - Cadogan Farms TREC Workshop April 2015 Regional TREC

TRECVID 2017 George Awad#* Alan Smeaton (Dublin City University) Ian Soboroff *

From video information retrieval to hypervideo management Olivier Aubert Yannick Pri

TRECVID 2015 Paul Over^ George Awad# Alan Smeaton (Dublin City University) Ian Soboroff *

MCG-ICT-CAS TRECVID 2008 Automatic Video 2008 Automatic Video Retrieval System Retrieval System

TRECVID 2012 Paul Over* Alan Smeaton (Dublin City University) George Awad# Wessel Kraaij

9. Evaluation Outline 9.1. Cranfield Paradigm & TREC 9.2. Non-Traditional Measures 9.3.