Tasks at Past NTCIRs - PowerPoint PPT Presentation

NTCIR: NII Testbeds and Community for Information access Research Research Infrastructure for Evaluating IA A series of evaluation workshops designed to enhance research in information-access technologies by providing an ■ Data sets, evaluation methodologies, and forum infrastructure for large-scale evaluations. Project started in late 1997 17 NTCIR-9 90 119 17 NTCIR-8 65 Once every 18 months 109 15 NTCIR-7 82 12 Data sets (Test collections or TCs) Data sets (Test collections or TCs) NTCIR-6 85 # of countries 15 15 NTCIR-5 NTCIR-5 77 77 10 Scientific, news, patents, web, CQA, # of Active NTCIR-4 74 9 participant groups NTCIR-3 65 Wikipedia, Entrance Exams # of registered 8 NTCIR-2 36 6 Chinese, Korean, Japanese, and English NTCIR-1 28 0 50 100 150 Tasks (Research Areas) QA ： Monolingual tasks, cross-lingual tasks IR: Cross-lingual tasks, patents, web, Geo Summarization, trend info., patent maps Opinion analysis, text mining Community-based Research Activities 1

Tasks at Past NTCIRs ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ������������ '���� �������% ■ ■ ■ ■ Module-Based ���������������� �������� ■ ■ ■ ■ ■ ■ ■ ■ �������������� ■ ■ ■ ■ ■ ■ ■ ■ ��������������������� IR for Focused ■ ■ ■ ■ ■ ■ ■ ■ ������������� Domain ■ ■ ■ ■ ■ ■ ■ ■ ������������� □ □ □ □ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ���!������� □ □ □ □ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ "����� ■ ■ ■ ■ ■ ■ ■ ■ �������#�����!���� ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ �������� ������ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ������������� ���(����� Cross Link Discovery ■ ■ ■ ■ ■ ■ ■ ■ $�����%&����� ■ ■ ■ ■ ■ ■ ■ ■ ���������� ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ Summarization / �������������� �������������� ��������� ��������� ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ !����)������#��������������� Consolidation ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ !���%������*������+����� ■ ■ ■ ■ Interactive !�����������+����� ■ ■ ■ ■ ■ ■ ■ ■ *������+����������� ■ ■ ■ ■ ■ ■ ■ ■ �������)����� ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ,�- ��������������" ■ ■ ■ ■ ■ ■ ■ ■ Crosslingual ■ ■ ■ ■ ■ ■ ■ ■ ,�- ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ������������)! ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ���������������� Text Retrieval ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ .���������/������/ �%�0�����&���������� (����/��%2�!/��������������%�34�����/��-����� 2

NTCIR-9/10: Objectives • Solid foundation • Community-led task organisation – New structure – Sustainability of • Task diversity research – Covers a wide context – Covers a wide context – Seed Funding in Information Access – Studies rich media • Promotion of research types resources – Show case in the NTCIR Meeting 3

.!���5� 検索システムの力くらべ http://research.nii.ac.jp/ntcir/ 4 4

NTCIR-10: Structure TO • General Co-Chairs • Task Organizers – Tsuneaki Kato (U Tokyo) – 48 researchers all over the world – Noriko Kando (NII) – Particiapnts (you!) – Douglas W. Oard – Douglas W. Oard (University of Maryland) • EVIA 2013 Co-Chairs – Mark Sanderson (RMIT) – Ruihua Song (MSRA) • Program Co-Chairs – William Webber (University of Maryland) – Hideo Joho (U Tsukuba) – Tetsuya Sakai (MSRA) 5

http://research.nii.ac.jp/ntcir/ntcir-10/organizers.html 6

NTCIR-10 Program Committe • Charles Clarke (University of Waterloo, Canada) • Kalervo Järvelin (University of Tampere, Finland) • Hideo Joho (Co-chair, University of Tsukuba, Japan) • Gareth Jones (Dublin City University, Ireland) • Noriko Kando (NII, Japan) Noriko Kando (NII, Japan) • Tsuneaki Kato (The University of Tokyo, Japan) • Douglas W. Oard (University of Maryland) • Tetsuya Sakai (Co-chair, Microsoft Research Asia, PRC) • Mark Sanderson (RMIT, Austraria) • Ian Soboroff (NIST, US) 7

NTCIR-9 Tasks CORE TASKS • [Intent] Intent (with One-Click Access) – Subtask 1 CLICK -> CORE • [RITE] Recognizing Inference in Text • [GeoTime] Geotemporal information retrieval (x) [GeoTime] Geotemporal information retrieval (x) • [SpokenDoc] IR for Spoken Documents PILOT TASKS • [CrossLink] Cross-lingual Link Discovery -> Core • [Vis-Ex] Interactive Visual Exploration (x) • [PatentMT] Patent Machine Translation -> Core 8

NTCIR-10 Accepted Tasks Core • [Intent-2] Search intent and diversification • [1Click-2] One-Click Access • [RITE-2] Recognizing Inference in Text • [SpokenDoc-2] IR for Spoken Documents • [PatentMT-2] Cross-lingual access to Patent Docs • [CrossLink-2] Cross-lingual Link Discovery Pilot • [Math] Access to mathematical contents 9

Cross-lingual Link Discovery No link was created for this term, for Article: Australia Article: Australia � Links in other languages? finding articles in languages we � New articles? prefer traditionally we do: … � Missing links? Ranked third in the Index of Economic � Not what we are looking for? Freedom (2010),[178] Australia is the world's thirteenth largest economy and has the ninth � What about other relevant highest per capita GDP; higher than that of the Search Translate United Kingdom, Germany, France, Canada, links? Japan, and the United States. The country was ranked second in the United Nations 2010 Human Development Index and first in Legatum's 2008 Prosperity Index.[179] All of Australia's major cities fare well in global comparative livability surveys;[180] Melbourne Cross-lingual Link Discovery Cross-lingual Link Discovery reached first place on The Economist 's 2011 reached first place on The Economist 's 2011 经济学人 World's Most Livable Cities list, followed by 이코노미 … 스트 Sydney, Perth, and Adelaide in sixth, eighth, The … Economist and ninth place respectively.[181] Total … government debt in Australia is about $190 � Cross-lingual Links エコノミ billion.[182] Australia has among the highest スト … � New Links house prices and some of the highest household debt levels in the world. � Better Links … � More options How to automatically create cross-lingual links for a document if no links existing � � � � yet? • All about multi-lingual knowledge discovery in knowledge bases (e.g. Wikipedia) • All about easy and efficient information access

INTENT-2: underspecified Query Harry Potter

Search Results Diversification Produce SERP to sartisfy the various usrs’s intentions against the underspecified queries (SERP)

INTENT-1 & INTENT-2 INTENT-1@NTCIR-9 INTENT-2@NTCIR-10 INTENT-1 INTENT-2 topics Topics INTENT-1 INTENT-2 Systems Systems INTENT-1 system test s on INTENT-2 INTENT-2 system tests on Topics (Test the INTENT-1 topics relationship of the two topic sets) (show improvments from INTENT-1

Traditional Search = More-than-One Click Access 湘南厚木病院 (Shonan Atsugi Hospital) Enter query Click SEARCH button Scan ranked list of URLs Click URL Read URL contents Get all desired information