NTCI R Workshop is : Overview of the 4 th NTCI R Workshop A series of evaluat ion workshops designed t o enhance research in inf ormation access t echnologies by providing inf rast ruct ure of large-scale evaluat ion. Project started late 1997, Once per 1½ years Noriko Kando st : Nov.1,1998- Sept .1,1999 1 Nat ional I nst it ut e of I nf ormat ics 2 nd : J une,2000– March,2001 3 rd : Sept 2001- Oct 2002 <- results http:/ / research. nii. ac. jp/ ntcir/ 4 th : Apr 2003 – June 2004 <- dry run & progress kando@nii.ac.j p * N ii T est C ollect ion f or I nf ormat ion R et rieval syst ems * Co-sponsored by NI I and MEXT Gr ant -in-Aid on I nf or mat ics ntcir4-ov 2004-06-02 1 ntcir4-ov 2004-06-02 2 Purposes of NTCI R Evaluation Workshops • "evaluation workshop" or "evaluation". Large-scale reusable t est collect ions f or experiment s of • provides to participants a set of data inf ormat ion access t echnologies usable f or experiments and unif ied procedures f or evaluation NTCI R enables: An int ernat ional f orum of * Cross-syst em researcher groups – Each participating research group conducts comparison on experiments with various approaches and can common evaluat ion participate with own purpose. I nvest igat ion on evaluat ion inf rast ruct ure met hodologies and met rics • Successf ul examples; TREC * Rapid improvement of t he ef f ect iveness • I mplications are various and t echnology t ransf er of lat est researches ntcir4-ov 2004-06-02 3 ntcir4-ov 2004-06-02 4 Tasks (Research Areas) of NTCI R Workshops Focus of NTCI R Project start ed lat e 1997 1st 2nd 3rd 4t h New Challenges Japanese I R Lab-t ype I R Test T Cross- lingual I R I nt ersect ion of I R + NLP a Asian Languages/ cross-language s Patent Retrieval To make inf ormat ion in t he Variet y of Genre k document s more usable f or Parallel/ comparable Corpus users! s Web Retrieval Realist ic eval/ user t ask Term Extraction/ Role Analysis Question Forum f or Researchers Answering Text I dea Exchange Summarization Nov 98 Discussion/ I nvest igat ion on Apr 2003 Evaluat ion met hods/ met rics About once per 1 ½ years ntcir4-ov 2004-06-02 5 ntcir4-ov 2004-06-02 6 1
NTCI R テストコレクション Tasks of NTCI R- 4 D o c u m e n t s t o p i c . / Q R e l e v a n c e / C o l l e c t i o n t a s k A n s w e G e n r e S i z e L a n g u a g e L a n g u a g e r CLI R: Chinese, Korean, J apanese, English N T C I R - 1 I R A c a d e m i c 5 7 7 M B J E J 3 - Single Language I R; Bilingual CLI R; Mult ilingual CLI R; C I R B 0 1 0 I R N e w s 1 3 2 M B C t C t E 4 Pivot CLI R I R N T C I R - 2 A c a d e m i c 8 0 0 M B J E J E 4 Pat ent Ret rieval N T C I R - 2 S u m m S u m m N e w s 1 8 0 d o c s J J - Main : I nvalidit y Search (Search pat ent by pat ent ) I R C t K J E N T C I R - 3 C L I R N e w s 8 8 4 M B C t K J E 4 - Feasibilit y: Aut omat ic Pat ent Map Gener at ion I R N T C I R - 3 P A T E N T P a t e n t 1 8 G B ( + 5 G B ) J ( J E ) C s C t K J E 3 Quest ion Answering N T C I R - 3 Q A Q A N e w s 2 8 2 M B J J ( E ) e x a c t - 5 possible answers; 1 set of all t he answers; series of Qs 6 0 d o c s ＋5 0 Text Summarizat ion N T C I R - 3 S u m m S u m m N e w s J － s e t s - Mult iple document summarizat ion --Aut omat ic evaluat ion! N T C I R - 3 W E B I R W E B 1 0 0 G B M u t l i p l e J ( E ) 4 + r e l a t i v e Web N T C I R - 3 C L I R I R N e w s c a 3 G B C t K J E C t K J E 4 C s C t K J E - I nf ormat ional; Navigat ional; Geographic; Clust ering N T C I R - 3 P A T E N T I R P a t e n t 4 5 G B J ( J E ) 3 N T C I R - 3 Q A Q A N e w s 7 7 6 M B J J ( E ) 4 N T C I R - 3 S u m m S u m m N e w s 3 0 s e t s J － N T C I R - 3 W E B I R W E B 1 0 0 G B M u t l i p l e J ( E ) C t ： T r a d i t i o n a C l h i n e s e 、 C s ： S i m p l i f e i d C h i n e s e 、 K ： K o r e a n 、 J ： J a p a n e s e 、 E ： E n g l i s h ntcir4-ov 2004-06-02 7 ntcir4-ov 2004-06-02 8 NTCI R Workshop 4 +CLI R Kuang-hua Chen, NTU NTCI R workshop: Sukhoon Lee, NCU (2003- 2004) Kazuaki Kishida, Surugadai U Number of Part icipat ing Groups Hsin-Hsi Chen, NTU Organizers Sung Hyon Myaeng, I I U 74 groups f rom 10 Kazuko Kur iyama, Shir ayur i U countries Noriko Kando, NI I +PATENT 74 4t h workshop At sushi Fuj ii, Tsukuba U 10 Makot o I wayama, Hit achi Noriko Kando, NI I 65 3rd workshop +Question Answering 9 J unichi Fukumot o, Rit sumeikan U Tsuneaki Kat o, U Tokyo 36 Fumit o Masui, Mie U 2st workshop 8 # of groups +Text Summarization Tsut omu Hirao, NTT-CS 28 Takahir o Fukusima, Ot emongakuin U 1st workshop Hidet sugu Nanba, Hiroshima C U 6 # of count ries Manabu Okumur a, TI TEC General chair : J un Adachi, NI I 0 2 0 4 0 6 0 8 0 +WEB Program chair : Nor iko Kando, NI I Koj i Eguchi, NI I Keizo Oyama, NI I ntcir4-ov 2004-06-02 9 ntcir4-ov 2004-06-02 10 [QAC] Number of Participants by Tasks [CLIR] AIST/University of Nagoya/Univeristy of Tsukuba (Japan) Chinese Academy of Sciences (China PRC) Communications Research Laboratory-1 (Japan) Clairvoyance Corporation and Justsystem (USA) Iwate Prefectural University (Japan) Communications Research Laboratory-1 (Japan) Keio University (Japan) Fu Jen Catholic University (Taiwan ROC) Matsushita Electoric Industiral-1 (Japan) 120 120 Hong Kong Polytechnic University (Hong Kong, China PRC) Mie University (Japan) Hummingbird (Canada) Nagaoka University of Technology (Japan) Institute of Inforcomm Research (Singapore) Nara Institute of Science and Technology-2 (Japan) Korea University (Korea) New York University (USA)/Communication Research Lobaratory-2 (Japan) Nara Institute of Science and Technology-1(Japan) NTT Communication Science Laboratories-1 (Japan) 100 100 QA # of ParticipatingGroups QA # of ParticipatingGroups National Institute of Informatics-1 (Japan) NTT DATA (Japan) National Taiwan University (Taiwan ROC) Oki Electric-2(Japan) Oki Electric-1 (Japan) Summarizat ion Pohang University of Science and Technology (Korea) Summarizat ion PATOLIS (Japan) Ritsumeikan University (Japan) 80 Pohang University of Science and Technology (Korea) Toshiba (Japan) 80 Term Ext ract ion Queens College City Univiersity of New York (USA) Toyohashi University of Technology-1 (Japan) Term Ext ract ion Ricoh-1 (Japan) University of Tokyo-1 (Japan) Royal Melbourn Intitute of Technology (Australia) Yokohama National University (Japan) Web Ret rieval Web Ret rieval Thomson Legal and Regulatory (USA) 60 60 Tianjin University (China PRC) [TSC] Toshiba (Japan) Communications Research Laboratory-2 (Japan) / New York University (USA) Pat ent Ret rieval Pat ent Ret rieval University of Arizona (USA) Graduate University for Advanced Studies (Japan) University of California Berkeley (USA) Hokkaido University (Japan) University of Chicago (USA) Pohang University of Science and Technology (Korea) NonJ apanese I R NonJ apanese I R 40 40 University of Neuchatel (Switzerland) Ritsumeikan University (Japan) chinese University of Tsukuba (Japan) Toyohashi University of Technology-1 (Japan) Chinese . Korean CLI R Yokohama National University (Japan) University of Electro-Communications (Japan) CLI R University of Tokyo-1 (Japan) Number of [PATENT] Yokohama National University (Japan) 20 20 J � E, E � J 、 J apanese I R J apanese I R Fujitsu Laboratories (Japan) J � E x � CJEK E � C IBM Research (Japan) [WEB] Participants Japan Patent Information Organization / Hitachi (Japan) Hokkaido University (Japan) Nagaoka University of Technology (Japan) Ibaraki University (Japan) 0 NTT DATA (Japan) Matsushita Electoric Industiral-2 (Japan) 0 Osaka Kyoiku University (Japan) NEC (Japan) PATOLIS (Japan) NII-2/Univ. of Tokyo-2/KYA Group (Japan) 1st (1998-9) 1st (1998-9) 2nd(2000-1) 2nd(2000-1) 3r d (2001-2) 3r d (2001-2) 4t h(2003-4) 4t h(2003-4) Ricoh-2 (Japan) NTT Communication Science Laboratories-2 (Japan) Tokyo Institute of Technology (Japan) Osaka Kyoiku University (Japan) University of Tsukuba (Japan) Tokyo Metropolitan University (Japan) ntcir4-ov 2004-06-02 11 ntcir4-ov 2004-06-02 12 Toyohashi University of Technology-1 (Japan) Toyohashi University of Technology-2 (Japan) 74 groups from 10 countries & areas University of Tsukuba/University of Nagoya 2
Recommend
More recommend