INEX 2012 Overview Shlomo Geva Jaap Kamps Ralf Schenkel
10 years! 2002-2012 INEX 2012 Overview Shlomo Geva Jaap Kamps Ralf Schenkel
Search changed a lot in 10 years!
INEX teams up with CLEF in 2012
INEX’11 Workshop was on Dec 13-15, 2011 So INEX’12 ran for only nine months... ... which was hard ... Apologies to the CLEF folks for running late’ish
Social Book Search Linked Data Tweet Contextualization Five tracks Snippet Retrieval Huize Bergen, Vught, The Netherlands, Dec 13-15, 2010 Relevance Feedback
Social Book Search Track
Topic title Narrative Recommended Group books name 300 topics + recommendations from the LT forum Also crowdsourcing recommendation/relevance
Pre- & Post Cataloguing
SBS Task Results Detailed results discussed in the INEX sessions
Task 2: “Prove it” task against scanned books “Structure Extraction” task @ ICDAR
Extensive use of crowdsourcing (topics, judgments)
Linked Data Track Corpus: DBpedia/YAGO + Wikipedia Investigate textual and highly structured data
Three LD Tasks • Ad hoc retrieval -- retrieve relevant Wikipedia entities. • Faceted search -- recommend a hierarchy of facet- values obtained from the RDF data that will optimally guide the user toward relevant Wikipedia article in a large result set. • Jeopardy! -- provide answers for natural language Jeopardy! clues which are manually translated into SPARQL queries extended with keyword conditions.
Ad hoc: Structured helped the best run(s) Faceted: evaluation is ongoing... Jeopardy!: SPARQL effective but text better
Tweet Contextualization Track Task description From : Guggenheim Museum On #InternationalWomensDay, we are proud to present our new Francesca Woodman exhibition opening Mar 16: http://t.co/AyuRH1OF What International Women's Day is? Who Francesca Woodman is? What kind of art it is? Where this exhibition is? What Guggenheim Museum is? ...
Task description Given a tweet and its metadata Select a number of related passages from - Wikipedia Order them so that it's a comprehensive - contextualization of the tweet → Multi-document summarization / answer aggregation Evaluation: Informativeness - Readability -
Evaluation 1000 tweets manual or automatically collected Evaluation of: Informativeness (63 topics) by organizers - Readability (16 topics) by participants - 33 submitted runs + 1 organizer baseline 13 teams (Canada, Chile, France, Germany, India, Ireland, Mexico, Russia, Spain, USA)
Snippet Retrieval Track
Relevance'Assessment' SnippetMbased%assessment% Assessor%reads%through%the%20%snippets%for%each%topic,% and%judges%each%as%relevant/irrelevant.% DocumentMbased%assessment% Each%document%is%reassessed%by%the%same%assessor%using% the%full%document%text%providing%the%‘groundMtruth’.% Evaluation%is%based%on%comparing%these%two%sets%of% judgments.%
Timeline' Round%1% Running%(very)%Late…% Round%2% Submissions%due:%%Oct%19% Assessment:%Oct%29%–%Nov%24% Results%released:%Dec%3%
Relevance Feedback Track (Open Source Retrieval workshop at SIGIR)
+ Evaluation Platform Track participants provided with the complete document collection in advance (2,666,192 docs INEX Wikipedia 2009) Evaluation platform provides the relevance feedback modules with topics Simulates a user in the loop, interacting with the search system Evalua&on) Relevance) Document) Assessments Feedback) Pla+orm) Collec,on) Algorithm)
2012'(preliminary)'results ' Exact Precision - Best non-RF and best RF submissions from each participant 0.7 0.6 0.5 BASE-IND 0.4 RRMRF-300D-L05 TOPSIG-2048 0.3 TOPSIG-RF4 0.2 0.1 0 @5 @10 @15 @20 @30 @100 @200 @500 @1000
Shlomo Geva Jaap Kamps Lecture Notes in Computer Science Ralf Schenkel The LNCS series reports state-of-the-art results in computer science Geva et al. (Eds.) Andrew Trotman (Eds.) re search, development, and education, at a high level and in both printed and electronic form. Enjoying tight cooperation with the R&D community, with numerous individuals, as well as with prestigious organizations and societies, LNCS has grown into the most comprehensive computer science research forum available. The scope of LNCS, including its subseries LNAI and LNBI, spans the whole range of computer science and information technology including LNCS 6932 interdisciplinary topics in a variety of application fields. The type of Comparative Evaluation material published traditionally includes – proceedings (published in time for the respective conference) 1 – post-proceedings (consisting of thoroughly revised final full papers) of Focused Retrieval – research monographs (which may be based on outstanding PhD work, research projects, technical reports, etc.) 9th International Workshop of the Inititative More recently,several color-cover sublines have beenadded featuring, LNCS beyond a collection of papers, various added-value components; these for the Evaluation of XML Retrieval, INEX 2010 6932 sublines in clude Vugh, The Netherlands, December 2010 – tutorials (textbook-like monographs or collections of lectures given at Revised Selected Papers of Focused Retrieval Comparative Evaluation advanced courses) – state-of-the-art surveys (offering complete and mediated coverage of a topic) – hot topics (introducing emergent topics to the broader community) In parallel to the printed book, each new volume is published electronically in LNCS Online. Detailed information on LNCS can be found at www.springer.com/lncs Proposals for publication should be sent to LNCS Editorial, Tiergartenstr. 17, 69121 Heidelberg, Germany E-mail: lncs@springer.com ISSN 0302-9743 INEX’11 LNCS in Sep/Oct; 2012 coming as well. ISBN 978-3-642-23576-4 9 7 8 3 6 4 2 2 3 5 7 6 4 123 › springer.com INEX 2010
Plans for INEX 2013 are under discussion
Recommend
More recommend
