Evaluation of Rich and Explicit Feedback for Exploratory Search - PowerPoint PPT Presentation

Nov 24, 2022 •483 likes •573 views

Evaluation of Rich and Explicit Feedback for Exploratory Search Esben Srig 1 , Nicolas Collignon 2 , Rebecca Fiebrink 1 , and Noriko Kando 3 1 Goldsmiths, University of London 2 University of Edinburgh 3 National Institute of Informatics, Japan

Evaluation of Rich and Explicit Feedback for Exploratory Search Esben Sørig 1 , Nicolas Collignon 2 , Rebecca Fiebrink 1 , and Noriko Kando 3 1 Goldsmiths, University of London 2 University of Edinburgh 3 National Institute of Informatics, Japan
Annotation feedback for exploratory IR • Current search systems have limited support exploratory searchers • Document annotation is a well- known strategy for active reading • We investigate the usage of annotations as an explicit feedback signal for IR systems. Alighieri, Dante. 1993. Convivio. Ed. Giorgio Inglese. Milano: Rizzoli
Previous work • Relevance feedback is a well-studied interactive, explicit feedback mechanism. User marks documents as relevant or irrelevant • Variants of relevance feedback such as interactive query expansion • Golovchinsky et al 1 studied the effect of annotation feedback on retrieval performance • Significant improvement in retrieval performance in single iteration of feedback • Fixed query, fixed search results, non-interactive, no user experience 1 G. Golovchinsky, M. N. Price, B. N. Schilit. 1999. From reading to retrieval: freeform ink annotations as queries. In Prooceedings of ACM SIGIR '99. ACM. 19-25.
Simulation retrieval performance • TREC Dynamic Domain 2017 user simulator • New York Times Annotated Corpus • Performance averaged over 60 search tasks
Simulation retrieval performance • TREC Dynamic Domain 2017 user simulator • New York Times Annotated Corpus • Performance averaged over 60 search tasks
Evaluation platform for annotation feedback
Experiment with human subjects • Goal : measure retrieval performance and user experience of real search users of annotation feedback • Two conditions: 1) Relevance feedback only. 2) Annotation feedback only • Between-subjects design • Amazon Mechanical Turk participants • Tasks chosen from user simulator • Pre- and post-task questionnaires to measure user experience
Discussion and questions • Is evaluation on TREC dataset representative of multi-session exploratory search contexts? • Are questionnaires sufficient to measure hypothesized benefit to user exploration, understanding, and engagement? • Questions?

Recommend

FedRec: Federated Recommendation with Explicit Feedback Guanyu Lin 1 , 2 # , Feng Liang 1 , 2 # ,

FedRec: Federated Recommendation with Explicit Feedback Guanyu Lin 1 , 2 # , Feng Liang 1 , 2 # , Weike Pan 1 , 2 and Zhong Ming 1 , 2 { linguanyu20161, liangfeng2018 } @email.szu.edu.cn, { panweike, mingz } @szu.edu.cn 1 National

801 views • 38 slides

Student Evaluation of Learning and Feedback for Instructors Student Evaluation of Learning and

Student Evaluation of Learning and Feedback for Instructors Student Evaluation of Learning and Feedback for Instructors SELFI Initial USAT Charge Phase 1 USAT committee (2014 15) Adopt a new set of quantitative (Likert style) questions

610 views • 31 slides

An Explicit Formula for the Zero-Error Feedback Capacity of a Class of Finite-State Additive

An Explicit Formula for the Zero-Error Feedback Capacity of a Class of Finite-State Additive Noise Channels Amir Saberi *, Farhad Farokhi and Girish Nair EEE Department, The University of Melbourne Email: a.saberi @ student.unimelb.edu.au ISIT

491 views • 32 slides

Evaluation or check ? Evaluation purpose? Feedback from exercise-lessons or best practise

Evaluation or check ? Evaluation purpose? Feedback from exercise-lessons or best practise Test the performance/capacity. Improving manuals-implementing result Improving the evaluation itself Goals from the old HELCOM

189 views • 16 slides

Explicit-State Abstraction: A New Method Abstractions for Generating Heuristic Functions

Explicit-State Abstraction Explicit-State Abstraction: A New Method Abstractions for Generating Heuristic Functions Projections Explicit-State Abstractions Evaluation Malte Helmert 1 Patrik Haslum 2 org Hoffmann 3 J Conclusion 1

485 views • 25 slides

Exploratory Analysis of Teaching Evaluation Data John Jordi, Ph.D., May 7, 2018 Learning &

Exploratory Analysis of Teaching Evaluation Data John Jordi, Ph.D., May 7, 2018 Learning & Organizational Development Specialist Office of Faculty Development & Teaching Excellence Introduction & Research Questions Student

154 views • 11 slides

NSF L.I.F.E. form Introduction February 4, 2020 Todd Abraham, PhD, MBA The Feedback Process

NSF L.I.F.E. form Introduction February 4, 2020 Todd Abraham, PhD, MBA The Feedback Process Level-of-Interest Feedback Evaluation aka L.I.F.E. Level-of-Interest Feedback Evaluation On-Line Feedback for Scientists on project proposals

245 views • 22 slides

Matrix Co-factorization for Recommendation with Rich Side Information and Implicit Feedback Yi

Matrix Co-factorization for Recommendation with Rich Side Information and Implicit Feedback Yi Fang and Luo Si Department of Computer Science Purdue University West Lafayette, IN 47906, USA fangy@cs.purdue.edu HetRec 2011 Yi Fang and Luo Si

367 views • 23 slides

Explicit and Implicit Discourse Relations: An Extrinsic Evaluation Peter Bourgonje and Manfred

Explicit and Implicit Discourse Relations: An Extrinsic Evaluation Peter Bourgonje and Manfred Stede Applied Computational Linguistics Universitt Potsdam Workshop on Coherence Relations Humboldt-Universitt zu Berlin January 17-18, 2020

342 views • 21 slides

Unbiased Offline Recommender Evaluation for Missing-Not-At-Random Implicit Feedback Serge

Unbiased Offline Recommender Evaluation for Missing-Not-At-Random Implicit Feedback Serge Belongie Deborah Estrin Lo Longqi Yang Yuan Xuan Chenyang Wang Yin Cui Funders: 1 Offline Evaluation of Recommendation Algorithm user-item

619 views • 36 slides

Course Feedback Uses: Improvement and Formal Evaluation Fall Faculty Conference 2018 Student

Course Feedback Uses: Improvement and Formal Evaluation Fall Faculty Conference 2018 Student Course Feedback Student Course Feedback Conversations with Students The Basics Tell your students: You value honest and constructive

664 views • 13 slides

Announcements: - Thank you for participating in our mid-quarter evaluation Thank you for

Announcements: - Thank you for participating in our mid-quarter evaluation Thank you for participating in our homework feedback polls! - - Course project - Average was ~80% Dont worry about grade but take feedback seriously - -

423 views • 41 slides

An Exploratory Study of How Developers Exploratory Study Seek, Relate, and Collect Relevant

Code Investigation Z ephyrin Soh Context and Paper An Exploratory Study of How Developers Exploratory Study Seek, Relate, and Collect Relevant Results Activities Information during Software Strategy Search Relate Maintenance Tasks

397 views • 23 slides

Exploratory Monitoring at Bing AUTOMATED SYNTHETIC EXPLORATORY MONITORING OF DYNAMIC WEB SITES

Exploratory Monitoring at Bing AUTOMATED SYNTHETIC EXPLORATORY MONITORING OF DYNAMIC WEB SITES USING SELENIUM Outline 1. Modern Engineering Principles 2. Monitoring Approaches 3. Statistical Models 4. Selenium 5. Exploratory Principles 6.

618 views • 26 slides

Health benefits of biofortification an economic ex-ante evaluation of iron-rich rice and wheat

Health benefits of biofortification an economic ex-ante evaluation of iron-rich rice and wheat in India Alexander J. Stein, J.V. Meenakshi, Matin Qaim, Penelope Nestel, H.P.S. Sachdev and Zulfiqar A. Bhutta HarvestPlus/IFPRI, Washington,

531 views • 40 slides

The main messages emerging from our analysis of data, self-evaluation and feedback from key

The main messages emerging from our analysis of data, self-evaluation and feedback from key stakeholders is: Keep going you are doing the right things, We are having an impact on learner outcomes and sustaining improvement where it is

729 views • 29 slides

EXPLORATORY PRACTICE Ins K. de Miller (PUC-Rio, Brasil) Exploratory Practice: work for

Ministry of Education Chile / British Council Chile Initial Teacher Education Seminar on Teacher-research 22-23 March 2016, Santiago EXPLORATORY PRACTICE Ins K. de Miller (PUC-Rio, Brasil) Exploratory Practice: work for understanding -

337 views • 16 slides

QMT QCD Multi Threading First steps Step 1: General Evaluation OpenMP vs.

QMT QCD Multi Threading First steps Step 1: General Evaluation OpenMP vs. Explicit Thread library (Chen) Explicit thread library can do better than OpenMP OpenMP performance is compiler dependent Intel compiler does

388 views • 5 slides

Im Implicit Feedback and Performance Evaluation in in Recommender Systems Shay Ben Elazar

Im Implicit Feedback and Performance Evaluation in in Recommender Systems Shay Ben Elazar Mike Gartrell Noam Koenigstein Gal Lavee Agenda Intro Universal Store Recommendations Extreme Classification with Matrix Factorization

905 views • 37 slides

Subgroup Discovery Exploratory Data Analysis Exploratory Data Analysis Classification:

Subgroup Discovery Exploratory Data Analysis Exploratory Data Analysis Classification: model the dependence of the target on the remaining attributes. problem: sometimes uses only some of the available dependencies, or classifier is

1.64k views • 77 slides

Session-Based Exploratory Session-Based Exploratory TestingWith a Twist TestingWith a

T19 Concurrent Session Thursday 10/25/2007 3:00 PM JUMP TO: Biographical Information The Presentation Session-Based Exploratory Session-Based Exploratory TestingWith a Twist TestingWith a Twist Presented by: Brenda Lee, Parallax,

461 views • 24 slides

Feedback on guidance document and MQO Application of Delta Tool for GEM-AQ model evaluation Pawel

Modelling support for near-surface ozone pollution assessment for 2012 - 2015 Poland Feedback on guidance document and MQO Application of Delta Tool for GEM-AQ model evaluation Pawel Durka, Joanna Struzewska, Jacek W. Kaminski Main

257 views • 25 slides

VISUALIZATION Jeff Goldsmith, PhD Department of Biostatistics 1 Exploratory data analysis

VISUALIZATION Jeff Goldsmith, PhD Department of Biostatistics 1 Exploratory data analysis Exploratory analysis is a loosely-defined process Roughly, the stuff between loading data and formal analysis is exploratory This

407 views • 15 slides

Clinical Teaching Microskills Microskills Clinical Teaching Evaluation and Feedback Evaluation

Clinical Teaching Microskills Microskills Clinical Teaching Evaluation and Feedback Evaluation and Feedback Bringing Education & Service Bringing Education & Service Together (BEST) Together (BEST) Learning Objectives Learning

629 views • 13 slides