ICTIR 2016 Slides - The Impact of Fixed-Cost Pooling Strategies on - PDF document

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/308120269 ICTIR 2016 Slides - The Impact of Fixed-Cost Pooling Strategies on Test Collection Bias Presentation · September 2016 CITATIONS READS 0 38 1 author: Aldo Lipani University College London 57 PUBLICATIONS 223 CITATIONS SEE PROFILE Some of the authors of this publication are also working on these related projects: Abstracting Domain-Specific Information Retrieval and Evaluation (ADmIRE) View project Space-time mapping and modelling of soil properties in Mediterranean and Temperate areas View project All content following this page was uploaded by Aldo Lipani on 15 September 2016. The user has requested enhancement of the downloaded file.

The Impact of 
 Fixed-Cost Pooling Strategies 
 on Test Collection Bias 1 Aldo Lipani, 2 Guido Zuccon, 1 Mihai Lupu, 3 Bevan Koopman, 1 Allan Hanbury 1 Vienna University of Technology 2 Queensland University of Technology 3 Australian e-Health Research Centre 15 Sep 2016 
 ICTIR 2016 - Newark (DE)

In the context of building a test collection for an IR task modelled by precision-based IR evaluation measures (P@10 and RBP) in presence of a fixed-budget , which of the tested pooling strategies introduces less pool bias ?

The Pooling Method

The Pooling Method

The Pooling Method

The Pooling Method

The Pooling Method

The Pooling Method

The Pooling Method

The Pooling Method

The Pooling Method

The Pooling Method

The Pooling Method

What is the Pool Bias?

What is the Pool Bias?

What is the Pool Bias? BIASED

What is the Pool Bias? BIASED It is the effect that documents that were not selected in the pool created from the original runs will never be considered relevant

Tested Pooling Strategies

Tested Pooling Strategies Standard pooling strategies:

Tested Pooling Strategies Standard pooling strategies:

Tested Pooling Strategies Standard pooling strategies: Take@10 ρ 1 ρ 2 ρ 3 ρ 4 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 r 1 r 2 r 3 r 4 r 5 1 ρ 1 ρ 2 2 ρ 3 3 4 ρ 4 5 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10

Tested Pooling Strategies Standard pooling strategies: Take@10 ρ 1 ρ 2 ρ 3 ρ 4 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 r 1 r 2 r 3 r 4 r 5 1 ρ 1 ρ 2 2 ρ 3 3 4 ρ 4 5 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10

Tested Pooling Strategies Standard pooling strategies:

Tested Pooling Strategies Standard pooling strategies: &

Tested Pooling Strategies Standard pooling strategies: & RBP based pooling strategies*: * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBP based pooling strategies*: * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBPBasedA@10&0.8 RBP based pooling strategies*: ρ 1 ρ 2 ρ 3 ρ 4 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 r 1 r 2 r 3 r 4 r 5 ρ 1 1 ρ 2 2 ρ 3 3 4 ρ 4 5 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBPBasedA@10&0.8 RBP based pooling strategies*: ρ 1 ρ 2 ρ 3 ρ 4 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 r 1 r 2 r 3 r 4 r 5 ρ 1 1 ρ 2 2 ρ 3 3 4 ρ 4 5 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBP based pooling strategies*: * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBP based pooling strategies*: * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBPBasedB@10&0.8 RBP based pooling strategies*: ρ 1 ρ 2 ρ 3 ρ 4 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 r 1 r 2 r 3 r 4 r 5 1 ρ 1 ρ 2 2 ρ 3 3 4 ρ 4 5 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBPBasedB@10&0.8 RBP based pooling strategies*: ρ 1 ρ 2 ρ 3 ρ 4 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 r 1 r 2 r 3 r 4 r 5 1 ρ 1 ρ 2 2 ρ 3 3 4 ρ 4 5 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBP based pooling strategies*: * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBP based pooling strategies*: * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBPBasedC@10&0.8 RBP based pooling strategies*: ρ 1 ρ 2 ρ 3 ρ 4 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 r 1 r 2 r 3 r 4 r 5 1 ρ 1 ρ 2 2 ρ 3 3 4 ρ 4 5 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBPBasedC@10&0.8 RBP based pooling strategies*: ρ 1 ρ 2 ρ 3 ρ 4 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 r 1 r 2 r 3 r 4 r 5 1 ρ 1 ρ 2 2 ρ 3 3 4 ρ 4 5 ρ 5 ρ 6 ρ 7 ρ 8 ρ 9 ρ 10 * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Tested Pooling Strategies Standard pooling strategies: & RBP based pooling strategies*: * A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

Methodology

Methodology Test Collections Ad Hoc 2-8 Web 9, Web 2001, Robust 2005 Genomics 2005 Legal 2006 Medical 2011 Microblog 2011

Methodology Domains Test Collections News Ad Hoc 2-8 Web Web 9, Web 2001, Robust 2005 Genomics Genomics 2005 Legal Legal 2006 Medical Medical 2011 Microblog Microblog 2011

Methodology Domains Test Collections News Ad Hoc 2-8 Web Web 9, Web 2001, Robust 2005 Genomics Genomics 2005 Legal Legal 2006 Medical Medical 2011 Microblog Microblog 2011 We compared Standard Pooling strategies RBP Based Pooling strategies (with p=0.80 and p=0.73)

Methodology Domains Test Collections News Ad Hoc 2-8 Web Web 9, Web 2001, Robust 2005 Genomics Genomics 2005 Legal Legal 2006 Medical Medical 2011 Microblog Microblog 2011 We compared Standard Pooling strategies RBP Based Pooling strategies (with p=0.80 and p=0.73) Metrics of Error MAE Mean Absolute Error SRE System Rank Error SRE* SRE with statistical significance (p<0.05)

Summary of the Results for MAE

Summary of the Results for MAE P@10 16 14 12 10 8 6 4 2 0 T T+ A80 B80 C80 A73 B73 C73 1st 2nd 3rd

Summary of the Results for MAE RBP (p=0.80) P@10 16 16 14 14 12 12 10 10 8 8 6 6 4 4 2 2 0 0 T T+ A80 B80 C80 A73 B73 C73 T T+ A80 B80 C80 A73 B73 C73 1st 2nd 3rd

Summary of the Results for MAE RBP (p=0.80) # Relevant Documents P@10 16 16 16 14 14 14 12 12 12 10 10 10 8 8 8 6 6 6 4 4 4 2 2 2 0 0 0 T T+ A80 B80 C80 A73 B73 C73 T T+ A80 B80 C80 A73 B73 C73 T T+ A80 B80 C80 A73 B73 C73 1st 2nd 3rd

Conclusion

Conclusion In the context of building a test collection for an IR task modelled by precision-based IR evaluation measures in presence of a fixed-budget , the best pooling strategy is

Conclusion In the context of building a test collection for an IR task modelled by precision-based IR evaluation measures in presence of a fixed-budget , the best pooling strategy is

Conclusion In the context of building a test collection for an IR task modelled by precision-based IR evaluation measures in presence of a fixed-budget , the best pooling strategy is However due to its limitations we recommend

Conclusion In the context of building a test collection for an IR task modelled by precision-based IR evaluation measures in presence of a fixed-budget , the best pooling strategy is However due to its limitations we recommend

Thank you 
 for you attention ! VisualPooling PoolBiasEstimators View publication stats View publication stats