Search Snippet Evaluation Mikhail Lebedev, Pavel Braslavski, Denis - PowerPoint PPT Presentation

Search Snippet Evaluation Mikhail Lebedev, Pavel Braslavski, Denis Savenkov CLEF 2011 CLEF 2011

What is a snippet What is a snippet #

Why is it so important Why is it so important Search is ranking & representation Bad snippets spoil good results Bad snippets spoil good results Overoptimized snippets lead to wrong clicks

When we use evaluation When we use evaluation Compare with competitors Choose between different Choose between different algorithms g Machine learning hi l i

Evaluation approach Evaluation approach User study Judgments Text ‐ based Text based Clicks metrics metrics

Evaluation approach Evaluation approach User study Judgments Text ‐ based Text based Clicks metrics metrics

Eye tracking

Eye tracking Eye tracking Different users, different strategies , g Title is much more important than body Highlighting is attractive hl h User clicks even if snippet contains answer U li k if i t t i Media content may be ignored Media content may be ignored

Evaluation approach Evaluation approach User study Judgments Text ‐ based Text based Clicks metrics metrics

Ideal snippet is: Ideal snippet is: I f Informative ti R Readable d bl

Making ideal snippets Making ideal snippets Machine learning Training set Training set Judgments Ab Absolute l t R l ti Relative

Relative assessment Relative assessment

Assessment issues Assessment issues Assessors learn Snippet quality depends on ranking Snippet quality depends on ranking Assessment tool interface matters Pairs vs groups: time vs quality Pairs vs groups: time vs quality

Evaluation approach Evaluation approach User study Judgments Text ‐ based Text based Clicks metrics metrics

Automated metrics Automated metrics Highlighting Neatness Neatness Number of empty snippets Unique query words Unique query words

Evaluation approach Evaluation approach User study Judgments Text ‐ based Text based Clicks metrics metrics

Click data Click data Dwell time Abandonment Abandonment Inversions Time to first click Time to first click

Conclusions Conclusions Different goals – different methods User ‐ study: making assumptions Judgments: expensive quality Judgments: expensive quality Text based metrics: fast but rough Text ‐ based metrics: fast but rough Clicks: ranking influence

Future Future integral metrics learning on clicks learning on clicks