Incorporating Clicks, Attention and Satisfaction into a SERP - PowerPoint PPT Presentation

Background Motivation Model & Metric Experimental Setup Results Summary Incorporating Clicks, Attention and Satisfaction into a SERP Evaluation Model Aleksandr Chuklin ¶ , § Maarten de Rijke § chuklin@google.com derijke@uva.nl ¶ Google Research Europe § University of Amsterdam AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 1

Background

Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Main problem Combining relevance of individual SERP items ( R k ) into a whole-page metric. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 3

Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples document 1 document 2 document 3 document 4 document 5 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4

Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples Precision at N: document 1 N P @ N = 1 � R k document 2 N k =1 document 3 document 4 document 5 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4

Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples Precision at N: document 1 N P @ N = 1 � R k document 2 N k =1 document 3 Discounted Cumulative Gain (DCG): document 4 N 1 � DCG @ N = log 2 (1 + k ) · R k document 5 k =1 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4

Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples Precision at N: document 1 N P @ N = 1 � R k document 2 N k =1 document 3 Discounted Cumulative Gain (DCG): document 4 N 1 � DCG @ N = log 2 (1 + k ) · R k document 5 k =1 Model-Based Metrics (Chuklin et al. 2013): N � Utility @ N = P ( C k = 1) · R k k =1 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4

Background Motivation Model & Metric Experimental Setup Results Summary Main Goal of This Paper Better measure for SERP utility Namely, improve this (Chuklin et al. 2013): N � P ( C k = 1) · R k k =1 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 5

Motivation

Background Motivation Model & Metric Experimental Setup Results Summary Complex Heterogeneous SERPs AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 7

Background Motivation Model & Metric Experimental Setup Results Summary Motivation 1: Non-Trivial Attention Patterns 1 3 4 2 5 6 7 8 9 Image credits: F. Diaz, R.W. White, G. Buscher, and D. Liebling. Robust models of mouse movement on dynamic web search results pages. In CIKM , 2013. ACM Press AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 8 1452

Background Motivation Model & Metric Experimental Setup Results Summary Motivation 2: Satisfaction Without Clicks High direct page utility (measured by DCG or ERR) leads to higher abandonment rate (SERPs with no clicks) direct page utility Image credits: from A. Chuklin and P. Serdyukov. Good abandonments in factoid queries. In WWW , 2012. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 9

Background Motivation Model & Metric Experimental Setup Results Summary Problems of Existing Models and Evaluation Metrics AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10

Background Motivation Model & Metric Experimental Setup Results Summary Problems of Existing Models and Evaluation Metrics existing models mostly do not model non-trivial user attention patterns AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10

Background Motivation Model & Metric Experimental Setup Results Summary Problems of Existing Models and Evaluation Metrics existing models mostly do not model non-trivial user attention patterns existing models do not use explicit user satisfaction data AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10

Model & Metric

Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP 𝜒 & 𝜒 ) 𝜒 * 𝐹 & 𝐹 ) 𝐹 * … 𝐷 & 𝐷 ) 𝐷 * Utility 𝑇 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 12

Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP 𝜒 & 𝜒 ) 𝜒 * 𝐹 & 𝐹 ) 𝐹 * … 𝐷 & 𝐷 ) 𝐷 * Utility 𝑇 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 13

Background Motivation Model & Metric Experimental Setup Results Summary Click Model Examination assumption : click happens only when an item was examined and attractive: P ( C k = 1) = P ( E k = 1) · P ( C k = 1 | E k = 1) AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 14

Background Motivation Model & Metric Experimental Setup Results Summary Click Model Examination assumption : click happens only when an item was examined and attractive: P ( C k = 1) = P ( E k = 1) · P ( C k = 1 | E k = 1) N.B. Here we assume that P ( C k = 1 | E k = 1) = α ( � R k ) where � R k comes from the raters and α is a logistic function. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 14

Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP 𝜒 & 𝜒 ) 𝜒 * 𝐹 & 𝐹 ) 𝐹 * … 𝐷 & 𝐷 ) 𝐷 * Utility 𝑇 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 15

Background Motivation Model & Metric Experimental Setup Results Summary Attention (Examination) Model Logistic regression model: P ( E k = 1) = ε ( � ϕ k ) , where � ϕ k is a vector of features for SERP item k . AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16

Background Motivation Model & Metric Experimental Setup Results Summary Attention (Examination) Model Logistic regression model: P ( E k = 1) = ε ( � ϕ k ) , where � ϕ k is a vector of features for SERP item k . Feature group Features # of features rank user-perceived rank of the SERP item 1 (can be different from k ) AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16

Background Motivation Model & Metric Experimental Setup Results Summary Attention (Examination) Model Logistic regression model: P ( E k = 1) = ε ( � ϕ k ) , where � ϕ k is a vector of features for SERP item k . Feature group Features # of features rank user-perceived rank of the SERP item 1 (can be different from k ) CSS classes SERP item type (Web, News, 10 Weather, Currency, Knowledge Panel, etc.) geometry offset from the top, first or second col- 5 umn (binary), width ( w ), height ( h ), w × h AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16

Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP 𝜒 & 𝜒 ) 𝜒 * 𝐹 & 𝐹 ) 𝐹 * … 𝐷 & 𝐷 ) 𝐷 * Utility 𝑇 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 17

Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18

Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention ; AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18

Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention ; P ( S = 1) = σ ( τ 0 + U ) = AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18

Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention ; P ( S = 1) = σ ( τ 0 + U ) = � � � � P ( E k = 1) u d ( � P ( C k = 1) u r ( � σ τ 0 + D k ) + R k ) k k where � D k and � R k are ratings assigned by the raters for direct snippet relevance and result relevance respectively. u d and u r are linear functions of rating histograms. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18

Background Motivation Model & Metric Experimental Setup Results Summary The CAS Metric Utility that determines the satisfaction probability: � � P ( E k = 1) u d ( � P ( C k = 1) u r ( � U = D k ) + R k ) k k AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 19