Towards Better Measurement of Attention and Satisfaction in Mobile Search Dmitry Lagun , Chih-Hung Hsieh, Dale Webster, Vidhya Navalpakkam
Thanks! Vidhya Navalpakkam Chih-Hung Hsieh Dale Webster 2
Mobile is popular! • 25% of Web page visits come from mobile [Statcounter.com, 2014] • Mobile browsing grew five fold since 2010 (5%) [Statcounter.com, 2014] • One in every 5 search queries is issued from a mobile device [RKG Digital Marketing Report, 2013] 3
Our Study Attention Measurement Satisfaction with Rich Results Knowledge Graph Result 4
Satisfaction with Rich Results on Mobile: Background • Long history of using clicks for measurement of search satisfaction and result relevance [ Joachims et al., SIGIR 2005; Agichtein et al., SIGIR 2006] • Result relevance and implicit indicators (mouse cursor hover, touch & swipe) [ Huang et al., CHI 2011; Lagun et al., SIGIR 2011; Guo et al., SIGIR 2013] • Rich Answers do not require to click and mouse hovers do not exist on mobile – What other implicit metrics can we use to infer result relevance/satisfaction without clicks/hovers? 5
User Study Design • Two Factor (within Subject) KG Not – Relevance KG Relevant Relevant – Presence 5 Tasks KG Present 5 Tasks 5 Tasks 5 Tasks • 20 Search Tasks KG Absent • Users were asked to provide explicit satisfaction score for each task (1-7 scale) 6
User Study Details • Participants – 24 users (diverse background, age, occupation) • Mobile Eye Tracker Setup • Calibration Directly on Phone Screen 7
Can Implicit User Metrics Indicate Answer Relevance? • Page and Task metrics – Time on SERP – Number of Scrolls – Time on Task Knowledge Graph Result • Gaze Metrics – Time on Rich Result (and %) – Total Time below Rich Result (and %) • Viewport Metrics – Time on Rich Result (and %) – Total Time below Rich Result (and %) 8
KG is Not Relevant More Scrolling 10
KG is Relevant Faster Search ( answer is found in KG without a click ) 11
No Impact on User Satisfaction when KG is Not Relevant! 12
Gaze Metrics vs. KG Relevance Not Relevant Relevant More Time Below the KG Result 13
%Viewport Time Below vs. KG Relevance More time on results below 30 Not Relevant KG % Viewport Time Below KG 25 20 15 10 5 0 Not Relevant Relevant 15
Satisfaction with Rich Results: Summary • We can use Page and Viewport metrics to infer KG relevance and satisfaction • No impact on user satisfaction when Not Relevant KG is shown • Users view more results below the KG, when it is Not Relevant 16
Our Study Attention Measurement Satisfaction with Rich Results Knowledge Graph Result 17
Attention Measurement in Search: Background • Eye Tracking – accurate, but limited in scale [ Granka et al., WWW 2004; Buscher et al., SIGIR, CHI 2008- 2010] • Mouse Cursor Tracking – less accurate, but scalable [ Huang et al., CHI 2011, 2012; Lagun et al., SIGIR 2011; Guo et al., CHI 2010, WWW 2012; Navalpakkam et al., WWW 2013] • Viewport Tracking – accurate (???), scalable (on mobile) 18
Viewport Time Calculation: Primer • Display Time = 10 sec • ViewportTime(R1) = ? R1 • Coverage – % of screen area occupied by the result (e.g. Coverage(KG) > Coverage(R1 )) KG • Exposure – % of result area visible on the screen (e.g. Exposure (R2) < 1.0) • ViewportTime(R) = DisplayTime * R2 Coverage (R) * Exposure (R) 19
Can we use Viewport Time to measure time spent on each result? one search result Pearson R = 0.57 Pearson R = 0.69 %Viewport Time Viewport Time Gaze Time %Gaze Time Correlation is high can use Viewport Time to accurately measure time spent on individual search result at scale 20
Are attention patterns similar on desktop and mobile? ? ? 21 Granka et al., WWW 2004
Viewing Time vs. Result Position Why? On desktop: 22 Granka et al., WWW 2004
Short Scroll Effect 23
Short Scroll Effect 24
Short Scroll Effect 25
Do users have position preference when reading on a mobile phone? 26
Conclusions • Viewport and Page metrics can be used to measure Rich Answer Relevance and Satisfaction • Viewport time provides accurate (R=0.69) estimate on time spent on search result • Users prefer to position content on top half of the phone’s screen 27
Results Summary Satisfaction with Attention Measurement Rich Results Pearson R = 0.69 %Viewport Time Viewport ≈ Gaze (on mobile) More results are viewed if %Gaze Time Answer is Not Top half of the Relevant screen receives more Attention Relevant Not Relevant No Impact on User Satisfaction “Short - Scroll” when KG is Not effect Relevant! Granka et al., WWW 2004 Mobile Desktop 28
Recommend
More recommend
