Health Cards for Consumer Health Search Jimmy 1,2 , Guido Zuccon 1 , - PowerPoint PPT Presentation

Health Cards for Consumer Health Search Jimmy 1,2 , Guido Zuccon 1 , Bevan Koopman 3 , & Gianluca Demartini 1 1 University of Queensland (UQ), Australia 2 University of Surabaya (UBAYA), Indonesia 3 AEHRC / CSIRO , Australia Presented at: 42 nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’19) July 21–25, 2019 Paris, France

Consumer Health Search (CHS) • CHS: people searching for health advice online . 59% of U.S. adults has searched online for health information (Fox & Duggan, 2013). • Search results strongly bias people ’s health decisions (Pogacar, 2017). • People struggle to understand health search results (Alpay, 2009). • 59% of self-diagnosers decided NOT to confirm their condition with a health professional (Fox & Duggan, 2013). � 2

Assisting Users in Understanding Health Search Results • Health cards have been used by commercial search engines to present coherent, easy to understand and trustworthy health information. • The appearance of a health card is triggered by queries that contains a health condition name or its aliases . • Our study investigated the benefits of health cards for broader search tasks than only to know more about a health condition. 3

RQ1: Are Health Cards Beneficial for CHS? 1. Selected as a source of information 2. Increase the number of correct answers 3. Reduce the time needed 4. Reduce the effort required 5. Reduce the perceived workload 6. Improve the user’s satisfaction 4

RQ2: How Does the Benefit Vary Across Search Intents? We considered two types of search scenario: • Factual scenarios consider search tasks related to a known health condition . E.g., Scenario: Your physiotherapist has mentioned you may have pelvic inflammatory disease and suggested you to go to a doctor. Task: Find out more information about how this disease can be treated. • Intellectual scenarios consider search tasks based on symptoms . E.g., Scenario: It’s few days now that you have been getting hiccups after eating. You felt you eat enough every time, in fact, you felt full. At the same time, you feel something in the back of your throat: like if you had a bump or lump. Task: Find out what you may have and when its time to make an appointment with a doctor. 5

Methods: User Study • 48 Participants x 8 health search tasks= 384 Data Points. • Participants worked on tasks using 2 types of user interface: 1. with a health card Start Perception Exit questionnaire questionnaire Informed 2. Without health card Consent and instructions Search scenario Finish Demographic User experience questionnaire questionnaire Yes No Finish all tasks? 6

User Interface with a Health Card 7

Impact of Health Cards on Search Behaviour On average, participants spent 55 % of their time to observe the health cards 8

Impact of Health Cards on Search Behaviour (2) Participants tend to consider health cards earlier in a session 100 Attention on Snippets Percentage of participants 75 50 25 Attention on Health Cards 0 Progress in the session start end 9

RQ1: The Benefits of Health Cards (1) Selected as a source of information Six Measurements: 49% 1. Selected Cards Selected 2. Correctness 3. Time 51% 4. E ff ort Cards Not Selected 5. Workload 6. Satisfaction 0 10 20 30 40 50 There was no significant difference between the number of tasks completed with and without selecting information from health cards. 10

Why Participants Did Not Select Information from Health Cards? • 28% (13) of participants never selected information from the health cards! • 94% of participants had searched online for health information but 41% of them never noticed health cards prior to this study. 11

RQ1: The Benefits of Health Cards (2) Increase the number of correct answers Six Measurements: 2.54 1. Selected Cards Selected 2. Correctness 2.26 3. Time Cards Not Selected 4. E ff ort 2.38 5. Workload No Cards 6. Satisfaction 0 1 2 Correctness The number of correct answers are comparable across conditions. 12

RQ1: The Benefits of Health Cards (3) Reduce the time needed Six Measurements: 231sec Cards Selected 1. Selected 2. Correctness 242sec 3. Time Cards Not Selected 4. E ff ort 240sec No Cards 5. Workload 6. Satisfaction 0 50 100 150 200 250 Duration The average times needed to complete a task are comparable across conditions. 13

RQ1: The Benefits of Health Cards (4) Reduce the Spent Effort Six Measurements: * 2.91 1. Selected Cards Selected 2. Correctness 3.39 3. Time Cards Not Selected 4. E ff ort * 3.61 5. Workload No Cards 6. Satisfaction 0 1 2 3 UserClicks The number of clicks is significantly less when information from health cards was selected. 14

RQ1: The Benefits of Health Cards (5) Reduce the perceived workload Six Measurements: 3.86 1. Selected Cards Selected 2. Correctness 3.69 3. Time Cards Not Selected 4. Effort 3.59 5. Workload No Cards 6. Satisfaction (harder) (easier) 0 1 2 3 4 Workload There were no significance differences in the level of perceived workload across conditions 15

RQ1: The Benefits of Health Cards (6) Improve the user’s satisfaction Six Measurements: * 3.8 Cards Selected 1. Selected 2. Correctness 3.59 Cards Not Selected 3. Time * 4. Effort 3.45 No Cards 5. Workload 6. Satisfaction 0 1 2 3 Satisfaction Participants felt significantly more satisfied with their own answers when selecting information from health cards to complete a search task. 16

Health Cards Bridged the Gap of Prior Knowledge p=0.0453* p=0.0346* p=0.122 3 3 3 • Participants prior knowledge 2.82 2.8 2.76 2.61 2.56 positively correlates with correctness 2.48 2.42 2.24 2.22 in their answers. This is inline with 2 2.08 Correctness findings by Hu & Haake (2010). • Selecting health cards helps 1 bridging the gap between knowledgable and less 0 knowledgable participants. No Cards Cards Not Selected Cards Selected none little some great deal Prior Knowledge 17

RQ2: Benefits Across Search Intents? Intellectual: Factual: 4.0 A B C more Factual Intellectual p = 0.055 100 Number of links followed effort Intellectual 100 46.88 100 55.21 % of scenarios Duration (Seconds) 260 1. Selected 1. Selected 75 Factual 3.5 50 240 53.12 44.79 2. Correctness 2. Correctness 25 220 Factual p = 0.035 0 3.0 No Yes No Yes less 3. Time Health Cards Displayed 3. Time 200 Intellectual p = 0.015 p = 0.163 effort Cards selected No Yes Search results only Health cards Search results only Health cards D E F 4. Effort 4. Effort 4.0 (easier) 4.0 (more) p = 0.014 p = 0.007 p = 0.028 3.9 Correctness (0 − 3) 3.9 Factual Satisfaction (1 − 5) 2.7 Workload (1 − 5) 5. Workload 5. Workload Factual 3.8 3.8 Factual p = 0.363 2.5 3.7 Intellectual 3.7 6. Satisfaction 6. Satisfaction p = 0.422 3.6 3.6 Intellectual 2.3 Intellectual (less) 3.5 p = 0.825 (harder) 3.5 Search results only Health cards Search results only Health cards Search results only Health cards Benefits of health cards were found to be significant for Factual Tasks. Health cards provide no significant benefits for Intellectual tasks 18

Health Cards Alone Are Generally Insufficient 50 48 44 40 36 33 30 Frequency 20 15 10 10 0 Factual Intellectual Health Cards Health Cards & WebPage WebPage Source of Information 19

Conclusions (1) • Overall, presenting health cards reduced the effort spent and improved the user’s satisfaction . • Health cards helped the less knowledgeable to perform as effective as the knowledgeable (in terms of correctness). • Health cards were significantly beneficial for well-defined health search tasks (Factual) . • In contrast, health cards provided no significant benefits for “exploratory” health search tasks (Intellectual) . 20

Conclusions (2) • 28% of participants never used any of the presented health cards . • While 94% of participants had searched for health information online, yet 41% of participants never experienced health cards prior to this study. • Lack of user engagement with health cards may leave the benefits of health cards unreaped. 21

Questions? For further details: http://ielab.io/publications/jimmy-2019-healthcard Acknowledgement • Jimmy is sponsored by the Indonesia Endowment Fund for Education (Lembaga Pengelola Dana Pendidikan / LPDP)(20151022014644). • Guido Zuccon is the recipient of an Australian Research Council DECRA Research Fellowship (DE180101579). • This work is supported in part by a Google Faculty Award. • Thank you to SIGIR for the travel scholarship.