yours is better
play

Yours is better! Participant Response Bias in HCI Nicola Dell 1 - PowerPoint PPT Presentation

Mar 24, 2023 •468 likes •966 views

Yours is better! Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of Washington 2 San Jose State University 3 Microsoft Research India HCI is expanding and

  1. “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of Washington 2 San Jose State University 3 Microsoft Research India

  2. HCI is expanding and diversifying 2

  3. HCI is expanding and diversifying There are large differences between the people being studied and those studying them 3

  4. Demand characteristics are aspects of a study that convey the investigator’s hypothesis to participants 4

  5. Demand characteristics may result in participant response bias 5

  6. Demand characteristics may result in participant response bias 6

  7. Demand characteristics may result in participant response bias Response bias can influence the outcome of studies with human participants 7

  8. What about demand characteristics in HCI? 8

  9. How do social and demographic factors influence demand characteristics? 9

  10. Let’s design an experiment to explore demand characteristics in HCI 10

  11. Let’s design an experiment to explore demand characteristics in HCI Completely identical video clips 11

  12. Observe participants interacting with technological devices 12

  13. Interview Script “Thank you for participating in my experiment. I am a computer scientist and I'm trying to improve video players on mobile phones. I want you to watch a short video on these two phones and tell me which one looks better, or if they look the same. The same video will play on both phones, but this phone uses my new player [indicate phone]. Please tell me your honest opinion and please concentrate because I will play each video only once. Do you have any questions? Ok, watch this one first. This one uses my new player [play video]. Now watch this one [play video]. Which one do you think looks better or do they look the same? Why? Thanks very much!” 13

  14. Two different interviewers Foreign interviewer and local interviewer 14

  15. Participants: Auto rickshaw drivers 250 male participants aged 19 to 72 (M=37, SD=11.2) 15

  16. Conditions tested with the identical video clip Rickshaw Drivers Foreign Interviewer 50 participants (+ translator) Local 50 participants Interviewer 16

  17. Participant Responses Preferred interviewer’s video Preferred other video Thought videos looked the same 50 40 Number of people 35 30 20 8 7 10 0 Foreign Interviewer 17

  18. Participant Responses Preferred interviewer’s video Preferred other video Thought videos looked the same 50 50 40 40 Number of people Number of people 35 30 30 22 19 20 20 9 8 7 10 10 0 0 Foreign Interviewer Local Interviewer 18

  19. Demand characteristics can result in substantial participant response bias 19

  20. Demand characteristics can result in substantial participant response bias Social and demographic factors influence the amount of response bias 20

  21. What happens if the two clips are different? 21

  22. What happens if the two clips are different? 21

  23. What happens if the two clips are different? 21

  24. What happens if the two clips are different? Interviewer always associated to the low-quality clip 21

  25. Conditions tested with the degraded video clip Rickshaw Drivers 50 participants Without association Foreign Interviewer 50 participants (+ translator) Local 50 participants Interviewer 22

  26. Participant Responses Preferred low-quality video Preferred high-quality video Thought videos looked the same 50 Number of people 38 40 30 20 8 10 4 0 Without association 23

  27. Participant Responses Preferred low-quality video Preferred high-quality video Thought videos looked the same 50 50 Number of people Number of people 38 40 40 27 30 30 19 20 20 8 10 10 4 4 0 0 Without association Foreign Interviewer 23

  28. The characteristics of the interviewer can determine the outcome of the study 24

  29. Explaining participant responses 25

  30. Explaining participant responses “The quality of the background color and figures is too light in that player while the quality of the color and graphics is better in [your] one. Will [your] new player be introduced in the market ?” 26

  31. Explaining participant responses “I prefer this one (associated) because the other one wasn't clear at all. This one was fully clear. I can definitely show you the difference if we watch them together. I've given you my honest opinion, so please don't be cross with me if it wasn't the right one.” 27

  32. Detailed and insightful participant comments can’t be taken at face value 28

  33. What about other participant groups? 200 male university students aged 19 to 41 (M=25, SD=3.8) 29

  34. Responses with university students • Identical videos: 2:1 response bias 30

  35. Responses with university students • Identical videos: 2:1 response bias • Degraded videos: order of magnitude difference 30

  36. Responses with university students • Identical videos: 2:1 response bias • Degraded videos: order of magnitude difference • “I feel that in the newer version which you have coded, whenever there was a significant color contrast between two parts of an image, your version was somewhat smoother and less pixelated .” 30

  37. There is still substantial response bias in studies with peers 32

  38. Limitations 33

  39. Limitations • Presence of a translator for some interviews 34

  40. Limitations • Presence of a translator for some interviews • The effects of social status and ethnicity will vary from culture to culture 35

  41. Limitations • Presence of a translator for some interviews • The effects of social status and ethnicity will vary from culture to culture • Many other aspects of demand characteristics 36

  42. What can we do to minimize demand characteristics in HCI? 37

  43. What can we do to minimize demand characteristics in HCI? • Be aware that response bias affects all studies 38

  44. What can we do to minimize demand characteristics in HCI? • Be aware that response bias affects all studies • Dissociate from a particular design or solution 39

  45. What can we do to minimize demand characteristics in HCI? • Be aware that response bias affects all studies • Dissociate from a particular design or solution • Minimize the differences in social status between investigators and participants 40

  46. What can we do to minimize demand characteristics in HCI? • Be aware that response bias affects all studies • Dissociate from a particular design or solution • Minimize the differences in social status between investigators and participants • Use triangulation to validate data collected 41

  47. Demand characteristics influence the outcome of studies with human participants 42

  48. “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of Washington 2 San Jose State University 3 Microsoft Research India nixdell@uw.edu http://cs.washington.edu/homes/nixdell

Recommend

More recommend