Human-Robot Interaction Elective in Artificial Intelligence - PDF document

Human-Robot Interaction Elective in Artificial Intelligence Lecture 10 – User Evaluation Luca Iocchi DIAG, Sapienza University of Rome, Italy HRI Evaluation • HRI requires evaluation methods involving users • Evaluation with users is not easy, because it always involves some subjectivity • Many issues: • Trained vs. untrained users • Recruited vs. non-recruited users • Experimental protocol 2 L. Iocchi - User Evaluation

HRI Evaluation • Different kinds of interactions • Supervisor • Operator • Teammate (trained and knowledgeable about the robot) • Bystander (no trained and knowledgeable about the robot) Bystanders need information about what the robot can do! 3 L. Iocchi - User Evaluation HRI Experimental factors 1. Subjects 2. Variables 3. Hypothesis 4. Experimental protocol 4 L. Iocchi - User Evaluation

HRI Evaluation subjects Selection of subjects is a crucial factor. • sufficient sample set • independent identically distributed • … 5 L. Iocchi - User Evaluation HRI Evaluation variables Variables: quantities to control and measure • Independent variables (conditions) controlled during the experiment • Dependent variable (condition) measured in the experiment 6 L. Iocchi - User Evaluation

HRI Evaluation measures • Typical HRI/HCI evaluation measures • Effectiveness (how much the task is completed) • Efficiency (how much time/resources are needed) • User satisfaction (user feelings about the process) 7 L. Iocchi - User Evaluation HRI Evaluation hypotheses Hypotheses or research questions H. Variable H is affected by variables/conditions X, Y, and Z Q. How variable H is affected by variables/conditions X, Y, and Z? Null hypothesis: conditions X, Y, Z do not affect H 8 L. Iocchi - User Evaluation

HRI Evaluation example Research question: how a user reacts when s/he encounters a robot in the corridor with respect to its motion behavior and the use of voice to acknowledge the presence of the user? Independent variables (conditions) 1. Robot motion behavior (possible values: straight, slow_down_and_move_right) 2. Robot vocalization (possible values: no speech, speech explaining that the robot would avoid the user) Dependent variable 1. User comfort: how much the user trusts the robot (i.e., s/he continues walking in the corridor in a natural way) 9 L. Iocchi - User Evaluation HRI Evaluation protocol • Evaluation protocol • Within-subject (same user exposed to different conditions, evaluating variation over time of user performance) • Between-subject (different users exposed to different conditions, evaluating differences between users) 10 L. Iocchi - User Evaluation

HRI Evaluation protocol Example of Within-subject protocol: 1. Assess a baseline performance on a basic system (e.g., basic HRI) with a group of users 2. Evaluate a more sophisticated system (e.g., HRI with a new feature A) with same group of users Result = improved performance of the new feature A Advantage: it requires less users Disadvantage: possibly biased results due to dependencies between different phases of the tests for the same user (e.g., fatigue and practice) 11 L. Iocchi - User Evaluation HRI Evaluation protocol Example of Between-subject protocol 1. Assess a baseline performance on a basic system (e.g., basic HRI) with a first group of users 2. Evaluate a more sophisticated system (e.g., HRI with a new feature A) with a second group of users Result = improved performance of the new feature A Advantage: it overcomes biased results Disadvantage: it requires more users, results may be affected by individual variability 12 L. Iocchi - User Evaluation

HRI Analysis of data Analysis of quantitative data Several statistical techniques are used: • t-test • ANOVA • … Choosing the right test to perform is a key issue 13 L. Iocchi - User Evaluation HRI Analysis General approach: • Specify the null hypothesis (there is no difference) • Collect data • Compute the p-value (using an appropriate tool) • Draw a conclusion p<0.05 – statistically significant (*) p<0.01 – highly statistically significant (**) p<0.001 – extremely statistically significant (***) 14 L. Iocchi - User Evaluation

HRI Analysis Student’s t-test is a statistical test to verify a statistical significance between two sets of measures. Example: measure the time to complete an HRI task in two different conditions A and B. 15 L. Iocchi - User Evaluation HRI Analysis Example: measure the time to complete an HRI task in two different conditions A and B. Repeat the test with condition A nA times and collect values T A = { t 1 A } A , …, t nA Repeat the test with condition B nB times and collect values T B = { t 1 B } B , …, t nB Compute the mean values µ A and µ B (with corresponding standard deviations) Is the difference significant or it is due to randomness? 16 L. Iocchi - User Evaluation

HRI Analysis Example: measure the time to complete an HRI task in two different conditions A and B. Use a Between-subject protocol to collect data T A and T B Perform an independent (unpaired) t-test to assess significance in the results (rejection of null hypothesis). How to compute: Excel, R 17 L. Iocchi - User Evaluation HRI Analysis Excel T-test - Positive if P-value < 0.05 Values Values Group A Group B 30.5 32 Mean A 31.05 31.5 28 Stddev A 1.83 34 31 Mean B 29.75 31 28 Stddev B 1.30 30.5 30.5 31 30 t-test P-value 0.08549 34 28.5 … … 18 L. Iocchi - User Evaluation

HRI Analysis Example: measure the time to complete an HRI task in three different conditions A, B, and C. Collect data T A , T B , and T C Perform a two-way analysis of variance (ANOVA) How to compute: Excel, R 19 L. Iocchi - User Evaluation