New Techniques for Usability Evaluation of Mobile Systems Jesper - PowerPoint PPT Presentation

New Techniques for Usability Evaluation of Mobile Systems Jesper Kjeldskov & Jan Stage Department of Computer Science Aalborg University Denmark

Background � Mobile technologies and systems… � PDAs, wearables, mobile phones, tablet computers … challenge usability testing methods � Users are physically mobile during use � Use involve activities in physical surroundings � Use context can be difficult to recreate realistically � This paper explore new techniques for usability testing mobile systems in laboratory settings Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 2

Two Evaluation Approaches � Field experiments � Realistic use context � Difficult to control � Complicated data collection � Complex and time consuming � Safety and ethical issues � Laboratory experiments � Experimental control � High quality data collection � Lack of realism Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 3

Laboratory vs. Field � Most usability evaluations of mobile systems are currently conducted in laboratory settings � A recent literature study revealed that… � 41% of mobile HCI research involve evaluation � 71% of this is done in laboratory settings � It is a widely adopted point of view that mobile systems require field evaluations, but… � It is difficult to conduct field evaluations � The added value of testing in the field is unknown � Additional problems come at a high cost (time & effort) Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 4

Overall Research Question How can new techniques for usability tests of � mobile systems increase realism of use in a laboratory setting? … while facilitating systematic data collection in a controlled environment Previous studies: Specialized use contexts � This study: Physical mobility during use � Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 5

Two Experiments Two experiments comparing techniques for lab- and field- � based usability testing of mobile systems were conducted Experiments explored different techniques requiring… � Different levels of physical movement 1. Divided cognitive attention 2. Example application: use of Short Message Service (SMS) � on PDAs and mobile phones Narrow focus on interaction rather than broad focus on use � Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 6

Five Laboratory Techniques… Attention needed to navigate None Conscious None 1. Sitting at a table n/a or standing Body Constant 2. Walking on a 4. Walking at Motion treadmill with constant speed on a constant speed changing track Varying 3. Walking on 5. Walking at a treadmill with varying speed on a varying speed changing track � The experiment: measuring the relative strengths and weaknesses with reference to field testing Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 7

The Laboratory Experiments � 5 conditions (6 test subjects per condition) � Number of usability problems � Performance (task completion time) � Subjective workload (NASA TLX) Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 8

The Field Experiment � 1 condition: walking in a pedestrian street (6 test subjects) � Number of usability problems � Performance (task completion time) � Subjective workload (NASA TLX) Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 9

Findings (1) Mean number of usability problems identified by each technique Problems detected 14 1. Sitting at a table 12 2. Walking on a treadmill with constant speed 10 10 ,8 8 3. Walking on a treadmill with varying speed 7,5 6 6 ,7 6 ,7 6 ,3 4. Walking at constant speed on a changing track 5,2 4 5. Walking at varying speed on a changing track 2 0 6. walking in a pedestrian street 1 2 3 4 5 6 Technique � More problems found on average when seated at a table � Statistical significance � Lab techniques with physical movement comparable to field evaluation Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 10

Findings (2) Number of identified usability problems categorized by severity Techniques Total Lab 1 Lab 2 Lab 3 Lab 4 Lab 5 Field Critical 4 4 3 4 3 3 4 Serious 11 11 9 9 9 8 17 Cosmetic 19 8 8 8 6 12 32 Total 34 23 20 21 18 23 53 � No technique identified all problems � Most problems found when seated at table (34) � Comparable numbers of critical problems found (3-4) Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 11

Findings (2) Number of identified usability problems categorized by severity Techniques Total Lab 1 Lab 2 Lab 3 Lab 4 Lab 5 Field Critical 4 4 3 4 3 3 4 Serious 11 11 9 9 9 8 17 Cosmetic 19 8 8 8 6 12 32 Total 34 23 20 21 18 23 53 � More than double the number of cosmetic problems were found while seated compared to the other lab techniques Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 12

Findings (2) Number of identified usability problems categorized by severity Techniques Total Lab 1 Lab 2 Lab 3 Lab 4 Lab 5 Field Critical 4 4 3 4 3 3 4 Serious 11 11 9 9 9 8 17 Cosmetic 19 8 8 8 6 12 32 Total 34 23 20 21 18 23 53 � Only 3 out of 4 critical usability problems were identified on basis of the field technique Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 13

Findings (3) Subjective experience of workload with the different techniques Techniques Lab 1 Lab 2 Lab 3 Lab 4 Lab 5 Field Mental demands 29 75 204 126 185 148 Physical demands 92 117 112 118 127 194 Effort 52 163 106 228 178 186 Overall workload 27 35 48 55 48 54 � Sitting at a table (lab 1) required significantly less mental activity compared to all other techniques but lab 2 � Overall, sitting or walking at constant speed is experienced significantly less demanding than any other technique Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 14

Discussion (1) � Seating test subjects at a table… superior? � Supported the identification of most usability problems � Difference mostly accounted for by cosmetic problems � Less workload facilitated more thinking-aloud � Relevance of cosmetic problems can be questioned � Increasing workload in lab � Helped approximating the field condition but resulted in fewer problems identified compared to when being seated � Better focus missing vital problems?? Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 15

Discussion (2) � Usability problems and mobility � Physical motion in the lab triggered unique interaction problems also found in the field � Added value in relation to e.g. layout and button sizes � Data collection in the field � All field tests were recorded with a camcorder � Difficult to capture good images of screen � The “bodyguard” effect � Changing the role of the test monitors? � Mounting small cameras on test subject and device? Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 16

Future work � Cost/benefit of different techniques and settings � Time and effort spent per problem found � When should lab or field tests ideally be applied? � How can field test techniques be improved? � Improving data collection? � Enforcing more experimental control? � Doing something completely different? Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems 17