Trade Studies A Hypothetical Example 1 Human Factors Research to - PowerPoint PPT Presentation

Trade Studies A Hypothetical Example 1

Human Factors Research to Inform the Human-Machine Systems Engineering Process Needs, Problems, Opportunities Generalizable Research Question(s) Hypothesis Formulation Operation,Test Analysis & Evaluation Research Design HMS: Humans, Data Collection Users, Requirements Machines, Operators, Processes Data Analysis & Subject Matter HFE Principles (Model, Mockup, Hypothesis Testing Experts & Guidelines Prototype, Product) Interpretation & Application of Results Implementation Design Design Specifications 2

Human Factors Trade Studies to Inform the Human-Machine Systems Engineering Process Needs, Problems, Opportunities Operation,Test Analysis & Evaluation Design Options HMS: Humans, Users, Requirements Machines, Prototype Construction Operators, Processes Subject Matter HFE Principles (Model, Mockup, Experts & Guidelines Study Design Prototype, Product) Data Collection Implementation Design Data Analysis Interpretation & Application of Results Design Specifications 3

Conducting a Trade Study: An Example ● NB: 1)This example is hypothetical, but based on the instructor's experience with real trade studies. 2)It is probably more elaborate than most trade studies, but it illustrates what can be done when right answers to the questions are important. 4

Design Question For a digital camera UI, which menu geometry configuration yields the best speed, accuracy, and user satisfaction? ● Linear Horizontal? ● Linear Vertical? ● Circular? 5

Menu Geometry Design Options A: Linear Horizontal B: Linear Vertical C: Circular Camera Close Settings Camera Settings Playback Flash Flash Timer Playback Timer Close Camera Settings Flash Timer Playback Close 6

Menu Geometry Design Options A: Linear Horizontal B: Linear Vertical C: Circular Camera Close Settings Camera Settings Playback Flash Flash Timer Playback Timer Close Camera Settings Flash Timer Playback Close Settings Quality Close Red Eye Quality Quality Red Eye Red Eye Rev Time Beep Beep Beep No Card No Card Rev Time Rev Time No Card Quality Red Eye Beep No Card Rev Time Close Settings Close Close Settings Settings Beep On Off Close On On Off Close Off Beep Settings Beep Close Settings 7

Study Design ● Participants ● Apparatus ● Data Collection ● Data Analysis ● Interpretation of Results 8

Study Design Participants: representative sample of 12 users ● 6 female, 6 male – various ages – novices & experts – 9

Study Design Participants: representative sample of 12 users ● 6 female, 6 male – various ages – novices & experts – Apparatus: ● 10

Apparatus Options 1. Fully functional prototype camera – most expensive – longest time to build – highest fidelity 2. Part-functional prototype (e.g., just display with menu and a few controls) – less expensive – less time consuming – moderate fidelity (but interactive) 3. PowerPoint slides with links – minimal cost – quick – low fidelity (limited interaction) 4. Paper – minimal cost – quick? – lowest fidelity (not interactive) 11

Study Design Participants: representative sample of 12 users ● 6 female, 6 male – various ages – novices & experts – Apparatus: part-functional prototype camera ● display – menu/photos – D-pad control – fire (shutter-release) button – Data Collection: ● each participant uses 3 menus (random order) and for each – ● hears explanation, instructions ● performs series of tasks (speed & accuracy recorded) ● answers SUS questionnaire ● answers comparison questionnaire: Hardest/Easiest?, etc. ● answers interview questions: 3 best things? 3 worst things? Comments? Suggestions? ranks 3 menus, best to worst – 12

Data Collection Participant Tasks Participant Menu Order 1.Set Date and Time 1. CHV 2. VCH 2.Turn on Date Display 3. HCV 3.Turn on Flash 4. VCH 5. CVH 4.Select Red-Eye Reduction 6. CHV 5.Quickly “take” 3 photos 7. CVH 6.Review photos on display 8. VHC 9. HVC 7.Delete 2 nd photo 10. HCV (Prototype records time & action 11. HVC in log file for later analysis.) 12. VHC 13

Study Design Participants: representative sample of 12 users ● 6 female, 6 male – various ages – novices & experts – Apparatus: part-functional prototype camera ● display – menu/photos – D-pad control – fire (shutter-release) button – Data Collection: ● each participant uses 3 menus (random order) and for each – ● hears explanation, instructions ● performs series of tasks (speed & accuracy recorded) ● answers SUS questionnaire ● answers comparison questionnaire: Hardest/Easiest?, etc. ● answers interview questions: 3 best things? 3 worst things? Comments? Suggestions? ranks 3 menus, best to worst – Data Analysis → Results: ● Descriptive statistics – Box-and-whisker charts – Paired t-tests – 14

Data Analysis: Data Completon Time (sec) Data Data p01 p02 p03 p04 p05 p06 p07 p08 p09 p10 p11 p12 Horiz Completon Time (sec) 14.5 14.4 14.0 22.6 21.4 19.4 21.7 25.8 25.3 20.4 12.1 20.9 Vert Completon Time (sec) 14.5 14.8 14.5 14.2 16.0 16.7 18.0 17.7 16.9 12.5 16.7 17.0 Circ Completon Time (sec) 13.3 26.0 19.6 17.4 15.5 14.6 24.7 19.0 21.3 27.8 25.2 17.7 Errors Data Data p01 p02 p03 p04 p05 p06 p07 p08 p09 p10 p11 p12 Horiz Errors 4 6 2 3 3 3 5 1 3 3 1 2 Vert Errors 0 2 2 0 2 2 3 0 0 2 2 4 Circ Errors 7 2 2 10 0 3 0 8 6 2 4 8 SUS Overall Score Data Data p01 p02 p03 p04 p05 p06 p07 p08 p09 p10 p11 p12 Horiz SUS Overall Score 60 64 67 62 83 87 78 74 71 64 66 63 Vert SUS Overall Score 87 98 100 90 79 92 72 82 91 86 88 85 Circ SUS Overall Score 96 53 69 76 79 78 80 43 72 58 77 77 Best/Worst Comparisons Data Data p01 p02 p03 p04 p05 p06 p07 p08 p09 p10 p11 p12 Comparison: Easiest to use Vert Circ Horiz Horiz Vert Vert Horiz Circ Vert Vert Horiz Vert Comparison: Hardest to use Horiz Horiz Circ Circ Horiz Circ Circ Horiz Circ Horiz Circ Circ Comparison: Most efcient / Fastest Vert Vert Vert Circ Circ Horiz Horiz Vert Vert Vert Vert Vert Comparison: Least efcient / Slowest Circ Horiz Horiz Horiz Horiz Circ Circ Circ Horiz Circ Horiz Horiz Comparison: Easiest to learn Horiz Vert Horiz Circ Vert Horiz Circ Horiz Circ Vert Vert Vert Comparison: Hardest to learn Circ Circ Circ Horiz Horiz Circ Horiz Circ Horiz Circ Circ Horiz Comparison: Most intuitve Vert Vert Horiz Vert Vert Horiz Circ Horiz Circ Vert Vert Vert Comparison: Least intuitve Circ Circ Circ Circ Horiz Circ Horiz Vert Horiz Circ Circ Circ Rankings Data Data p01 p02 p03 p04 p05 p06 p07 p08 p09 p10 p11 p12 Vert Circ Vert Horiz Vert Vert Circ Vert Vert Vert Horiz Vert Ranking: 1 st Ranking: 2 nd Horiz Vert Horiz Vert Circ Horiz Vert Horiz Horiz Circ Vert Horiz Ranking: 3 rd Circ Horiz Circ Circ Horiz Circ Horiz Circ Circ Horiz Circ Circ 15

Data Analysis & Results: Task Completion Time Completion Time 30 Time (sec): min, 1st qtl, median, 3rd qtl, max 25 20 15 10 5 0 Horiz Vert Circ Completon Time (sec) Statstcs Data n min 1 st qtl mean median 3 rd qtl max Horiz Completon Time (sec) Horiz 12 12.1 14.5 19.4 20.7 21.9 25.8 Vert Completon Time (sec) Vert 12 12.5 14.5 15.8 16.3 17.0 18.0 Circ Completon Time (sec) Circ 12 13.3 16.9 20.2 19.3 24.8 27.8 Completon Time (sec) Paired t-Tests |t stat| > t(α=0.05, Diferences & Paired t-Tests n mean std. dev. t statstc t(α=0.05, ν=n-1) p(2-tailed) ν=n-1) 2.900 True 0.0144 Horiz – Vert 12 3.6 4.3 2.201 -0.391 0.7033 Horiz – Circ 12 -0.8 7.1 2.201 False -2.852 True 0.0157 Vert – Circ 12 -4.4 5.3 2.201 16

Data Analysis & Results: Errors Errors 12 Errors: min, 1st qtl, median, 3rd qtl, max 10 8 6 4 2 0 Horiz Vert Circ Errors Statstcs 1 st qtl 3 rd qtl Data n min mean median max Horiz Errors Horiz 12 1.0 2.0 3.0 3.0 3.3 6.0 Vert Errors Vert 12 0.0 0.0 1.6 2.0 2.0 4.0 Circ Errors Circ 12 0.0 2.0 4.3 3.5 7.3 10.0 Errors Paired t-Tests |t stat| > t(α=0.05, Diferences & Paired t-Tests n mean std. dev. t statstc t(α=0.05, ν=n-1) p(2-tailed) ν=n-1) 2.609 True 0.0243 Horiz – Vert 12 1.4 1.9 2.201 -1.109 0.2909 Horiz – Circ 12 -1.3 4.2 2.201 False -2.279 True 0.0436 Vert – Circ 12 -2.8 4.2 2.201 17

Data Analysis & Results: Errors SUS Overall Score 120 SUS Score: min, 1st qtl, median, 3rd qtl, max 100 80 60 40 20 0 Horiz Vert Circ SUS Overall Score Statstcs Data n min 1 st qtl mean median 3 rd qtl max Horiz SUS Overall Score Horiz 12 60.0 63.8 69.9 66.5 75.0 87.0 Vert SUS Overall Score Vert 12 72.0 84.3 87.5 87.5 91.3 100.0 Circ SUS Overall Score Circ 12 43.0 66.3 71.5 76.5 78.3 96.0 SUS Overall Score Paired t-Tests |t stat| > t(α=0.05, Diferences & Paired t-Tests n mean std. dev. t statstc t(α=0.05, ν=n-1) p(2-tailed) ν=n-1) -4.472 True 0.0009 Horiz – Vert 12 -17.6 13.6 2.201 -0.332 0.7462 Horiz – Circ 12 -1.6 16.5 2.201 False 3.199 True 0.0085 Vert – Circ 12 16.0 17.3 2.201 18