Testing & Validation Human Workload Modeling for Autonomous - PowerPoint PPT Presentation

Modeling & Simulation, Testing & Validation Human Workload Modeling for Autonomous Ground Vehicles Presen enting ting: Dr. C.J. J. Hutto, Senior Research Scientist, Human Systems Engineering Branch, Georgia Tech Research Institute (GTRI) Dr. Vl Vlad Pop, Former Research Faculty/Research Scientist, GTRI W. Stuart t Michel elson, son, Research Faculty/Research Scientist, GTRI 8/22/2018 GTRI

Modeling & Simulation, Motivation Testing & Validation • Autonomous Ground Systems (AGS) play a significant role in the DoD’s Third Offset Strategy . 8/22/2018 GTRI

Modeling & Simulation, Motivation Testing & Validation • AGS & Human-Automation Interaction Pros Cons Decrease driving workload Introduce new cognitive demands Increase in efficiency Inconsistent attentional demands Improvements in safety New type of driving mgt tasks • To achieve optimal workload in AGS, designers must be able to assess the operator workload levels. 8/22/2018 GTRI

Modeling & Simulation, Background Testing & Validation • Human Workload Modeling: Multiple Resource Theory (MRT)  Wickens (1984) argued that human workload is not the result of one central processing resource, but rather multiple resource channels (Wickens, C. D., 1984)  Tasks can be performed concurrently, but will interfere with each other  Increasing the resource demands exercised by one task will decrease resource availability for another task (typically, with performance degradation) 8/22/2018 GTRI

Modeling & Simulation, Background Testing & Validation • Applying MRT via VCAP Workload Scales Visual Auditory 1.0 Visually Register/Detect (detect occurrence of image) 1.0 Detect/Register Sound (detect occurrence of sound) 3.7 Visually Discriminate (detect visual differences) 2.0 Orient to Sound (general orientation/attention) 4.0 Visually Inspect/Check (discrete inspection/static condition) 4.2 Orient to Sound (selective orientation/attention) 5.0 Visually Locate/Align (selective orientation) 4.3 Verify Auditory Feedback (detect anticipated sound) 5.4 Visually Track/Follow (maintain orientation) 4.9 Interpret Semantic Content (speech) 5.9 Visually Read (symbol) 6.6 Discriminate Sound Characteristics (detect auditory differences) 7.0 Visually Scan/Search/Monitor (continuous/serial inspection) 7.0 Interpret Sound Patterns (pulse rates, etc.) Cognitive Psychomotor 1.0 Automatic (simple association) 1.0 Speech 1.2 Alternative Selection 2.2 Discrete Actuation (button, toggle, trigger) 3.7 Sign/Signal Recognition 2.6 Continuous Adjustive (flight control, sensor control) 4.6 Evaluation/Judgment (consider single aspect) 4.6 Manipulative 5.3 Encoding/Decoding, Recall 5.8 Discrete Adjustive (rotary, thumbwheel, lever position) 6.8 Evaluation/Judgment (consider several aspects) 6.5 Symbolic Production (writing) 7.0 Estimation, Calculation, Conversion 7.0 Serial Discrete Manipulation (keyboard entries)  Consistent with MRT, McCracken and Aldrich (1984) describe workload as a function of several processing resources: visual, cognitive, auditory, and psychomotor (VCAP) 8/22/2018 GTRI

Modeling & Simulation, Background Testing & Validation • Computerized Discrete-Event Simulation (DES) and the Task Network Model Analyst input variables affecting workload: • System design alternatives (e.g., automation, layout, workflow process, etc.) • Crew size, composition/attributes, & division of labor • Scripted, event-driven simulation scenarios, e.g., • Typical mission scenario (demanding but realistic) • Atypical intense situation (time critical, dangerous) DES TN model determinants of workload: • MRT (VCAP) related task demand • Task time distribution variability • Mean, standard deviation, distribution type • Rule-based task release conditions, e.g., • Operator(s) must be capable (qualified/trained), able (reach, see, etc), and available (not busy, not incapacitated, not off duty, etc) • Task sequencing/branching logic • Task beginning and ending effects, e.g., • Make operator(s) “unavailable” • Increase/decrease or start/stop workload metrics • Task queue discipline (FIFO, LIFO, HAF) 8/22/2018 GTRI

Modeling & Simulation, Surrogate AGS Example Testing & Validation • Modeling Human Workload in a Toy Example GM Cadillac Super Cruise Tesla Model S Autopilot http://www.thedrive.com/tech/17083/the-battle-for-best-semi-autonomous-system-tesla-autopilot-vs-gm-supercruise-head-to-head 8/22/2018 GTRI

Modeling & Simulation, Systems Comparison Testing & Validation • Primary Differences  Domain (GPS-limited to pre-mapped roads vs intuitive cruise control-like conditions)  Updates (OTA w/ “Fleet Learning” vs Quarterly dealer -managed updates and company-only learning)  Ease of System Activation/Engagement (and [accidental] disengagement)  Driving Mode Awareness, and Mode Transition Awareness  Hands-on vs Hands-off, and Driver Monitoring System  Lane Changing, Lane Keeping, and Radar Handling for Cut-ins GM Cadillac Super Cruise Tesla Model S Autopilot 8/22/2018 GTRI

Modeling & Simulation, Systems Comparison Testing & Validation • Primary Differences  Driving Situation Awareness Tesla Model S Autopilot GM Cadillac Super Cruise 8/22/2018 GTRI

Modeling & Simulation, Surrogate Example Results Testing & Validation • Found greater visual workload with the system indicating that the automation is engaged by changing the color of an icon versus the additional presence of light or image. Visual Workload Manual Driving Tesla Autopilot Cadillac Super Cruise™ 14.0 12.0 Visual Workload Rating 10.0 8.0 6.0 4.0 2.0 0.0 Engaging Confirming "Driving" with Automation Manual Automation Engagement Automation Disengagement Disengagement 8/22/2018 GTRI

Modeling & Simulation, Surrogate Example Results Testing & Validation • The hands free system resulted in lower workload than the hands on system Psychomotor Workload Manual Driving Tesla Autopilot Cadillac Super Cruise™ 14.0 Psychomotor Workload Rating 12.0 10.0 8.0 6.0 4.0 2.0 0.0 Engaging Confirming "Driving" with Automation Manual Automation Engagement Automation Disengagement Disengagement 8/22/2018 GTRI

Modeling & Simulation, Surrogate Example Results Testing & Validation • Only slight auditory differences between systems Auditory Workload Manual Driving Tesla Autopilot Cadillac Super Cruise™ 14.0 12.0 Auditory Workload Rating 10.0 8.0 6.0 4.0 2.0 0.0 Engaging Confirming "Driving" with Automation Manual Automation Engagement Automation Disengagement Disengagement 8/22/2018 GTRI

Modeling & Simulation, Surrogate Example Results Testing & Validation • Similar cognitive demand for both systems Cognitive Workload Manual Driving Tesla Autopilot Cadillac Super Cruise™ 14.0 12.0 Cognitive Workload Rating 10.0 8.0 6.0 4.0 2.0 0.0 Engaging Confirming "Driving" with Automation Manual Automation Engagement Automation Disengagement Disengagement 8/22/2018 GTRI

Modeling & Simulation, Surrogate Example Results Testing & Validation • Increased workload during automation engagement and automation disengagement but decreased workload while automation was in use Total Workload Manual Driving Tesla Autopilot Cadillac Super Cruise™ 28.0 Total Workload Rating 21.0 14.0 7.0 0.0 Engaging Confirming "Driving" with Automation Manual Automation Engagement Automation Disengagement Disengagement 8/22/2018 GTRI

Modeling & Simulation, Conclusions Testing & Validation • A (purposely) simple example demonstrates several key concepts: – Workload modeling can be useful for objectively comparing systems, comparing design alternatives, or evaluating automation design decisions to help optimize human-machine interaction. – Overall this type of analysis shows where the addition of automation is beneficial to operator workload and where it is not. – The analysis also reveals some differences between the two systems. • M&S approach scales easily to a variety of driving tasks and scenarios, system design alternatives, environmental conditions, etc. • M&S approach extends beyond evaluation of system design; simultaneously gain insights about human and system performance 8/22/2018 GTRI

Modeling & Simulation, Questions Testing & Validation • • Contact the author: Contact the presenter: Mr. Stuart Michelson Dr. C.J. Hutto Stuart.Michelson@gtri.gatech.edu cjhutto@gatech.edu 404.407.6162 404.407.6887 Georgia Tech Research Institute Georgia Tech Research Institute