Adaptive Collaborative Jamison Heard Vanderbilt University PhD Candidate Robotics March 7 th 2019
2 Health Care Human-Robot Interaction Social Robotics Military Manufacturing Space
3 Develop new methodologies that facilitate in intelligent and effective robot collaborations wit ith a human.
Dynamic Environments 4 Supervisory Peer
Workload and Performance 5 Good Performance Acceptable Level Poor Underload Normal Load Overload Overall Workload
Workload Components 6 Overall Workload Cognitive Physical Auditory Visual Speech
Envisioned System 7 Adaptive Human-Robot Teaming System Architecture Future Tasks Activity Recognition Task Current Tasks (Re-)Allocations i-CiFHaR Workload Models Workload Component Workload Interaction Estimates Metrics Decision Workload Assessment Framework Algorithm Predicted Interaction Performance Changes Communication Performance Modality Prediction Assess and Predict Determine Adaptations Apply Adaptations
Envisioned System 8 Adaptive Human-Robot Teaming System Architecture Future Tasks Activity Recognition Task Current Tasks (Re-)Allocations i-CiFHaR Workload Models Workload Component Workload Interaction Estimates Metrics Decision Workload Assessment Framework Algorithm Predicted Interaction Performance Changes Communication Performance Modality Prediction Assess and Predict Determine Adaptations Apply Adaptations
Outline 9 • Prior Work • Workload Assessment • Activity Recognition • Future Work • Adaptive Interactions • Research at RIT • Potential Funding Sources • Teaching Experience
Envisioned System 10 Adaptive Human-Robot Teaming System Architecture Future Tasks Activity Recognition Task Current Tasks (Re-)Allocations i-CiFHaR Workload Models Workload Component Workload Interaction Estimates Metrics Decision Workload Assessment Framework Algorithm Predicted Interaction Performance Changes Communication Performance Modality Prediction Assess and Predict Determine Adaptations Apply Adaptations
Objective Workload Metrics 11 Cognitive Physical Postural Measures Brain-Activity Measures Heart-Rate Cardiovascular Measures Skin Temperature Visual Auditory Speech Noise Level Eye-Tracking Measures Response Time Speech-Based Measures Respiration-Rate Noise Level
Challenges with Physiological Signals 12 Age Individual Differences Physical Fitness Training Circadian Rhythms Day-to-Day Variability Stressful Events
Related Work 13 Workload Assessment Algorithm Structure Physiological Machine Workload Signals Learning Classification J. Heard, C. E. Harriott, and J. A. Adams. A Survey of Workload Assessment Algorithms. IEEE Transactions on Human-Machine Systems. 2018.
Algorithmic Limitations 14 • Typically only assess Cognitive Workload • Do not detect the Underload State • Do not account for Individual Differences • Only analyzed in a Single Task Domain J. Heard, C. E. Harriott, and J. A. Adams. A Survey of Workload Assessment Algorithms. IEEE Transactions on Human-Machine Systems. 2018.
Objective Workload Metrics 15 Cognitive Physical Postural Measures Brain-Activity Measures Heart-Rate Cardiovascular Measures Skin Temperature Visual Auditory Speech Noise Level Eye-Tracking Measures Response Time Speech-Based Measures Respiration-Rate Noise Level
Workload Assessment Algorithm 16 Objective Workload Metrics Heart-Rate Heart-Rate Variability Respiration Rate Posture Magnitude Signal Processing and Neural Networks Feature Extraction Noise Level Speech-Rate Pitch Speech Intensity Workload Estimates J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.
Signal Processing 17 30 Second Adaptive Window Exponential Filtering J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.
Feature Extraction 19 Features Mean StDev. 30 Second Adaptive Window Exponential Filtering Slope Gradient J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.
Neural Network Architecture 20 Input Layer Hidden Layers Output Layer 4 x N + 2 128 128 128 128 1 Neurons Neurons Neurons Neurons Neurons Neuron N = number of objective Fully connected layers Workload Component workload metrics TANH activation functions Estimate J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.
Workload Estimates 21 Cognitive Physical Neural Networks Overall ∑ Auditory Workload Speech Workload Models Visual J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.
Workload State Classification 22 Underload Normal Load Overall Workload Threshold Overload J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.
Experimental Design 23 Supervisory-Based Evaluation Peer-Based Evaluation Search and Rescue Remotely Piloted Aircraft J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.
Algorithmic Analysis 24 • Three Trained Algorithms: PEER, SUP, and BOTH • Four Analysis: • Population Generalizability • Train on 70% of the participants, test on the other 30% • Cross-Teaming Generalizability • Train on peer evaluation Data, Test on supervisory • Vice-Versa • Emulated Real-World Conditions • Train on supervisory data, Test on second day supervisory data • Real-Time • Train on second day supervisory data, test in real-time J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.
Peer-Based Evaluation 25 96 96 95 95 100 90 90 88 Classification Accuracy 81 80 63 60 40 20 0 Cognitive Workload Physical Workload Overall Workload PEER SUP BOTH J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.
Supervisory-Based Evaluation 26 100 100 99 98 98 98 100 Classification Accuracy 80 67 66 63 60 40 20 0 Cognitive Workload Physical Workload Overall Workload SUP PEER BOTH J. Heard, R. Heald, C. E. Harriott, and J. A. Adams. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human-Robot Teams. ACM Transactions on Human-Robotic Interaction. 2019.
Workload Transitions 27 Underload Normal Load Overload Underload Overload Normal Load Underload J. Heard and J. A. Adams. Multi-Dimensional Human Workload Assessment for Supervisory Human-Machine Teams. Journal of Cognitive Engineering and Decision Making. 2019.
Real-Time Evaluation 28 Communications Tracking Resource System Management Monitoring
Real-Time Results 29
Real-Time Results 30
Summary ry 31 • Assess Overall Workload and Contributing Components • Assess workload in Peer and Supervisory Human Robot Teams • Assess Workload Transitions • Real-Time Workload Assessments
Outline 32 • Prior Work • Workload Assessment • Activity Recognition • Future Work • Teaching Experience
Envisioned System 33 Adaptive Human-Robot Teaming System Architecture Future Tasks Activity Recognition Task Current Tasks (Re-)Allocations i-CiFHaR Workload Models Workload Component Workload Interaction Estimates Metrics Decision Workload Assessment Framework Algorithm Predicted Interaction Performance Changes Communication Performance Modality Prediction Assess and Predict Determine Adaptations Apply Adaptations
EMS Procedures 34 Intravenous Airway Therapy Management • Nasal Airway • Placing an IV • Oral Airway • Administer IV Medication • Intubation • IO Line Wound and Vital Fractures Checking High Trauma • Tourniquet • Stethoscope • Combat Gauze • Placing Monitoring Equipment • Chest Decompression • Splinting • Cardiopulmonary Resuscitation (CPR)
Hierarchical Task Analysis 35 Check for Breathing Preparation Lift Patient’s Chin Use Bag-Valve Mask CPR Give Breaths Use Mouth Compressions 30 Chest Compressions
Wearable Sensors 36 Apple Watch MYO • Accelerometer • Accelerometer • Gyroscope • Gyroscope • EMG
Accelerometer Data Example 37 CPR Compressions Breath
Video: OpenPose 38
OpenPose Challenges 39
Recommend
More recommend
