Cyber Analyst Training Susan Adams, Elizabeth Fleming, Siobhan - PowerPoint PPT Presentation

Human Factors Approach to Cyber Analyst Training Susan Adams, Elizabeth Fleming, Siobhan Heiden and Liza Kittinger Presented by: Elaine Raybourn Sandia National Laboratories, United States of America

The Team 2 Susan Adams, PhD Scottie Fleming Siobhan Heiden, PhD Liza Kittinger, MA Principal technical staff Senior technical staff Technical staff member Lindsley, PhD member member Senior technical staff member Specializations : Specializations : Process Specializations : Training, Specializations : Engineering design and systems analysis and technology adoption, Experimental design, processes, collaborative design, knowledge organizational task analysis, decision decision making management development making

Sandia’s Cyber Mission Area 3 Sandia built a network intrusion detection Sandia’s research efforts in cybersecurity tool that helps cyber analysts detect: are focused in three broad areas: • Cyber attacks 1. Trusted hardware, software, and systems; • Data exfiltration 2. Networks and systems architectures and • System compromise analysis; and • Data manipulation 3. Effective cyber defense systems • Insider threat

Background 4 Motivation: Current training for tool to help cyber analysts’ identify pertinent risks did not sufficiently address their knowledge gaps Goal: Create evidence-based training materials to support novice cyber analysts’ needs at various stages of their learning Source: Sandia Labs (CC BY-NC-ND 2.0)

Challenges 5 • Limited access to end-users (i.e., cyber analysts) • End-users from a variety of organizations and cultural backgrounds • End-users separated by location and time from each other and the design team • Tool is constantly updated and modified • Need for both instructor-led training and post-training reference materials Source: Sandia Labs (CC BY-NC-ND 2.0)

Human Factor Approaches Used 6 Expert Heuristic Task Analysis Elicitations Evaluations Iterative Ethnography Design

Expert Elicitations 7 Expert Task Heuristic Elicitations Analysis Evaluations Definition: A process of obtaining judgements and knowledge from experts to a particular problem or scenario Iterative Ethnography Design Approach Explained the Elicitation Select Experts Refined Issues Context • Engineers and • Focused on issues of • Used task-oriented exercises with • Described the purpose designers of the tool learning and usability standardized question sets to elicit of the meeting and their conceptual understanding, expected outcomes • for the end user technical reasoning, and mental organization of the information most relevant for solving a particular issue. Findings ◦ Understand the decisions and reasons for solving particular cyber issues ◦ Identify commonalities and differences expert analysts might take ◦ Begin identifying locations where scaffolding would be appropriate

Task Analysis 8 Expert Task Heuristic Elicitations Analysis Evaluations Definition : A process of breaking a job task into smaller parts Iterative Ethnography Design Approach ◦ Used a general task analysis method where we focused on identifying the relationships one task had with another task in addition to terminology used ◦ Think-Aloud-Protocol: Experts were asked to talk while performing a given task Findings ◦ Allowed for the design team to observe aspects of the analyst’s behavior with various levels of detail and at various stages of the task ◦ Allowed the design team to understand sequential steps in completing tasks Attribution

Heuristic Evaluation 9 Expert Task Heuristic Elicitations Analysis Evaluations Definition : An analysis of the computer interface to ensure it is “user friendly” Iterative Ethnography Design Approach ◦ Used usability standards to evaluate how easy the interface was to use ◦ Considerations were given around: learnability, efficiency, memorability, errors and satisfaction Findings ◦ Results and recommendations for modern tool enhancements were given to developers (e.g. interface organization, features, search, etc.) ◦ Interface limitations influenced some aspects of how team designed training

Ethnography: Participant Observation 10 Expert Task Heuristic Elicitations Analysis Evaluations Definition : A strategy of observation and direct participation to understand the trainee’s perspective Iterative Ethnography Design Approach ◦ Participated in training sessions similar to an end user of the tool ◦ Completed readings and exercises a new analyst would experience ◦ Tried triaging cyber issues the way a new analyst is expected to do Findings ◦ Identified gaps in the learning process and where information became too advanced too quickly ◦ Identified assumptions instructors had of their students

Iterative Design 11 Expert Task Heuristic Elicitations Analysis Evaluations Definition : A method of prototyping, testing, analyzing and refining, then restarting the process. Iterative Ethnography Design • Non-linear process which involved continuous evaluation and feedback from users and designers to identify opportunities for improvement • Training was updated multiple times Design Evaluate Prototype

Lessons Learned 12 • Understanding the end user is key to any training design • Experts in the field are great resources, but effort is needed to scale down their level of knowledge to be appropriate for novice learners • Anticipate small and big changes to software to occur throughout the development of training

Conclusions 13 • Designing a learning program takes time • Some enhancements suggested by Human Factors may be beyond the scope of training (e.g., tool functionality) • The “ideal situation” is not always realistic – constraints, barriers and changes are inherent • Feedback and evaluation are key to a successful training program • Partnering with experts who were accessible was crucial to our success

Acknowledgements 14 Special thanks to the subject matter experts who participated in the design and development of this training. This presentation describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government. Sandia National Laboratories is a multi-mission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2019-5167 C.

Questions? 15 Susan Adams – smsteve@sandia.gov