Taking Immersive Leap in Training of Landing Signal Officers Clay - PowerPoint PPT Presentation

Taking Immersive Leap in Training of Landing Signal Officers Clay ¡(Larry) ¡“Sea ¡Fog” ¡Greunke ¡ Amela ¡Sadagic ¡ US ¡Navy ¡ Naval ¡Postgraduate ¡School ¡

Overview • Problem Space and Motivation • Background • Task Analysis • Survey of User Domain • Prototype System and Development • Feasibility Study • Discussion and Future Work 2 ¡

Who are the LSOs? • Task : Facilitate the "safe and expeditious recovery" of Naval aircraft aboard aircraft carriers . • Time on task : 20 seconds between the pilot's start position until landing, to guide pilot toward an acceptable position on the carrier. • Originally just a single LSO - evolved into a team • Main Roles: – Deck Calling LSO – Controlling LSO – Backup LSO – CAG LSO 3 ¡

LSOs through the History 4 ¡

Problem Space & Motivation • Issue : 108 landing-related mishaps between 2005 – Jul 2015. 99 involved LSOs performance. • Training on-the-job : Majority of training for junior LSOs occurs while a carrier is deployed out to sea. • Other training : 2 weeks of Initial Formal Ground Training (IFGT) – an LSO spends 6 hours working with the large LSO Trainer (2H111). Agreement in domain : Additional training opportunities are very much needed. 5 ¡

Current training Solution: 2H111 Simulator • LSO School in Oceana, VA • Two stories tall room • Several large screens + physical interfaces / actual instruments 6 ¡

Our Objective • Fill the training gap - Create conditions for unlimited number of training opportunities unrestricted by location and time. • Use Commercial off-the-shelf (COTS) components to create a light-weight , portable, fully virtual system that replicates the aircraft carrier environment and allows LSOs to practice as individual or as a team . 7 ¡

Background • Objective of any training system : Positive transfer of training (skills & knowledge) [1][2][3][4] • Issues that can get in the way in VR : – Technical characteristics of VR system – Judgment of distance in VR [5][6][7][8] – Cybersickness [9][10][11][12] – Appropriateness of training approaches – Richness of scenarios – Length of exposure • All elements were reviewed before designing new LSO trainer. 8 ¡

Design Requirements 1. Support all sensory cues (system feedback) and user input essential for training of LSOs à Perform task analysis vetted by Subject Matter Experts (SMEs). 2. Support all major capabilities of 2H111. 3. Acquire a thorough understanding of users' training needs , domain view of benefits and shortcomings of training with 2H111. à Address shortcomings and bring additional capabilities non-existent in 2H111. 9 ¡

Task Analysis • Analyzed the three most important positions on the LSO team: 1. Controlling LSO, 2. Backup LSO, and 3. Deck Caller LSO • Results vetted by SMEs from the LSO School • Major difference between 2H111 and our system: Absence of haptic feedback i.e. lack of physical instrumentation that 2H111 has (LSO can press real buttons and switches) 10 ¡

Survey of User Domain • Our objective: Acquire an accurate understanding about: 1. Current state of training practices in the LSO community, 2. LSOs’ perception of elements of training with the 2H111 simulator – (a) benefits and good characteristics, and (b) negative issues and obstacles in training practice. • Built full IRB documentation. • Executed online survey using LimeSurvey. • 35 LSOs responded. 11 ¡

Survey of User Domain: Illustration of Results Skills, knowledge and concepts qualified as most difficult to acquire by an LSO (more results in [13]) à This plays a role when deciding what to emphasize in new training system. 1. Eye calibration: Determining the aircraft position as it relates to the ideal glideslope angle 2. Judgment: LSO's situational awareness 12 ¡

Design Goals for Prototype System 1. Support all major capabilities and training objectives currently supported by the 2H111. 2. Use only COTS components - remain a lightweight training system. 3. Invest considerable efforts to minimize potential for cybersickness (extended exposure to training solution needed to acquire LSOs skills). 4. Integrate a variety of typical COTS input options for trainee and instructor. 13 ¡

Prototype System & Development: System Architecture Head tracking Leap Motion Oculus Controller DK2 Display Voice Unity Application Recognition App. Virtual Environment Xbox Controller Immersed LSO (a trainee) Headphones Xbox Controller Keyboard Microphone Instructor LSO 14 ¡

Prototype System & Development: System Architecture & Tools • Laptop platform : Alienware 17 R2 with Intel Core i7-4980HQ CPU @ 2.80 GHz, 16 GB RAM and GeForce GTX 980M GPU • Input devices : Two Xbox Controllers (one for a trainee LSO + one for instructor), Leap Motion Controller, microphone • Output devices : Oculus DK2, headphones • Oculus DK2: 960 x 1080 per eye, max refresh rate of 75 Hz, field of view (FOV) 100 degrees, and weight of device: .97 lbs. • Dev. Tools : Unity, Blender, Audacity 15 ¡

Prototype System & Development: 3D Assets 16 ¡

Feasibility Study (1 of 2) 1. Technical System Performance Measurement : Frame rate is 37 – 60 FPS 2. Cross Comparison with 2H111 : a. LSOs VR prototype portable, unlike 2H111 b. 2H111 has restricted FOV c. 2H111 supports team training (all positions need to be present), and LSOs VR prototype can support training of individual positions (other positions could be simulated - agents); d. 3 rd person view possible in LSO prototype - enable novel learning points 17 ¡

Feasibility Study (2 of 2) 3. Informal Demo Feedback (13 LSOs) 1. Speech Recognition System 2. Visual Cues: – Daytime and nighttime aircraft recoveries – Signals from Arresting Gear Officers (AGO) visible – Strobe light patterns to be fixed – Add “day ID light” on the Super Hornet variants 3. Landing Signal Officer Display System (LSODS) 4. Manually Operated Visual Landing Aid System (MOVLAS) 18 ¡

Feasibility Study Daytime & nighttime recovery MOVLAS & LSODS in 2H111 and VR 19 ¡

Feasibility Study Portion of Deck Caller LSO’s perspective with Arresting Gear Officer being visible on the flight deck 20 ¡

Going Beyond 2H111 Capabilities ‘Magic book’ AR using a smartphone à added capability for top-down view of landing operation 21 ¡

Going Beyond 2H111 Capabilities • Federation of LSO trainers for team training • Utilizing VR to prototype and test new capabilities (e.g. AR concepts) - https://youtu.be/d79oT3PxT2U 22 ¡

Future work 1. Development of additional capabilities 2. System validation (models and data) 3. Development of federation of simulations/ trainers 4. Formal usability study (GUI) 5. Training effectiveness study 6. System re-design 7. Transfer of training study 23 ¡

Conclusion 1. Major 2H111 training objectives supported 2. Prototype succeeded to remain lightweight – all components COTS solutions 3. Achieved high frame rate – all LSOs operations were possible (note: additional efforts could still be invested to improve frame rate in complex scenes) 4. Supported a range of COTS input devices and interactive modalities. Time make immersive leap has come! 24 ¡

References [1] R.P. Darken, and W.P. Banker. Navigating in Natural Environments: A Virtual Environment Training Transfer Study. Virtual Reality Annual International Symposium , 1998. Proceedings., IEEE 1998. [2] N.E. Seymour, A. G. Gallagher, S.A. Roman, M.K. O’Brien, V.K. Bansal, D.K. Andersen, and R.M. Satava. Virtual Reality Training Improves Operating Room Performance. Annals of Surgery , Vol. 236, No. 4, 2002 Oct, pages 458–464. [3] J.M. Nolan, and J.M. Jones. Games For Training: Leveraging Commercial Off The Shelf Multiplayer Gaming Software For Infantry Squad Collective Training, Master Thesis, Naval Postgraduate School, 2005. [4] B. Brown. A Training Transfer Study of Simulation Games. Master thesis, Naval Postgraduate School, 2010. [5] P. Willemsen, M.B. Colton, S.H. Creem-Regehr, and W.B. Thompson. The effects of head-mounted display mechanics on distance judgments in virtual environments. APGV '04 Proceedings of the 1st Symposium on Applied perception in graphics and visualization , 2004. [6] W.B. Thompson, P. Willemsen, A.A. Gooch, S.H. Creem-Regehr, J.M. Loomis, and A.C. Beall. Journal Presence: Teleoperators and Virtual Environments , Vol. 13, No. 5, Oct. 2004, pages 560 – 571. 25 ¡