V-PANE Virtual Perspectives Augmenting Natural Experiences GTC 2017 - PowerPoint PPT Presentation

V-PANE Virtual Perspectives Augmenting Natural Experiences GTC 2017 The views, opinions and/or findings expressed are those of the author(s) and should not be interpreted as representing the official Kerry Moffitt views or policies of the Department of Defense or the U.S. Government. Scientist Distribution Statement “A” (Approved for Public Release, Kerry.Moffitt@Raytheon.com Distribution Unlimited). DARPA DISTAR #27938

Contents • Background: DARPA, GXV-T • V-PANE Overview and Architecture • Geometry • Video • Ongoing Work

DARPA Ground X-Vehicle Technologies (GXV-T) Goal: Improved vehicle survivability and mobility Development Areas: • Increased agility • Enhanced mobility • Crew augmentation • Signature management Source: http://www.darpa.mil/program/ground-x-vehicle-technologies

V-PANE Overview Objective: Develop the next generation of ground-vehicle human- machine interfaces that fuse real-time sensor feeds with video data projected onto a 3D geometric model of the environment. The program aims to develop a fully user-controlled, multiple- perspective, live virtual representation of the vehicle’s surroundings. Networked IED Report Boomerang Shot Detection IED Synthetic 1 st ATAK Route Person View Plan Popular Press: Secondary View & Touchscreen Wired Magazine National Defense American Security Today Product Design and Development:

High-Level V-PANE Vision Boomerang Shot Detection LWIR Lidar Ground X-Vehicle Color Additional Onboard LWIR Video Video Image Array Sensors Gunner View & Controls Point Cloud Future Driver/Commander View & Controls Imagery Projected onto 3D Geometry ‘Live’ 3D View Renderer 3D Model Path Static Imagery 1. The real-time fusion of: SA System (e.g., ATAK) Plan & Maps A. Lidar point clouds into a 3D model and B. Multiple 2D video streams onto that model and C. Other 2D or 3D threat, position or mapping data 2. The real-time rendering of that model with video into 2D displays from multiple perspectives 3. The real-time control of the multiple perspectives

V-PANE User Capabilities 360˚ Vehicles Visualization People Arbitrary Object Visualization Detection Perspectives Obstructions Obstacles Real-time Multi-Spectral Route Analysis Image Fusion Dynamic Slope Controls After- Include Offline Action Reports Pre-existing Viewing Data Reconnaissance Location Fused Semantic Position Probing Routing Information Cue, Slew, Range, Slope Blue Force Shot Bearing, (pitch/roll) Track a Locations Reports Elevation Location

Sensor Array

Real-Time Scanning, Modeling, Rendering • Lidar + Video + IMU = The World

Real-Time Scanning, Modeling, Rendering

V-PANE Workstation Architecture Lidar IMU/GPS Cameras 2 x Velodyne 1 NovAtel 1 IOI 4KSDI HDL-32E 1 VectorNav 5 BMD HD 2 LWIR NTSC 2 x BMD Q2 LAN USB Video Grabbers P100 K80 CPU1 CPU2 Fusion Compression P100 M6000 P100 Video Frusta Video Cache, Raycasting Projection SSD Rendering

V-PANE Data Pipeline Processing and Recording in a Single Server – Maximum-Load Analysis with PEX 8747 PCIe Switches Lidar IMU/GPS Cameras Inter-GPGPU (P2P) 2 x Velodyne 1 IOI 4KSDI 1 NovAtel HDL-32E 1 VectorNav Inter-GPGPU (non-P2P) 5 BMD HD CPU-Driven Position, 2 LWIR NTSC QPI Orientation 1400k pts/s 800 Hz 3 MB/s 2 x 40 KB/s DisplayPort 2 x BMD Q2 NB: LAN USB Video Grabbers 1 - P2P links require no transfer Depths, to or from CPU 2 - Non-P2P links count twice Indices Raw Lidar Raw IMU Live Video Compressed 3 - QPI link hits both CPUs 4 GB/s 3 MB/s 80 KB/s 2.9 GB/s Video Voxels 500 MB/s P100 K80 100 MB/s CPU1 CPU2 Fusion Compression Processed Voxels Live Video Voxels Lidar 5 GB/s 2.9 GB/s 1 GB/s 18 MB/s Video, Depths, Indices Voxels 4 GB/s 1.4 GB/s Video 3.3 GB/s Voxels Depths, Indices Voxels HD Video 5 GB/s Video, 4 GB/s 500 MB/s 5 GB/s Pixel Voxels Positions P100 M6000 600 MB/s P100 3 GB/s Video Frusta Video Cache, Raycasting HD Video Projection SSD Rendering 500 MB/s

Lidar Projection: Geometry Fusion • Lidar lasers (1) reflect back from surfaces in the world (2), sampling depth 2 • During each fusion update, every voxel reverse- projects (3) to lidar focal point to determine 3 distance from voxel to surface 4 • Depth samples stored as rectangular array with 1 fixed distance between samples (4) to optimize lookup (this requires resampling fixed-angle- delta lidar data)

Beam Modeling 100 m 2.32 m 1.33°

Wobbler • Fill in the vertical gaps between lasers

Ray Casting • Over pixels in current view – Project into voxel array – Find nearest zero-crossing

Raycast: Determine 3D Point per Pixel • For every pixel to be 3 rendered on screen (1), project from virtual camera through pixel (2) into voxel space, to find intersection point with nearest surface on that ray (3) • Output is a 3-space point 1 per pixel to be rendered 2

Video Projection: Color Index per Pixel • For every pixel to be rendered on screen (1), reverse-project from 3D point to real-world camera frame (2), to find 2 1 intersection point with 3 image captured in that frame (3) • Any given scene may involve 100s of frames

V-PANE – Data Processing Frequency and Latency Update Frequency Geometry Fusion Geometry Fusion Cop Geometry: 20 Hz . . . y Video Capture: 60 Hz < 50 ms Transfer Voxels Rendering: 60 Hz GPGPU 1 < 50 ms Latency Ray Cast Geometry: < 148 ms Video: < 132 ms GPGPU 2 < 16 ms Video Project GPGPU 3 Video Capture Video Capture Render Grabber GPU 1 < 16 ms < 100 ms < 16 ms < 16 ms Video Latency: < 132 ms Geometry Latency: < 148 ms

Profiling Geometry Fusion and Ray Casting Geometry Fusion Cycle Time: < 50 ms Ray Casting: < 16 ms

V-PANE Workstation Architecture Lidar IMU/GPS Cameras 2 x Velodyne 1 NovAtel 1 IOI 4KSDI HDL-32E 1 VectorNav 5 BMD HD 2 LWIR NTSC 2 x BMD Q2 LAN USB Video Grabbers P100 K80 CPU1 CPU2 Fusion Compression P100 M6000 P100 Video Frusta Video Cache, Raycasting Projection SSD Rendering

Image Processing in V-PANE From Live Cameras [Cache] YUV to Pinned Color Build Copy cudaMemcpy Undistort Host Memory Convert Frame Frame Fusion Projection Compression Copy Bits R Compress e Download n d From Disk e DXT5 to Pinned r Upload Host Memory Decompress Copy Bits [Cache] Build Copy Undistort Frame Frame Fusion Projection CPU GPU Host CUDA OpenGL

Thread Activity (Video I/O) 16.6 ms Main Thread (M6000) Decompress Process Upload Render VBLANK Decompress… Dequeue Dequeue Video Load/Receive Threads Enqueue … Enqueue Compression Thread (K80) Dequeue Process Upload Compress Store Process…

Video Latency: Best Observed Case Single 1080p60 Stream Event in Captured Output Visible on World Signal Signal Display Hardware * Camera Grabber Host Display Capture Capture Processing Hardware Software Cycle: 17 ms D Wait D Capture M for M Thread A VB A … After Capture, S U D S Render Render w p r w a l a a Can Thread p d w p … Upload Cycle: 17 ms * Total latency and host processing latency are observed 0 25 50 75 100 Time (milliseconds) Hardware latency is inferred

LWIR Integration

Ongoing Work: Object Detection • Image classification

Ongoing Work: Level of Detail • Add a second voxel array: • 10x range, to 1 km radius 1/10 th voxel resolution per dimension • • Ray caster uses per pixel when no hit in primary (hi-res) voxels • Requires only 0.1% compute to update given 100 m lidar range 2,000 m 200 m

Ongoing Work and Challenges • Voxel cache, Voxel LOD • GPS/altitude • Occlusion testing for video projection – Bandwidth vs. render quality • Timestamps – GPS from IMU and lidar, but not from cameras – Off by 100 ms = 2.5m • What if video but no geometry? (Skybox) • Voxel precision – Image quality vs. geometry update rate (20 cm voxels? 10? 5?)

The End Questions?