FOSS Virtual & Augmented Reality Monado & OpenXR Jakob - PowerPoint PPT Presentation

FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020 1

Jakob Bornecrantz FDO GH @Wallbraker jakob@collabora.com Open First 2

● What is AR & VR ● OpenXR ● Monado ● Status ● Demos ● Wrapping up 3

● What is AR & VR OpenXR Monado Status Demos Wrapping up 4

What is Augmented Reality? ● Augment ━ Verb ━ / ɔːɡˈ m ɛ nt/ ━ make greater by adding to it; increase 5

Open First 6

Open First 7

What is Virtual Reality? ● I reject your reality and substitute my own ━ Dungeonmaster, 1984 ■ (popularized by Adam Savage) 8

Open First 9

Open First 10

AR + VR = ? ● Also known as XR ● X = A | V ● Not eXtended Reality 11

What is AR & VR ● OpenXR Monado Status Demos Wrapping up 12

How do we XR? Program Platform Hardware 13

How do we XR? Program API Platform Hardware 14

15

What’s cool about OpenXR? ● It’s just like OpenGL & Vulkan ━ Made by Khronos ● Out now ● Action Based ● FOSS Implementation! :D 16

Action Based ● Type of controller not in focus ━ Only after the fact ● Context ━ From program ━ To runtime ━ Runtime decides what button does what 17

What is AR & VR OpenXR ● Monado Status Demos Wrapping up 18

How do we XR? Program Platform Hardware 19

Where does Monado fit in? Program Monado Hardware 20

Monado Program OpenXR OpenXR State Tracker xrt_prober xrt_device xrt_compositor Platform Compositor Drivers Hardware 21

Complex Truth Program OpenXR OpenXR Loader OpenXR State Tracker xrt_prober xrt_device xrt_gl_compositor GL Wrapper Platform xrt_fd_compositor Drivers Compositor Auxiliary Hardware 22

Desktop Stack Program OpenXR OpenXR State Tracker xrt_prober xrt_device xrt_compositor Platform Drivers Compositor Misc APIs Linux APIs Vulkan, X11, Wayland and KMS Linux Hardware Hardware 23

With IPC - Anticipated OpenXR State Tracker xrt_fd_compositor xrt_prober xrt_device IPC Layer Process boundary xrt_prober xrt_device xrt_fd_compositor Platform Drivers Compositor Misc APIs Linux APIs Vulkan and KMS Linux Hardware Hardware 24

XRT Interfaces ● Between different components ━ Auxiliary code has their own interfaces ● Purely C ● Not stable! [1] 25

xrt_device ● HMDs ● Functions ● Controllers Input Sync ━ Output Set ━ ● Aspect pattern View/Position Retrieval ━ Optional structs ━ Info (immutable) ● HMD info ━ Input/Outputs ━ Tracking Origin ━ Screen position (HMD) ━ Distortion (HMD) ━ 26

xrt_compositor ● Base class ● Functions xrt_[gl|vk|fd]_compositor More explicit then OpenXR ━ ━ GL & VK are wrappers Create Swapchain ━ ━ ■ Client side only Prepare, Begin & End ━ FD is “ real ” compositor ━ Wait, Begin, Discard & End ━ ■ Written in Vulkan Poll Events (Refactor?) ━ ■ Only “ talks ” in FDs ● Info (immutable) ● xrt_swapchain Supported Formats ━ xrt_[gl|vk|fd]_swapchain ━ acquire, wait & release ━ 27

xrt_prober ● Policy holder ● Functions Reads configuration Look for devices ━ ━ Decides what devices open Select (opens devices) ━ ━ List & Open Video devices ● Device prober ━ Misc USB ━ Used by OpenXR ST ━ ● Info (immutable) Platform code ● None ━ Abstract native interfaces ━ Used by device drivers ━ ● Should be split in two xrt_prober ━ xrt_platform ━ 28

What is AR & VR OpenXR Monado ● Status Demos Wrapping up 29

Status ● Mostly complete OpenXR support ● In-process Compositor ━ Mesh shader ● Video processing framework ● Tracking PS Move ● Nifty debug UI 30

Hardware support ● North Star* ● PlayStation VR ● Daydream Ctrl* ● PlayStation Move ● OSVR HDK ● Razer Hydra ● Vive (Pro) & Index ● OpenHMD ━ Positional tracking ━ Port to native drivers through libsurvive where it makes sense. 31

Code ● 35.5K lines of codes ━ Drivers: 6.9K 9 drivers ■ Vive: 1.5K ■ HDK: 0.5K ■ ━ Compositor: 6.6K ━ Auxiliary: 7.6K ━ State Trackers: 10.1K ━ Targets: 1.3K 32

Next Up ● Complete OpenXR support ● Improve PS Move Tracking ● PSVR Tracking ● Out of process compositor 33

Even further out ● Safe space ● System UI ● Setup UI ● Lighthouse Tracking ● AR and SLAM 34

What is AR & VR OpenXR Monado Status ● Demos Wrapping up 35

Tracking 36

Debug UI 37

Tracking (back-up) 38

What is AR & VR OpenXR Monado Status Demos ● Wrapping up 39

Rant ● Gnome-Shell ━ Output hot-plug ━ non-desktop ● Intel Missing Extension ━ GL_EXT_memory_object_fd 40

Some thoughts ● Lots of work needed ● GNOME & KDE UIs? ━ Monado interface library? ● Switching between apps? 41

FOSS XR Community @FossXR 42

Links ● Monado ━ https://monado.dev/ ━ https://monado.pages.freedesktop.org/monado ━ https://gitlab.freedesktop.org/monado/monado ● OpenXR ━ https://www.khronos.org/openxr ● Work on GFX & XR ━ Internships @ Collabora ━ Work @ Collabora 43

Talk to me about ● XR, VR & AR FOSS XR ● ● Volt Programming Language ● FPGAs & FOSS ● Amiga (FPGA), mc68k (LLVM) Voxel/SVO rendering ● ● FOSS & Society ● Joining Collabora! Open First 44

FOSS Virtual & Augmented Reality Questions? 45