Shifting Media app development into high gear Using virtual drivers to speed up development Helen Koike helen.koike@collabora.com Open First Creative Commons Attribution-ShareAlike 4.0
Summary ● Classic V4L2 API → Vivid Driver ● Media API (extension) → Vimc Driver ● Vimc: ● Submodules ● Current state ● Confgfs API ● Future work ELC 2018 2
Summary ● Classic V4L2 API → Vivid Driver ● Media API (extension) → Vimc Driver ● Vimc: ● Submodules ● Current state ● Confgfs API ● Future work ELC 2018 3
ELC 2018 4
Sepia Sensor Composer Scaler Greyscale ELC 2018 5
APP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- Kernel Driver space ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 6
APP Read/Write/MMAP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- Driver Video stream Kernel space ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 7
APP Read/Write/MMAP User IOCTLs space -------------------------------------------------------- /dev/video* -------------------------------------------------------- Driver Standard Custom Video stream General confgs Controls Controls * Contrast * DRV DEF 1 * Img fmt Kernel * Brightness * DRV DEF 2 * Bufers space * Gamma * DRV DEF 3 * Video std . . * Frame rate . . . . . . ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 8
APP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- Kernel Driver space ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 9
APP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- Kernel Driver space ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 10
APP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- The Virtual Video T est Driver (VIVID) Kernel space ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 11
APP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- The Virtual Video T est Driver (VIVID) Kernel space ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 12
Vivid driver ELC 2018 13
Summary ● Classic V4L2 API → Vivid Driver ● Media API (extension) → Vimc Driver ● Vimc: ● Submodules ● Current state ● Confgfs API ● Future work ELC 2018 14
Media API Motivation ● Embedded systems: great variety of devices ● Hardware complexity ● Similar confguration for diferent entities ELC 2018 15
contrast Sepia Sensor Composer Scaler Greyscale contrast ELC 2018 16
APP Read/Write/MMAP User IOCTLs space -------------------------------------------------------- /dev/video* -------------------------------------------------------- Driver Standard Custom Video stream General confgs Controls Controls * Contrast * DRV DEF 1 * Img fmt Kernel * Brightness * DRV DEF 2 * Bufers space * Gamma * DRV DEF 3 * Video std . . * Frame rate . . . . . . ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 17
Sepia Sensor Composer Scaler Greyscale ELC 2018 18
Sepia Sensor Composer Scaler Greyscale ELC 2018 19
/dev/video0 Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2 /dev/v4l-subdev1 Greyscale ELC 2018 20
APP Subdev API contrast IOCTLs Sepia /dev/v4l- subdev0 APP APP R/W/MMAP IOCTLs Scaler IOCTLs /dev/v4l- /dev/video0 subdev2 APP contrast IOCTLs Greyscale /dev/v4l- subdev1 ELC 2018 21
What is part of the device? What is the topology? APP ? ? ? ? ? ? ? ? ? ? ? ? /dev/v4l-subdev0 /dev/v4l-subdev98 /dev/v4l-subdev11 /dev/v4l-subdev14 /dev/video2 /dev/video0 /dev/v4l-subdev6 /dev/video42 /dev/v4l-subdev21 /dev/v4l-subdev31 ELC 2018 22
/dev/video0 Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2 /dev/v4l-subdev1 Greyscale ELC 2018 23
/dev/video0 out Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2 /dev/v4l-subdev1 Greyscale ELC 2018 24
APP Media API contrast IOCTLs Sepia /dev/v4l- subdev0 APP APP APP R/W/MMAP IOCTLs IOCTLs Scaler IOCTLs /dev/media0 /dev/v4l- /dev/video0 subdev2 APP contrast IOCTLs Greyscale /dev/v4l- subdev1 ELC 2018 25
APP /dev/video0 IOCTLs out /dev/media0 Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2 /dev/v4l-subdev1 Greyscale ELC 2018 26
APP /dev/video0 IOCTLs out /dev/media0 Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2 /dev/v4l-subdev1 Greyscale ELC 2018 27
APP User space -------------------------------------------------------- /dev/media* -------------------------------------------------------- Kernel Driver space ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 28
APP User space -------------------------------------------------------- /dev/media* -------------------------------------------------------- Kernel Driver space ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 29
APP User space -------------------------------------------------------- /dev/media* -------------------------------------------------------- The Virtual Media Controller Driver (VIMC) Kernel space ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 30
APP User space -------------------------------------------------------- /dev/media* -------------------------------------------------------- The Virtual Media Controller Driver (VIMC) Kernel space ? ---------------------------------------------------------------------------------------------------------------------------------- Physical device ELC 2018 31
Summary ● Classic V4L2 API → Vivid Driver ● Media API (extension) → Vimc Driver ● Vimc: ● Submodules ● Current state ● Confgfs API ● Future work ELC 2018 32
APP Developer User space --------------- VIMC Confgfs API --------------------- /dev/media0 --------------------- /dev/media1 ----------- Kernel create space create ELC 2018 33
Vimc: submodules Subdev1.ko Implementation of subdevice Subdev2.ko drivers are modularized and doesn’t need to alter Vimc’s core code. SubdevN.ko ELC 2018 34
vimc-sensor.ko vimc-scaler.ko APP Developer Standard vimc-debayer.ko vimc-capture.ko ... Emulate diferent topologies to test the app in several scenarios custom-sd1.ko custom-sd2-ko Manufacturer custom-sd3.ko ... ELC 2018 35
Summary ● Classic V4L2 API → Vivid Driver ● Media API (extension) → Vimc Driver ● Vimc: ● Submodules ● Current state ● Confgfs API ● Future work ELC 2018 36
Vimc Driver ● Proposed by Laurent Pinchart for Outreachy in 2015 ● Merged in Kernel 4.12 ● Moving slowly ELC 2018 37
Vimc Driver: current state ● Basic standard submodules: – vimc-capture.ko – vimc-sensor.ko – vimc-debayer.ko – vimc-scaler.ko ● Hard-coded topology (re-compilation required) ELC 2018 38
ELC 2018 39
Vimc Driver ELC 2018 40
Userspace tools ● v4l-utils media-ctl -d /dev/media0 --print-dot > /tmp/out.dot && xdot /tmp/out.dot media-ctl -d /dev/media0 -V " 'Sensor A':0 [ fmt: RGB888_1X24/600x600]" media-ctl -v -d /dev/media0 --links "'Debayer A':1 -> 'Scaler':0 [0] " ● Yavta (Yet Another V4L2 T est Application) yavta --format RGB24 --size 600x600 /dev/video0 ELC 2018 41
Summary ● Classic V4L2 API → Vivid Driver ● Media API (extension) → Vimc Driver ● Vimc: ● Submodules ● Current state ● Confgfs API ● Future work ELC 2018 42
Confgfs API: Topology ● Entities ‘Sensor A’ vimc-sensor.ko – Name 0 – Submodule ● Pads 0 ‘Debayer A’ vimc-debayer.ko – Source 1 – Sink ● Links 0 ‘Raw Capture 0’ vimc-capture.ko ELC 2018 43
mkdir “MEDIA_NAME” ELC 2018 44
Recommend
More recommend