27 March 2018 Mikael Arguedas and Morgan Quigley - PowerPoint PPT Presentation

27 March 2018 Mikael Arguedas and Morgan Quigley

● ● ● ● ●

● ○ ● ○ ○ ● ○ ○

● ○ ○ ○ ● ● ○ ○

USB3 Camera Separate devices: USB3 USB Host Camera (prototypes 0-3) USB2 IMU Unified camera: USB3 FPGA Imager USB Host (prototypes 4-5) IMU Imager Unified system: Imager PCIe PCIe root FPGA (prototypes 6+) Imager IMU

● Global-shutter 1.3 MPix imagers, 20cm baseline ● FPGA+DRAM+USB3 on daughterboard ● InertialSense µ IMU-2

Imagers capable of ~2 Gbit pixel rate (each) DRAM DRAM buffer required since USB3 = 4Gbit - scheduling Artix-7: 16 bit @ 400 MHz DDR = ~13 Gbit - overhead 40 30 40 8 USB3 Imagers / / / Imagers FPGA Any PC PHY USB3-based design PCIe-based design PCIe x2 bandwidth is similar to (low-end) FPGA DRAM bus ! 30 Imagers PCIe root on SBC Imagers / FPGA 12 (NVIDIA TX2) PCIe PHY / PCIe Gen 2 x2 = 8 Gbit full-duplex

● Designed around TX2 ● FPGA is PCIe endpoint ● Self-contained computer vision: "just supply power"

https://cad.onshape.com/documents/12b7e4d13bded8c95b2b0603/w/4fe61ad3cda4bc70ee895fc7/e/e46ce3bf3f9ed56eb291d5e2 ● Imager active area is not centered ● Use 3d-printed lens holders ● PLA is OK. Carbon-fiber is better ● Heat-set inserts for mounting + lens lock

● Stereo systems need to be very stiff ● PCB is clamped to carbon-fiber tube

● always-on MCU waits for TX2 boot ● During TX2 boot: MCU loads stage-1 FPGA image ● TX2-FPGA PCIe link established ● TX2 loads stage-2 FPGA image over PCIe ● sensors initialized over PCIe MMIO UART MCU IMU TX2 SPI PCIe FPGA Imager Imager

FPGA TX2 PCIe PCIe sync IMU decimate DRAM Image timing via IMU sync DMA write arbiter Image sensors trigger

FPGA TX2 PCIe PCIe DMA arbiter DRAM FIFO image decode fix column deserialize sensors framing ordering

Extreme close-up of typical indoor navigation feature (sprinkler pipe joint)

7x7 circle corner

7x7 circle (discretized) corner

7x7 circle (discretized) corner "unrolled" discretized circle

"unrolled" circle subtract center threshold 1) Find (in parallel) if there is a contiguous sequence of >= 9 pixels above threshold value. 2) For non-max suppression, find (in parallel) "how far" the sequence is above the threshold

Imagers produce 4 pixels per clock. Solution: search in parallel.

● An example of FPGA reducing latency in (simple) pixel-wise operations ● 100's of operations per clock: 8-bit subtractions, comparisons, etc ● Deterministic timing, keeps up with pixel rate ● Many other algorithms are FPGA-friendly: pyramids, gradients, ...

FPGA TX2 PCIe SPI IMU BRAM SPI sequencer PCIe IMU sync decimate control DRAM register BRAM CPU GPU Image sensors DMA write arbiter trigger register file SPI pixel pixel image pixel corner FIFOs FIFOs stats FIFOs detectors link clock deserializer serializers train sync pixel ADC array data image image / and c 4 decoders

● Initialization ○ allocate PCIe-visible RAM block ○ configure FPGA core, imager SPI registers, IMU registers ● Every frame ○ FPGA writes pixels via DMA to TX2 RAM, sends interrupt ○ kernel re-syncs CPU caches ○ kernel unblocks userland thread in ROS node ○ ROS node copies image into ROS message, sends it downstream TX2 DMA RAM image ROS image Imager PCIe image Imagers FPGA consumer consumer driver consumer nodes kernel nodes nodes module MSI

● Dynamic distributed message-passing framework ● Huge collection of open-source nodes ● Tools to parameterize, configure, and debug nodes sensor actuator hardware hardware

camera vision camera downstream driver node nodes evil data publish evil sniffing image data evil prevented by prevented by encryption authorization node

https://github.com/osrf/tensorflow_object_detector

https://github.com/osrf/tensorflow_object_detector

PCIe root TX2 (Tegra) SoC System Fabric SSD GbE GPU Flash PCIe root GbE GPU USB USB (etc) USB USB FPGA "Traditional" system TX2-based system ● all peripherals on PCIe ● only the FPGA hangs off PCIe ● PCIe cannot reset / re-enumerate ● PCIe kernel driver can be reloaded ● PCIe devices ready within 100ms ● FPGA configure/reconfigure at any time ● elaborate "fast" configuration not needed

● all connectors on same side ● no configuration MCU ● FPGA upgrade (?) ● stack boards to reduce footprint

For more information: http://open.vision.computer Morgan Quigley morgan@openrobotics.org