Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms - PowerPoint PPT Presentation

Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms Christiaan Gribble Applied Technology Operation SURVICE Engineering GTC 2017

Acknowledgments • University of Washington – Steve Brunton, PhD – Ben Erichson, PhD – Nathan Kutz, PhD • SURVICE Engineering – Rob Baltrusch – Mark Butkiewicz – Shawn Recker, PhD 2 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Overview Combines modern data reduction & analysis techniques with machine learning via DNNs and massively parallel computing architectures to enable next-gen ISR 3 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Key elements Compressed Machine NVIDIA Advanced DMD learning GPUs UIs 4 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Key elements Compressed Machine NVIDIA Advanced DMD learning GPUs UIs • Provides least-squares fitting of temporal data • Enables real-time performance with compression 5 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Key elements Compressed Machine NVIDIA Advanced DMD learning GPUs UIs • Employs deep neural networks • Supports advanced object detection tasks 6 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Key elements Compressed Machine NVIDIA Advanced DMD learning GPUs UIs • Permit fast training for design & optimization • Enable inference on mobile platforms 7 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Key elements Compressed Machine NVIDIA Advanced DMD learning GPUs UIs • Extend users’ natural abilities • Reduce cognitive burden 8 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Dynamic mode decomposition • Frame X t → snapshot of dynamics • X t+1 = AX t for A : R n → R n [Grosek & Kutz 2014] 9 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Dynamic mode decomposition • Frame X t → snapshot of dynamics • X t+1 = AX t for A : R n → R n DMD estimates A and its eigenvalues to characterize system dynamics [Grosek & Kutz 2014] 10 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Dynamic mode decomposition time modes amplitudes evolution space ≈ space … … … time video stream flattened frame reshaped video dynamic mode decomposition [Erichson et al. 2016] 11 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Background/foreground separation [Grosek & Kutz 2014] 12 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Compressed DMD data modes DMD 𝚾 , 𝚳 full X , X’ 𝚾 = X’V Y S Y -1 W Y C compressed cDMD 𝚾 Y , 𝚳 Y Y , Y’ Accelerates DMD by operating on compressed data [Erichson et al. 2016] 13 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Compressed DMD time × … → space compression matrix … … … video stream flattened reshaped frame video reshaped video compressed video [Erichson et al. 2016] 14 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Compressed DMD time modes amplitudes evolution space space … ≈ … time compressed video dynamic mode decomposition [Erichson et al. 2016] 15 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Background/foreground separation [Erichson et al. 2016] 16 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Implementation 17 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Intel Core i5 500 450 400 Runtime Performance 350 (frames per second) 300 Compression Level 1 250 0.5 0.1 200 0.01 150 0.001 100 50 0 3840x2160 1920x1080 1280x720 1024x768 640x480 Video Resolution 18 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

NVIDIA Tesla K40c 900 800 700 Runtime Performance (frames per second) 600 Compression Level 1 500 0.5 400 0.1 300 0.01 0.001 200 100 0 3840x2160 1920x1080 1280x720 1024x768 640x480 Video Resolution 19 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Jetson TX1 – CPU 50 45 40 Runtime Performance 35 (frames per second) 30 Compression Level 1 25 0.5 0.1 20 0.01 15 0.001 10 5 0 1920x1080 1280x720 1024x768 640x480 Video Resolution 20 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Jetson TX1 – GPU 100 90 80 Runtime Performance 70 (frames per second) 60 Compression Level 1 50 0.5 0.1 40 0.01 30 0.001 20 10 0 1920x1080 1280x720 1024x768 640x480 Video Resolution 21 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Object detection Traditional approach Haar DMD classifier DNN approach Mini-quad CNN DJI 22 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

DetectNet Output Error FCN Input [Image source: NVIDIA Parallel Forall] 23 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Initial results 24 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

In progress • Enhance DMD • Fabricate IVA modules • Integrate advanced UIs • Deploy on mobile platforms [Kutz et al. 2016] 25 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

In progress • Enhance DMD • Fabricate IVA modules • Integrate advanced UIs • Deploy on mobile platforms Sentinel-enabled IVA module 26 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

In progress • Enhance DMD • Fabricate IVA modules • Integrate advanced UIs • Deploy on mobile platforms 27 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

In progress • Enhance DMD • Fabricate IVA modules • Integrate advanced UIs • Deploy on mobile platforms JTARV – Hoverbike 28 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Key DMD references Brunton, B. W., Brunton, S. L., Proctor, J. L. & Kutz, J. N. (2015) Optimal sensor placement and enhanced sparsity for classification. SIAM Journal on Applied Mathematics , https://arxiv.org/abs/1310.4217. Erichson, N. B., Brunton, S. L., & Kutz, J. N. (2015) Compressed dynamic mode decomposition for background modeling. Journal of Real-Time Image Processing , https://arxiv.org/abs/1512.04205v2. Grosek, J. & Kutz, J. N. (2014) Dynamic mode decomposition for real-time background/foreground separation in video. IEEE Transactions on Pattern Analysis & Machine Learning , https://arxiv.org/abs/1404.7592. Kutz, J. N., Fu, X., & Brunton, S. L. (2016) Multi-resolution dynamic mode decomposition. SIAM Journal on Applied Dynamical Systems , https://arxiv.org/abs/1506.00564. 29 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Contact information Address Applied Technology Operation SURVICE Engineering 4695 Millennium Drive Belcamp, MD 21017 E-mail christiaan.gribble@survice.com Web http://www.survice.com/employees/~cgribble/ 30 C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms