Efficient Deep Vision for Aerial Visual Understanding Dr Christos Kyrkou KIOS Research and Innovation Center of Excellence, University of Cyprus KIOS Seminar Series, 01/06/2020 kyrkou.christos@ucy.ac.cy funded by: @ChristosKyrkou christoskyrkou.com
Computer Vision (CV) finally works. Now What? Similarly large accuracy improvements on tasks such as Semantic Segmentation Object Detection 3D reconstruction …and so on Mostly Deeper Networks Intricate Structures Millions of training images Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 2
CV/DL Deployment accelerating Rapidly Cloud PC/Workstation Mobile Image Sensor Benefits ! Requirements: Less Power Consumption Less Memory Usage Fast Response Cost Saving Security/Privacy Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 3
Markets Demands Scalability for Machine Learning Cloud Edge Analytics Intelligence 1000s of classes <10 classes Large Workloads Frame Rate: 15-30 fps Highly Efficient Power 1W-5W (Performance/W) Cost: Low Varying Accuracy Varying Accuracy Server Form Factor Custom Form Factor Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 4
Small Models have Big Advantages #1 Fewer parameter weights means bigger opportunities for scaling training Bigger networks increase the cost of communication between machines for distributed training Credit: Forrest Iandola “ Small Deep Neural Networks - Their Advantages, and Their Design” Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 5
Small Models have Big Advantages #2 Smaller number of weights enables complete on-chip integration of CNN model with weights – no need for off-chip memory Dramatically reduces the energy for computing inference Gives the potential for pushing the data-processing close to the data gathering (e.g., onboard cameras and other sensors) Limited memory of embedded devices makes small models absolutely essential for many applications. Credit: Song Han “Bandwidth -Efficient Deep Learning ——from Compression to Acceleration” Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 6
Small Models Have Big Advantages #3 Small models enable continuous wireless updates of models Each time any sensor discovers a new image/situation that requires retraining, all models should be updated. Data is uploaded to cloud and used for training But… how to update all the vehicles that are running all the model? At <500KB downloading new model parameters is easy. Continuous Updating of CNN Models Credit: Forrest Iandola “ Small Deep Neural Networks - Their Advantages, and Their Design” Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 7
Model + Hardware Specialization - Convolution, ReLU and Pooling operations are inherently highly parallel in nature -They are best accelerated by dedicated hardware in the FPGA But how much Convolution, ReLU and Pooling operations is needed? Credit: Song Han, Hardware Design Automation for Efficient Deep Learning, Samsung Forum Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 8
Application of small DNNs to UAVs Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 9
Challenges State-of-the-art CV algorithms often require extensive hardware: limited payload! Contradiction! Remote processing of images: solution? Use of ground station High bandwidth, minimal latency, ultra reliable connection Severe limitations! (especially when targeting autonomous UAVs!) On-board processing: specific inherent challenges Limited computational power Limited weight, power consumption Extreme optimization of HW and SW is the solution for on-board processing! Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 10
Autonomous patrolling and recognition … Image Image Acquisition Sensor Automated Embedded UAV System Embedded Platform Path Planning Software Fire Flood & Collapsed Collapsed Building Buildings Fire Fire Flood Flood Flood Collapsed Collapsed Car Crash Car Crash Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 11
Vision System for disasters and incidents Aerial Image Dataset for Emergency Response (AIDER) Order of magnitude more images than previous works C. Kyrkou and T. Theocharides, "EmergencyNet: Efficient Aerial Image Classification for Drone-Based Emergency Monitoring Using Atrous Convolutional Feature Fusion," in IEEE JSTARS, vol. 13, pp. 1687-1699, 2020 C. Kyrkou , T. Theocharides "Deep-Learning-Based Aerial Image Classification for Emergency Response Applications using Unmanned Aerial Vehicles", CVPR 3d International Workshop in Computer Vision for UAVs, Long Beach, CA, 16-20 June, 2019, pp. 517-525. Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 12
Pretrained Networks For transfer learning established networks are used which have also been used in prior works for disaster monitoring [1,2] . [3] [4] [5] [3] K. Simonyan and A. Zisserman, “Very deep convolutional networksfor large- scale image recognition,” CoRR, vol. abs/1409.1556, 2014.[Online]. Available: http://arxiv.org/abs/1409.1556 [4] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for imagerecognition ,” CoRR, vol. abs/1512.03385, 2015. [Online]. Available:http://arxiv.org/abs/1512.03385 [5] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand,M. Andreetto , and H. Adam, “ Mobilenets: Efficient convolutional neuralnetworks for mobile vision applications,” CoRR, vol. abs/1704.04861,2017. [Online]. Available: http://arxiv.org/abs/1704.04861 Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 13
How do you create a small DNN? Credit: Forrest Iandola “ Small Deep Neural Networks - Their Advantages, and Their Design” Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 14
Atrous Convolutional Feature Fusion C. Kyrkou and T. Theocharides, "EmergencyNet: Efficient Aerial Image Classification for Drone-Based Emergency Monitoring Using Atrous Convolutional Feature Fusion," in IEEE JSTARS, vol. 13, pp. 1687-1699, 2020 C. Kyrkou , T. Theocharides "Deep-Learning-Based Aerial Image Classification for Emergency Response Applications using Unmanned Aerial Vehicles", CVPR 3d International Workshop in Computer Vision for UAVs, Long Beach, CA, 16-20 June, 2019, pp. 517-525. Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 15
Macro-Architecture Design Choices Reduced Cost of First Layer and Early downsampling 16 channels with strided convolution Canonical Architecture A progressive reduction of spatial resolution with an increase in depth of up to 256 channels. Fully Convolutional Architecture Inference No dense layers 255 Network Depth 7 main blocks Capped leaky ReLU Capped from [0,…255] with different modes during training and inference Training Capped leaky ReLU C. Kyrkou and T. Theocharides, "EmergencyNet: Efficient Aerial Image Classification for Drone-Based Emergency Monitoring Using Atrous Convolutional Feature Fusion," in IEEE JSTARS, vol. 13, pp. 1687-1699, 2020 C. Kyrkou , T. Theocharides "Deep-Learning-Based Aerial Image Classification for Emergency Response Applications using Unmanned Aerial Vehicles", CVPR 3d International Workshop in Computer Vision for UAVs, Long Beach, CA, 16-20 June, 2019, pp. 517-525. Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 16
Performance Evaluation C. Kyrkou and T. Theocharides, "EmergencyNet: Efficient Aerial Image Classification for Drone-Based Emergency Monitoring Using Atrous Convolutional Feature Fusion," in IEEE JSTARS, vol. 13, pp. 1687-1699, 2020 C. Kyrkou , T. Theocharides "Deep-Learning-Based Aerial Image Classification for Emergency Response Applications using Unmanned Aerial Vehicles", CVPR 3d International Workshop in Computer Vision for UAVs, Long Beach, CA, 16-20 June, 2019, pp. 517-525. Dr Christos Kyrkou , “Efficient Deep Vision for Aerial Visual Understanding”, RCML2020, 4 September 2020 17
Recommend
More recommend
