Machine Learning-aided Disaster Response Supporting faster - PowerPoint PPT Presentation

Machine Learning-aided Disaster Response Supporting faster humanitarian relief efforts Ben Bischke Jakub Fil Ramona Pelich Tim G. J. Rudner Marc Rußwurm Dr Simon Jackman simon.jackman@mpls.ox.ac.uk

The Team Marc Rußwurm Ramona Pelich Ben Bischke Jakub Fil Tim G. J. Rudner

Our Mentors Veronika Kopaˇ ckov´ a Piotr Bili´ nski Artificial Intelligence Earth Observation

Case Studies Houston Haiti Floods Hurricane Aug. 2017 Oct. 2016 Ecuador Earthquake Apr. 2016

Our Approach Key idea Fast, high-accuracy building and damage detection by fusion of multi-resolution and multi-temporal satellite imagery. Input data sources: ◮ Radar: Sentinel-1 (public) ◮ Optical: Sentinel-2 (public) ◮ Very high resolution (commercial)

Multi-temporal Data disaster time radar radar radar radar radar optical optical optical optical VHR optical

Multi-temporal Data disaster time radar radar radar radar radar optical optical optical optical VHR optical pre post during

Multi-temporal Data disaster time radar radar radar radar radar optical optical optical optical VHR optical coherence pre coherence post high correlation values correlation decreases coherence RGB pre & post

Multi-temporal Data disaster time radar radar radar radar radar optical optical optical optical VHR optical VHR post

Multi-resolution Data 0.5m post-disaster 10m pre-disaster 10m pre-disaster very high resolution optical radar

Multi-resolution Data 0.5m post-disaster 10m post-disaster 10m post-disaster very high resolution optical radar

Ground truth, towards two objectives building footprints damaged sites Open Street Map UNOSAT

Semantic Segmentation with PSP-Net CNN (ResNet) classifier Input Hengshuang Zhao et al. “Pyramid scene parsing network”. In:IEEE Conf. on Computer Vision and PatternRecognition (CVPR). 2017

Semantic Segmentation with PSP-Net CNN PSP (ResNet) Decoder Feature Input Output Maps Hengshuang Zhao et al. “Pyramid scene parsing network”. In:IEEE Conf. on Computer Vision and PatternRecognition (CVPR). 2017

Semantic Segmentation with PSP-Net conv conv conv CNN pool (ResNet) conv Feature Feature Input Output Maps Maps Pyramid Spatial Pooling (PSP) Module Hengshuang Zhao et al. “Pyramid scene parsing network”. In:IEEE Conf. on Computer Vision and PatternRecognition (CVPR). 2017

Extending PSP-Net to Multi-resolution Input PSP ResNet 192px Module 96px PSP ResNet 96px Module 96px

Extending PSP-Net to Multi-resolution Input PSP ResNet 192px Module 96px PSP ResNet 96px Module 96px PSP ResNet 768px Module 96px

Houston—Building Prediction Results Individual data sources: Data mIoU Building IoU Accuracy S2 (10m & 20m bands) 73.1% 66.7% 85.4% S1 (Int, Int MT , Coh) 69.3% 63.7% 82.6% VHR 78.9% 74.3% 88.8% Fused data sources: Data mIoU Building IoU Accuracy S1 (Int MT , Coh) + S2 (10m & 20m bands) 76.1% 70.5% 87.3% S1 (Int MT , Coh) + S2 (10m & 20m bands) + VHR 79.9% 75.2% 89.5% Training epochs = 20, Batch size = 4, learning rate = 0.001

Extending PSP-Net to Multi-temporal Input pre PSP ResNet Module post pre PSP ResNet Module post PSP ResNet Module

Houston—Damage Prediction Results Data mIoU Building IoU Accuracy S1 (Int MT , Coh) + S2 (10m & 20m bands) 59.7% 34.1% 86.4% VHR 74.2% 56.0% 93.1% S1 (Int MT , Coh) + S2 (10m & 20m bands) + VHR 75.3% 57.5% 93.7% Training epochs = 20, Batch size = 4, learning rate = 0.001

Houston—Building Prediction (1/2) Sentinel based predictions: target (10m) prediction RGB input

Houston—Building Prediction (2/2) VHR based predictions: RGB input target (2m) prediction

Houston—Damage Prediction (1/4) RGB input target prediction

Houston—Damage Prediction (2/4) VHR only VHR + Sentinel difference overlap added by fusion removed by fusion

Houston—Damage Prediction (3/4) RGB input target prediction

Houston—Damage Prediction (4/4) VHR only VHR + Sentinel difference overlap added by fusion removed by fusion

Use Cases & Users Use cases ◮ Response prioritization: Reliable infrastructure mapping in developed and developing countries ◮ Improvement of response time: Integration of our model into existing disaster response pipelines to improve and speed up disaster response ◮ Better long-term planning: Rapid and accurate estimation of extent and total economic cost of damage Users ◮ Humanitarian aid organizations ◮ National and local authorities in affected areas

Future Directions Next steps ◮ Scale models to disaster-prone areas globally ◮ Create flexible model suite readily available for deployment ◮ Work with disaster relief organizations to optimize model usability ◮ Further extend models using multi-task and transfer learning Interested in collaborating? Come talk to us—we’d love to work with you!

Many Thanks to our Partners Dr Simon Jackman simon.jackman@mpls.ox.ac.uk