Spotting Violence from Space The Detection of Housing Destruction in - PowerPoint PPT Presentation

Data Sources GDELT and ICEWS: scrape internet/news sources, give CAMEO scale events, give source information ACLED: coded violence events starting January 2017 Carter Centre: control data for thousands of locations from 2014-2017 Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 12 / 31

Data Sources GDELT and ICEWS: scrape internet/news sources, give CAMEO scale events, give source information ACLED: coded violence events starting January 2017 Carter Centre: control data for thousands of locations from 2014-2017 UNOSAT/UNITAR labels for six Syrian cities (up to 4 times) Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 12 / 31

Data Sources GDELT and ICEWS: scrape internet/news sources, give CAMEO scale events, give source information ACLED: coded violence events starting January 2017 Carter Centre: control data for thousands of locations from 2014-2017 UNOSAT/UNITAR labels for six Syrian cities (up to 4 times) Google Earth archive imagery Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 12 / 31

Data Sources GDELT and ICEWS: scrape internet/news sources, give CAMEO scale events, give source information ACLED: coded violence events starting January 2017 Carter Centre: control data for thousands of locations from 2014-2017 UNOSAT/UNITAR labels for six Syrian cities (up to 4 times) Google Earth archive imagery Unit of analysis is currently city but we are working on a match to control points. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 12 / 31

Media Reporting Dataset News reports from GDELT and ICEWS of fighting (and heavy weaponry) Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 13 / 31

Media Reporting Dataset News reports from GDELT and ICEWS of fighting (and heavy weaponry) Monthly panel of more than 1100 cities 2011-2018 Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 13 / 31

Media Reporting Dataset News reports from GDELT and ICEWS of fighting (and heavy weaponry) Monthly panel of more than 1100 cities 2011-2018 Many sources: we coded their country origin (e.g. UK for the Guardian) Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 13 / 31

Media Reporting Dataset News reports from GDELT and ICEWS of fighting (and heavy weaponry) Monthly panel of more than 1100 cities 2011-2018 Many sources: we coded their country origin (e.g. UK for the Guardian) We match this with: Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 13 / 31

Media Reporting Dataset News reports from GDELT and ICEWS of fighting (and heavy weaponry) Monthly panel of more than 1100 cities 2011-2018 Many sources: we coded their country origin (e.g. UK for the Guardian) We match this with: UNOSAT labels - destruction score Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 13 / 31

Media Reporting Dataset News reports from GDELT and ICEWS of fighting (and heavy weaponry) Monthly panel of more than 1100 cities 2011-2018 Many sources: we coded their country origin (e.g. UK for the Guardian) We match this with: UNOSAT labels - destruction score Carter centre control: government, isis, opposition, kurds Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 13 / 31

Media Reporting Dataset News reports from GDELT and ICEWS of fighting (and heavy weaponry) Monthly panel of more than 1100 cities 2011-2018 Many sources: we coded their country origin (e.g. UK for the Guardian) We match this with: UNOSAT labels - destruction score Carter centre control: government, isis, opposition, kurds ACLED fighting events, change of territory Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 13 / 31

Media Reporting Dataset News reports from GDELT and ICEWS of fighting (and heavy weaponry) Monthly panel of more than 1100 cities 2011-2018 Many sources: we coded their country origin (e.g. UK for the Guardian) We match this with: UNOSAT labels - destruction score Carter centre control: government, isis, opposition, kurds ACLED fighting events, change of territory Hypothesis: reporting is not consistent and function of control. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 13 / 31

Comparison of Fighting News Events and Housing Destruction (using UNOSAT labels) 5000 5000 hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra 4000 4000 UNOSAT destruction measure UNOSAT destruction measure 3000 3000 damascus damascus damascus damascus damascus damascus damascus damascus damascus damascus homs homs homs homs homs homs homs homs homs homs 2000 2000 aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo 1000 1000 aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo aleppo ar-raqqa ar-raqqa ar-raqqa ar-raqqa ar-raqqa ar-raqqa ar-raqqa ar-raqqa ar-raqqa ar-raqqa homs homs homs homs homs homs homs homs homs homs ar-raqqa ar-raqqa ar-raqqa ar-raqqa ar-raqqa aleppo aleppo aleppo aleppo aleppo ar-raqqa ar-raqqa ar-raqqa ar-raqqa ar-raqqa aleppo aleppo aleppo aleppo aleppo damascus damascus damascus damascus damascus damascus damascus damascus damascus damascus dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a hamra hamra hamra hamra hamra hamra hamra hamra hamra hamra ar-raqqa ar-raqqa ar-raqqa ar-raqqa ar-raqqa dar'a dar'a dar'a dar'a dar'a ar-raqqa dar'a dar'a ar-raqqa ar-raqqa dar'a dar'a ar-raqqa dar'a ar-raqqa dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a dar'a 0 0 0 2000 4000 6000 8000 10000 0 500000 1000000 1500000 2000000 ICEWS fighting measure GDELT fighting measure

Fighting News Events around Government Taking Control of City (at 0) .4 .2 0 -.2 -4 -3 -2 -1 0 1 2 3 4

Media Reporting Dataset Strong deviation between UNOSAT destruction and news reporting Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 15 / 31

Media Reporting Dataset Strong deviation between UNOSAT destruction and news reporting ACLED tries to explicitly tackle reporting bias by cross-verification and through additional sources. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 15 / 31

Media Reporting Dataset Strong deviation between UNOSAT destruction and news reporting ACLED tries to explicitly tackle reporting bias by cross-verification and through additional sources. We look at relationship between ACLED reports/UNOSAT and news reports on city i, in source j in month t through ln ( news ijt ) = α jt + θ ij + β 1 ∗ ln ( violence it ) + β 2 ∗ source j ∗ ln ( violence it ) + ǫ ijt Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 15 / 31

Media Reporting Dataset Strong deviation between UNOSAT destruction and news reporting ACLED tries to explicitly tackle reporting bias by cross-verification and through additional sources. We look at relationship between ACLED reports/UNOSAT and news reports on city i, in source j in month t through ln ( news ijt ) = α jt + θ ij + β 1 ∗ ln ( violence it ) + β 2 ∗ source j ∗ ln ( violence it ) + ǫ ijt The coefficient β 2 captures the fact that some sources report less. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 15 / 31

Media Reporting Dataset Strong deviation between UNOSAT destruction and news reporting ACLED tries to explicitly tackle reporting bias by cross-verification and through additional sources. We look at relationship between ACLED reports/UNOSAT and news reports on city i, in source j in month t through ln ( news ijt ) = α jt + θ ij + β 1 ∗ ln ( violence it ) + β 2 ∗ source j ∗ ln ( violence it ) + ǫ ijt The coefficient β 2 captures the fact that some sources report less. We look at sources from Syria, Russia, US and UK. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 15 / 31

News Reporting in Different Outlets: US and UK media vs. Russian and Syrian media (1) (2) (3) (4) fighting news heavy fighting fighting news fighting news VARIABLES events news events events events ACLED fighting events 0.178*** 0.0865*** (0.0230) (0.0156) ACLED fighting events * (Russian or Syrian news outlet) -0.0848*** -0.0527*** (0.0172) (0.0114) ACLED state gains territory 0.609*** (0.0989) ACLED state gains territory * (Russian or Syrian news outlet) -0.316*** (0.0797) ACLED opposition gains territory 0.419*** (0.0920) ACLED opposition gains territory * (Russian or Syrian news outlet) -0.0896* (0.0495) Source/city Fixed Effects YES YES YES YES Source/time Fixed Effects YES YES NO YES Observations 89,680 89,680 89,680 89,680 R-squared 0.617 0.564 0.617 0.613 Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. All variables x are in given in ln(x+1). Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 16 / 31

Our Method Supervised learning Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 17 / 31

Our Method Supervised learning Supervision - show the network 0s and 1s and it learns Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 17 / 31

Our Method Supervised learning Supervision - show the network 0s and 1s and it learns Need a set of 0s and 1s. Two ways we tried: Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 17 / 31

Our Method Supervised learning Supervision - show the network 0s and 1s and it learns Need a set of 0s and 1s. Two ways we tried: 1) mark destruction in images (first alley taken) Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 17 / 31

Our Method Supervised learning Supervision - show the network 0s and 1s and it learns Need a set of 0s and 1s. Two ways we tried: 1) mark destruction in images (first alley taken) 2) UNOSAT/UNITAR labels (currently best) Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 17 / 31

Our Method Supervised learning Supervision - show the network 0s and 1s and it learns Need a set of 0s and 1s. Two ways we tried: 1) mark destruction in images (first alley taken) 2) UNOSAT/UNITAR labels (currently best) 2) Offers no pixel-level labels but a LOT of labels (several thousand) Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 17 / 31

∙ destroyed, ∙ severe damage, ∙ moderate

Neural Network Architecture We use what is called a convolutional neural network (CNN) Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 19 / 31

Neural Network Architecture We use what is called a convolutional neural network (CNN) Tensorflow gives a lot of options for networks to use. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 19 / 31

Neural Network Architecture We use what is called a convolutional neural network (CNN) Tensorflow gives a lot of options for networks to use. We use a standard network called VGG16. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 19 / 31

Neural Network Architecture We use what is called a convolutional neural network (CNN) Tensorflow gives a lot of options for networks to use. We use a standard network called VGG16. 16 because it has 16 layers Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 19 / 31

Neural Network Architecture We use what is called a convolutional neural network (CNN) Tensorflow gives a lot of options for networks to use. We use a standard network called VGG16. 16 because it has 16 layers The first layers are based on many convolutional filters interrupted by max pooling layers. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 19 / 31

Idea of Convolutional Filter Use small filter (3X3), apply it to the different parts of the image. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 20 / 31

Idea of Convolutional Filter Use small filter (3X3), apply it to the different parts of the image. This leads to a scoring on the right. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 20 / 31

Idea of Max Pooling Make local summaries (example: 2X2, stride 2) Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 21 / 31

Idea of Max Pooling Make local summaries (example: 2X2, stride 2) Network ends with fully connected layers (voting on 0 or 1) Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 21 / 31

Idea of Max Pooling Make local summaries (example: 2X2, stride 2) Network ends with fully connected layers (voting on 0 or 1) For a fantastic explanation see Brandon Rohrer’s blog. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 21 / 31

(Modified) Very Deep Convolutional Networks for Large‐Scale Image Recognition, K. Simonyan, A. Zisserman

Method Based on UNOSAT/UNITAR Tags We do change detection: use before/after images. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 23 / 31

Method Based on UNOSAT/UNITAR Tags We do change detection: use before/after images. Make a patch around tag (64 X 64) pixels: "positives" Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 23 / 31

Method Based on UNOSAT/UNITAR Tags We do change detection: use before/after images. Make a patch around tag (64 X 64) pixels: "positives" Take a satellite image from the same place years before. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 23 / 31

Method Based on UNOSAT/UNITAR Tags We do change detection: use before/after images. Make a patch around tag (64 X 64) pixels: "positives" Take a satellite image from the same place years before. Sample "negatives" randomly, far away from positives. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 23 / 31

Method Based on UNOSAT/UNITAR Tags We do change detection: use before/after images. Make a patch around tag (64 X 64) pixels: "positives" Take a satellite image from the same place years before. Sample "negatives" randomly, far away from positives. Need to restrict sampling to urban area Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 23 / 31

Method Based on UNOSAT/UNITAR Tags We do change detection: use before/after images. Make a patch around tag (64 X 64) pixels: "positives" Take a satellite image from the same place years before. Sample "negatives" randomly, far away from positives. Need to restrict sampling to urban area This gives us 6 layers to feed into Neural Network. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 23 / 31

Method Based on UNOSAT/UNITAR Tags We do change detection: use before/after images. Make a patch around tag (64 X 64) pixels: "positives" Take a satellite image from the same place years before. Sample "negatives" randomly, far away from positives. Need to restrict sampling to urban area This gives us 6 layers to feed into Neural Network. We train and test with 5 folds and sample 20 negatives for one positive. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 23 / 31

All positives and negatives (20 negs/pos)

All positives and negatives (20 negs/pos)

Fold 1

Fold 2

Project is Now in Second Gear Instead of using it on pre-defined patches use it to "scan cities" Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 25 / 31

Project is Now in Second Gear Instead of using it on pre-defined patches use it to "scan cities" Goal: use trained classifier to scan unseen places or at least unseen times. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 25 / 31

Project is Now in Second Gear Instead of using it on pre-defined patches use it to "scan cities" Goal: use trained classifier to scan unseen places or at least unseen times. First problem: image quality, angle and lighting change. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 25 / 31

Project is Now in Second Gear Instead of using it on pre-defined patches use it to "scan cities" Goal: use trained classifier to scan unseen places or at least unseen times. First problem: image quality, angle and lighting change. → Domain transfer is very hard (our holy grail) Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 25 / 31

Project is Now in Second Gear Instead of using it on pre-defined patches use it to "scan cities" Goal: use trained classifier to scan unseen places or at least unseen times. First problem: image quality, angle and lighting change. → Domain transfer is very hard (our holy grail) Might be the reason why UNOSAT/UNITAR still use hand coding. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 25 / 31

Project is Now in Second Gear Instead of using it on pre-defined patches use it to "scan cities" Goal: use trained classifier to scan unseen places or at least unseen times. First problem: image quality, angle and lighting change. → Domain transfer is very hard (our holy grail) Might be the reason why UNOSAT/UNITAR still use hand coding. Second problem: imbalance explodes when scanning. Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 25 / 31

Problem in Applications Literature is focusing on 1:1 evaluation ( TPR = 0 . 8 , FPR = 0 . 12) We deviate from this on purpose. Reason: reality on the ground is far from 1:1 A LOT more patches contain no destruction. Statistics of 1:1 test are misleading Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 26 / 31

Why is this a problem? An example: True positive rate (share of 1’s predicted correctly - recall ) TP TPR = TP + FN = 80 % False positive rate (share of 0’s not predicted correctly) FP FPR = FP + TN = 12 % Imagine you have 1 million patches Imagine of these 100 are destroyed Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 27 / 31

Why is this a problem? 12% FPR means your model produces 1 Mio ∗ 0 . 12 FP = 120 , 000 FP 80% TPR means your model produces 100 ∗ 0 . 8 TP = 80 TP The probability that you are right if you find destruction is... 80 / 120 , 000 = 0 . 06 % !!! This means we need to get false positives ( FP ) down! Hannes Mueller (Trinity College) Spotting Violence from Space 22. February 2019 28 / 31