Using deep learning and Using deep learning and Google Street View - PowerPoint PPT Presentation

Using deep learning and Using deep learning and Google Street View to estimate Google Street View to estimate the demographic makeup of the demographic makeup of neighbourhoods across the US neighbourhoods across the US Gebru , T. et al. (2017) DOI: 10.1073/pnas.1700035114 1

In a nutshell... In a nutshell... The United States spends more than $250 million each year on the American Community Survey (ACS), a labor-intensive door-to-doors tudy that measures statistics relating to race, gender, education, occupation, unemployment, and other demographic factors. Although a comprehensive source of data, the lag between demographic changes and their appearance in the ACS can exceed several years. As digital imagery becomes ubiquitous and machine vision techniques improve, automated data analysis may become an increasingly practical supplement to the ACS. Here, we present a method that estimates socioeconomic characteristics of regions spanning 200 US cities by using 50 million images of street scenes gathered with Google Street View cars . Using deep learning-based computer vision techniques, we determined the make, model, and year of all motor vehicles encountered in particular neighborhoods. Data from this census of motor vehicles, which enumerated 22 million automobiles in total (8% of all automobiles in the United States), were used to accurately estimate income, race, education, and voting patterns at the zip code and precinct level . (The average US precinct contains 1,000 people.) The resulting associations are surprisingly simple and powerful. For instance, if the number of sedans encountered during a drive through a city is higher than the number of pickup trucks, the city is likely to vote for a Democrat during the next presidential election (88% chance); otherwise, it is likely to vote Republican (82%). Our results suggest that automated systems for monitoring demographics may effectively complement labor-intensive approaches, with the potential to measure demographics with fine spatial resolution, in close to real time. 2

In a nutshell... In a nutshell... The United States spends more than $250 million each year on the American Community Survey (ACS), a labor-intensive door-to-doors tudy that measures statistics relating to race, gender, education, occupation, unemployment, and other demographic factors. Although a comprehensive source of data, the lag between demographic changes and their appearance in the ACS can exceed several years. As digital imagery becomes ubiquitous and machine vision techniques improve, automated data analysis may become an increasingly practical supplement to the ACS. Here, we present a method that estimates socioeconomic characteristics of regions spanning 200 US cities by using 50 million images of street scenes gathered with Google Street View cars . Using deep learning-based computer vision techniques, we determined the make, model, and year of all motor vehicles encountered in particular neighborhoods. Data from this census of motor vehicles, which enumerated 22 million automobiles in total (8% of all automobiles in the United States), were used to accurately estimate income, race, education, and voting patterns at the zip code and precinct level . (The average US precinct contains 1,000 people.) The resulting associations are surprisingly simple and powerful. For instance, if the number of sedans encountered during a drive through a city is higher than the number of pickup trucks, the city is likely to vote for a Democrat during the next presidential election (88% chance); otherwise, it is likely to vote Republican (82%). Our results suggest that automated systems for monitoring demographics may effectively complement labor-intensive approaches, with the potential to measure demographics with fine spatial resolution, in close to real time. 3

American Community Survey: American Community Survey: Demographic statistics for all US cities/counties with population ≥ 65,000 Comprehensive, but demographic changes are typically reported with a lag of several years Expensive and labour-intensive 4

American Community Survey: American Community Survey: Demographic statistics for all US cities/counties with population ≥ 65,000 Comprehensive, but demographic changes are typically reported with a lag of several years Expensive and labour-intensive Computational methods can be useful in tackling such challenges facing topics in social science 5

Research question Research question Can demographic statistics and voter preferences be inferred from objective characteristics of images from a neighbourhood? 6

Research question Research question Can demographic statistics and voter preferences be inferred from objective characteristics of images from a neighbourhood? Machine learning methods can be used with public data to determine socioeconomic statistics and political preferences in the US 7

Methodology: Methodology: overview overview 1. Establishment of dataset based on 50 million images taken from Google Street View Exterior of houses, landscaping, vehicles parked on the street etc. 2. Machine vision framework based on convolutional neural networks (CNN) to classify vehicles Recognise vehicles and determine their characteristics (e.g. make, model etc.) 3. Classified vehicles were use to infer a range of demographic statistics and socioeconomic attributes 8

1. 1. Dataset Dataset 50 million Google street view images from 3068 ZIP codes and 39286 voting precincts across 200 cities Product shot images: 2657 visually distinct car categories Amazon Mechanical Turk to gather labeled car images from specific websites (cars.com, craigslist.org etc.) Expert annotation of a subset of Google street view images 9

2. 2. Machine vision framework Machine vision framework Car detection Key challenge: balance accuracy and efficiency Deformable Part Model (DPM) as object recognition algorithm Detection of 22 million distinct vehicles Car classification CNN trained to distinguish different types of vehicle Classify each vehicle into one of 2657 categories 10

3. 3. Demographic estimation Demographic estimation Experimental procedure Dataset partitioned based on county into training (A-C; 35/200) and test Based on 88 car-related attributes: (D-Z) sets average #cars per image, average car Ridge regression model for income price, %hybrid cars, %foreign cars, %cars of a specific make, etc. and voter affiliation estimation Socioeconomic data obtained from Logistic regression for race and ACS between 2008-2012 education prediction 2008 election data from another study Five models trained in each case using fivefold cross-validation to select regularisation parameter (average) 11

Results Results (training set) (training set) Strong associations between vehicle distribution and different demographic factors 12

Results Results (training set) (training set) Strong associations between vehicle distribution and different demographic factors Toyotas/Hondas → Asian Sedans → Democrat Pickup trucks → Caucausian/Republican Chrysler/Buick → African American 13

Results Results (test set) (test set) Strong correlation between test results and ACS values for all demographic variables at city resolution; 0.54 ≤ r ≤ 0.87 Test results at ZIP code resolution also exhibited close correspondence with ACS values 14

Voter Voter preferences preferences Strong correlation between estimates and actual voting results of 2008 elections at city resolution; r =0.73 Precinct-level estimates also closely matched ground truth data; r =0.57 15

Conclusions Conclusions Neighbourhood images can be used to accurately estimate demographic statistics and voter preferences in the US via automated machine learning algorithms. Only data from a few cities are required to provide up-to-date statistics for many cities and ZIP codes. Model could be improved by expanding object recognition incorporate global image features integrating other data types. 16

Thank you. Thank you. 17