Faades of Interest in Street View Panoramic Sequences Andr A. - PowerPoint PPT Presentation

Accurate Location of Façades of Interest in Street View Panoramic Sequences André A. Araújo, Jonas C. Sampaio, Raphael S. Evangelista, Altobelli B. Mantuan, Leandro A. F. Fernandes {andrealvarado,revangelista}@id.uff.br, {jsampaio,amantuan,laffernandes}@ic.uff.br This work was sponsored by

What are those places? SIBGRAPI 2015 2

Overview Query image • A real example Actual user’s location Visited vertex Selected vertex 3 Assigned user’s location Estimated building’s location Candidate vertex

Contributions • An algorithm to calculate the location of buildings in a completely automatic way • A strategy for computing gnomonic projections restricted to the sidewalks • A heuristic to select which are the best Street View environments facing the building of interest • A derivative of how to estimate the uncertainty associated with the computed locations SIBGRAPI 2015 4

Outline • Proposed approach • Results • Concluding remarks SIBGRAPI 2015 5

Pipeline Query Image + Inaccurate GPS Coordinates Breadth-First Select Best Triangulate the Estimate the Search Panoramic Views Target Building Uncertainty Building Location + Uncertainty SIBGRAPI 2015 6

Breadth-First Search The First Stage Query image ≥ 𝑙 ; or Stopping criteria: 𝑒𝑗𝑡𝑢( , ) ≥ 𝑠 𝑛𝑏𝑦 Actual user’s location Visited vertex Selected vertex SIBGRAPI 2015 7 Assigned user’s location Estimated building’s location Candidate vertex

Breadth-First Search Image Rectification • Equirectangular projection (native format) Actual user’s location Visited vertex Selected vertex SIBGRAPI 2015 8 Assigned user’s location Estimated building’s location Candidate vertex

Breadth-First Search Image Rectification • Partial cubic projection (Sampaio et al., 2015) Sampaio et al. “Determining the location of buildings given a single picture, environment 9 maps and inaccurate GPS coordinates,” in Proc. of the ACM SAC, 2015.

Breadth-First Search Image Rectification • Two gnomonic projections (our approach) Actual user’s location Visited vertex Selected vertex SIBGRAPI 2015 10 Assigned user’s location Estimated building’s location Candidate vertex

Breadth-First Search Query image Image Comparison • Partial cubic X gnomonic projection Sampaio et al. (2015) 18 matching ASIFT features Our Approach 190 matching ASIFT features Sampaio et al. “Determining the location of buildings given a single picture, environment 11 maps and inaccurate GPS coordinates,” in Proc. of the ACM SAC, 2015.

Selecting Best Views Query The Second Stage image Left projections Right projections The endpoint of white segments indicate detected features. Actual user’s location Visited vertex Selected vertex SIBGRAPI 2015 12 Assigned user’s location Estimated building’s location Candidate vertex

Selecting Best Views Query The Proposed Algorithm image Left projections Right projections Actual user’s location Visited vertex Selected vertex SIBGRAPI 2015 13 Assigned user’s location Estimated building’s location Candidate vertex

Selecting Best Views Histogram of Detected Features Actual user’s location Visited vertex Selected vertex SIBGRAPI 2015 14 Assigned user’s location Estimated building’s location Candidate vertex

Triangulating the Target Building The Third Stage Actual user’s location Visited vertex Selected vertex SIBGRAPI 2015 15 Assigned user’s location Estimated building’s location Candidate vertex

Error Propagation The Last Stage • Errors in input variables propagate to the results Data + Transformations Errors Data Location Transformations + + Errors Errors • Uncertain input data ▪ Location of selected vertices, 𝑞 𝑗 = 𝑦 𝑗 , 𝑧 𝑗 𝑈 , with independent bivariate normal distributions ▪ Angle of preferred directions, 𝜄 𝑗 , with independent normal distributions • We use first order error propagation SIBGRAPI 2015 16

Error Propagation Resulting Uncertain Location Actual user’s location Visited vertex Selected vertex SIBGRAPI 2015 17 Assigned user’s location Estimated building’s location Candidate vertex

Results • Implementation ▪ Java language ▪ ImageJ, EJML, Google Maps API and Affine-SIFT libraries • Experiments ▪ 30 pictures ▪ 27 places SIBGRAPI 2015 18

Results Successfully Accomplished Detections Detection result for case 6 Uncertain ellipse includes the target building. Detection result for case 13 Street View provides only one panoramic view of the target, and it was found. SIBGRAPI 2015 19

Results Partially Successful Detections Detection result for case 18 𝑒𝑗𝑡𝑢 , exceeded the searching radius threshold 𝑠 𝑛𝑏𝑦 = 200 meters. By increasing 𝑠 𝑛𝑏𝑦 , the target would be detected. SIBGRAPI 2015 20

Results Failure due to Feature Matching Issues Detection result for case 27 Selected vertices pointed to different parts of the building, leading to divergent preferential directions. SIBGRAPI 2015 21

Results Cases that Cannot be Handled by Our System Case 28 The user took the picture of a tree. It is not a façade! Case 29 Street View The only panorama that records the façade is blocked. Query Image Case 30 The query image was taken at night. SIBGRAPI 2015 22

Concluding Remarks • An automatic image-based approach to estimate the location of target buildings ▪ Single query image and inaccurate GPS coordinates ▪ Breadth-first search in Google Street View ▪ Gnomonic projections of the sidewalks ▪ Algorithm for selecting the best views of the building ▪ Analytic derivation of uncertainty in computed locations • Future work ▪ Replace Affine-SIFT by a faster feature extractor ▪ Implement it as a mobile application SIBGRAPI 2015 23

Thank you! SIBGRAPI 2015 24