Global Robot Ego-Localization C Combining Image Retrieval and HMM- - PowerPoint PPT Presentation

Global Robot Ego-Localization C Combining Image Retrieval and HMM- bi i I R i l d HMM based filtering Cédric LE BARZ ONERA, The French Aerospace Laboratory PhD PhD advisors: d i - M. CORD (Pierre&Marie Curie University - Paris) - S. HERBIN & M. SANFOURCHE (Onera - Palaiseau) ( ) - J-Y. DUFOUR (Thales company - Palaiseau) 1

Context illustration A Amazon Prime Air P i Ai 2

Autonomous navigation  Needs accurate & absolute localization   Accuracy required to follow trajectory and compute Accuracy required to follow trajectory and compute appropriate commands  Absolute localization required because mission is specified q p with absolute coordinates  GPS ?  May be absent (shadowing effect)  May be jammed (military missions)  Precision about 10m  An other information source is needed to deal with GPS drawbacks. Our proposal: use visual information…like humans Our proposal: use visual information…like humans 3

Our objective: visual absolute localization  How to find, in a geo-referenced image database, the image depicting the same scene as the last th i d i ti th th l t image acquired by the robot (request image) ?  Image retrieval problematic  Nice feature would be: Use a freely available and  wide coverage geo-referenced image database  … like Google Streetview 4

Image retrieval: State of the arts  Image Retrieval (IR) algorithms   Vote based methods (kNN) Vote based methods (kNN) Dictionary based methods (BOVW)    Descriptor modelization based methods (VLAD) Descriptor modelization based methods (VLAD)  Kernel based methods Based on local descriptors. Construction of a  compact signature, which is then used for IR task. p g , 5

Image retrieval Sometimes it works well ….. 6

Image retrieval 7

Image retrieval 8

Image retrieval Sometimes it does not … 9

Image retrieval 10

Image retrieval  Many disparities may occur: points of view, illumination conditions, focal length, scene objects (car, people, vegetation…)  Visual matching methods are sensitive to these disparities  How can we make visual localization more robust ? 11

Our approach (1/2)  Combine visual information with odometric measurements (hybrid system) measurements (hybrid system) 12

Our approach (2/2)  Goal: find the trajectory that best explain observations  T  Trajectory can be seen as a j t b sequence of states  Where each state is associated with a geo-referenced image ith f d i  Means: HMM  Hidden states are places p 13

HMM: ‘ Π ’ vector  Π is the initial state distribution vector: the probability that the first state/position is a particular state/positio  Example: If we are in state S3 with the uniform uncertainty hypothesis, then Π =[0;1/3;1/3;1/3;0;0]   ‘ Π ’ enables to take into account initial estimated position and initial Π enables to take into account initial estimated position and initial localization uncertainty 14

HMM: ‘A’ matrix  A={a ij } is the transition probability matrix between each state a ij =P(q t+1 =S j | q t =S i )  Example: From state S2, odometric measurements enables to estimate state S4. If we consider uniform uncertainty about odometric measurements, then a 2,j = [0;0;1/3;1/3;1/3;0] [ ] 2,j  ‘A’ enables to take into account odometric measurements and odometric uncertainty ( a priori information) 15

HMM: ‘B’ matrix  B is the likelihood to get a specific observation given the state P(O|S) the state P(O|S)  Enable to take into account visual similarity between observations and geo-referenced database geo referenced database images  Visual similarity is computed from the number of matching descriptors of matching descriptors between each request image and database images images 16

Qualitative results: Example 17

Qualitative results : Example 18

Experiments / Evaluation Image Image database number Request Google street view 1105 1 image images every ~10m Geo-referenced ‘Pittsburgh’ 2215 1 image database database (provided (p every ~5m y Images by Google company)  11 km trajectory  640x480 images 19

Results: Mean error localization vs. initial localization uncertainty   Odometric uncertainty 10m Odometric uncertainty 10m  5 observations (*) A Zamir and M Shah “Accurate image localization based on google maps street view ” in ( ) A. Zamir and M. Shah, Accurate image localization based on google maps street view, in Proceedings of the European Conference on Computer Vision. IEEE, 2010, pp. 255–268. 20

Result : Mean error localization vs. number of observations  Sensitivity of our solution to the number of past observations used, according the initial localization uncertainty U • Odometric uncertainty: 10m = > The higher U, the more observations number have to be considered to keep the mean error localization under a threshold. 21

Conclusions & Perspectives Conclusion We proposed a hybrid method that combines odometric  measurements with visual similarity measurements.  Advantages  Improve image retrieval / Reduce error localization Complete re-estimation of the last part of the trajectory when  required (long-term navigation) i d (l t i ti )  Developed framework can easily integrate new IR solutions   Can be used for the kidnapped robot problem Can be used for the kidnapped robot problem Improvements   Learn HMM parameters (‘A’ matrix) Learn HMM parameters ( A matrix)  Improve visual similarity measurements (‘B’ matrix)   Find discriminative and representative elements thanks to learning Find discriminative and representative elements thanks to learning  Improve localization accuracy thanks to pose estimation ( 22

Thank you for your attention Questions ? 23