Interest point detection Nicolas ROUGON ARTEMIS Department - PowerPoint PPT Presentation

IMA4103 Extraction d’Information Multimédia Interest point detection Nicolas ROUGON ARTEMIS Department Nicolas.Rougon@telecom-sudparis.eu Institut Mines-Télécom

Problem statement ■ We hereafter review methods for extracting local geometric features of interest in gray level images, useable in a variety of image matching problems ● Image registration ► image stitching | augmented reality ● Image retrieval & object recognition/categorization ► image & video indexing ● 3D scene/object reconstruction ► vision-based 3D photogrammetry ● Tracking & navigation ► simultaneous localization and mapping (SLAM) Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Problem statement ■ Motivation Matching techniques using local features of interest are significantly more robust to large variations of scene geometry, including ● strong viewpoint ● partial occlusion ● object deformation change than approaches assessing similarity between image (sub)domains Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Problem statement ■ Requirements Relevant features of interest should be distinctive, and satisfy properties ensuring stable and efficient detection / matching ● Structural properties  generic ► compactness  sparse ► computational efficiency ► robustness  numerous  occlusions | clutter | cropping  uniformly distributed Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Problem statement ■ Requirements Relevant features of interest should be distinctive, and satisfy properties ensuring stable and efficient detection / matching ● Invariance properties ► repeatability  sensor photometric calibration  contrast transforms  scene lighting monotonic luminance transforms  spatial transforms  sensor geometric calibration  viewpoint isometries | scalings | affine transforms Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Problem statement ■ Requirements Relevant features of interest should be distinctive, and satisfy properties ensuring stable and efficient detection / matching ● Robustness properties ► repeatability ► accuracy  digital image acquisition  sampling & quantization  coding scheme  sensor model  noise Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Problem statement ■ Candidate features Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Problem statement ■ Candidate features Edges are not eligible as features of interest ● Generic, sparse, uniformly distributed ● Reasonably invariant to contrast changes ● Not distinctive ► matching ambiguity occurs along edges ? ? ? L target = c n L source = c t ● Not invariant to spatial transforms Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Problem statement ■ Candidate features Corners provide relevant features of interest ● Distinctive ● Generic, sparse, numerous, uniformly distributed ● Invariant to contrast changes ● Invariant to spatial transforms except scalings Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Problem statement ■ Interest point matching ● Detection  Extract a set of distinctive & 1 x 1 repeatable interest points 2 x 3  Define an invariant interest patch 1 x 2 around each keypoint 1 x 3 ● Description 2  Normalize & transform patches x 2 2 x 1 into invariant local coordinates ▼ ▼  Compute a patch local descriptor 1 2 f 1 f 1 ● Matching ► ◄ ► ◄  Match local descriptors based 1 1 2 2 d( , ) f d f i f j f d on some similarity metrics 1 2 f i f j Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Example applications ■ Image stitching View #1 View #2 Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Example applications ■ Image stitching Corners #1 Corners #2 ► Corners capture the geometry of textured shapes Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Example applications ■ Image stitching Corner matching Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Example applications ■ Image stitching View stitching Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Example applications ■ Video Tracking Strong viewpoint changes | Partial occlusions Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Example applications ■ Video Tracking Strong viewpoint changes | Partial occlusions | Object deformations Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Example applications ■ Multi-view 3D scene reconstruction Feature extraction > corner points Feature matching > motion vectors Camera + sparse depth estimation > 3D point cloud Surface reconstruction + texturing > 3D textured mesh Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Problem statement ■ Requirements Expected performances of relevant interest point detectors ● Good detection ◄ generic framework for performance assessment Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Performance assessment ■ Confusion matrix ● True Positives (TP) T rue F alse Correct detections Success ● True Negatives (TN) Y = X T rue F alse P ositive Correct rejections P ositives P ositives Type I error Type I error ● False Positives (FP) Detection (Y) Type I error Wrong detections F alse T rue False alarm | Type I error Failure N egative N egatives N egatives Y ≠ X ● False Negatives (FN) Type II error Type II error Wrong rejections Type II error Miss | Type II error Ground Truth (X) Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Problem statement ■ Requirements Expected performances of relevant interest point detectors  few Failures ● Good detection  few false positives ◄ dedicated error metrics  few false negatives Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Performance metrics ■ Recall | Sensitivity | True Positive Rate ● Probability of relevant samples T rue F alse to be detected > P[Y=1|X=1] TP T rue F alse 𝑠𝑓𝑑𝑏𝑚𝑚 = P ositive TP + FN P ositives P ositives Type I error Type I error ● 𝑠𝑓𝑑𝑏𝑚𝑚 ↗ 1 when FN ↘ 0 Detection (Y) ► assessment of false negatives F alse T rue N egative ► false positives not addressed N egatives N egatives Type II error Type II error Ground Truth (X) Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Performance metrics ■ Precision | Positive Predicted Value ● Probability of detections T rue F alse to be relevant > P[X=1|Y=1] TP T rue F alse 𝑞𝑠𝑓𝑑𝑗𝑡𝑗𝑝𝑜 = P ositive TP + FP P ositives P ositives Type I error Type I error ● 𝑞𝑠𝑓𝑑𝑗𝑡𝑗𝑝𝑜 ↗ 1 when FP ↘ 0 Detection (Y) Type I error ► assessment of false positives F alse T rue N egative ► false negatives not addressed N egatives N egatives Type II error Type II error Type II error Ground Truth (X) Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Performance metrics ■ Specificity | True Negative Rate ● Probability of irrelevant samples T rue F alse to be rejected > P[Y=0|X=0] TN T rue F alse 𝑡𝑞𝑓𝑑𝑗𝑔𝑗𝑑𝑗𝑢𝑧 = P ositive TN + FP P ositives P ositives Type I error Type I error ● 𝑡𝑞𝑓𝑑𝑗𝑔𝑗𝑑𝑗𝑢𝑧 ↗ 1 when FP ↘ 0 Detection (Y) Type I error ► assessment of false positives F alse T rue N egative N egatives N egatives Type II error Type II error Type II error Ground Truth (X) Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Performance metrics ■ Negative Predicted Value ● Probability of rejections T rue F alse to be irrelevant > P[X=0|Y=0] TN T rue F alse 𝑂𝑄𝑊 = P ositive TN + FN P ositives P ositives Type I error Type I error ● 𝑂𝑄𝑊 ↗ 1 when FN ↘ 0 Detection (Y) Type I error ► assessment of false negatives F alse T rue N egative N egatives N egatives Type II error Type II error Type II error Ground Truth (X) Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Performance metrics ■ Accuracy ● Probability of correct decision T rue F alse (detection/rejection) > P[Y=X] TP + TN T rue F alse 𝐵𝐷𝐷 = P ositive TP + TN + FP + FN P ositives P ositives Type I error Type I error ● 𝐵𝐷𝐷 ↗ 1 when ( FP, FN) ↘ 0 Detection (Y) ► joint assessment of false F alse T rue N egative positives & false negatives, N egatives N egatives from detections & rejections Type II error Type II error Ground Truth (X) Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Performance metrics ■ F-score ● Harmonic mean of precision T rue F alse and recall 𝐺 = 2 𝑞𝑠𝑓𝑑𝑗𝑡𝑗𝑝𝑜 ∙ 𝑠𝑓𝑑𝑏𝑚𝑚 T rue F alse P ositive 𝑞𝑠𝑓𝑑𝑗𝑡𝑗𝑝𝑜 + 𝑠𝑓𝑑𝑏𝑚𝑚 P ositives P ositives 2 TP Type I error Type I error Detection (Y) = 2 TP + FP + FN F alse T rue N egative ● 𝐺 ↗ 1 when ( FP, FN) ↘ 0 N egatives N egatives ► joint assessment of false Type II error Type II error positives & false negatives, Ground Truth (X) focusing on detections Institut Mines-Télécom IMA 4103 - Nicolas ROUGON

Problem statement ■ Requirements Expected performances of relevant interest point detectors ● Good detection  high sensitivity / recall = all (most) true positives ► repeatability  high specificity / precision = few false positives ● Robustness against noise ● Good localization ► accuracy ● Computational efficiency Institut Mines-Télécom IMA 4103 - Nicolas ROUGON