Video Manifolds Selen Atasoy MICCAI 2011 Tutorial Image Spaces - PowerPoint PPT Presentation

Image Similarities for Learning Video Manifolds Selen Atasoy MICCAI 2011 Tutorial

Image Spaces

Image Manifolds Tenenbaum2000 Roweis2000 Tenenbaum2000 [ Tenenbaum2000: J. B. Tenenbaum, V. Silva, J. C. Langford : A global geometric framework for nonlinear dimensionality reduction. Science , 290 (5500), 2000.] [ Roweis2000: S. T. Roweis, L. K. Saul: Nonlinear dimensionality reduction by locally linear embedding. Science , 290 (5500), 2000]

Video Manifolds Pless2003 Atasoy2010 [ Pless2003: R. Pless: Using Isomap to Explore Video Sequences: ICCV , 2003.] [ Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI , 2010.] [ Atasoy2011: S. Atasoy, D. Mateus, A. Meining, G.Z. Yang, N. Navab: Targeted Optical Biopsies for Surveillance Endoscopies, MICCAI , 2011.]

Theoretical Background

Manifold Learning • High dimensional data points lying on or near a manifold • Low dimensional representation • Find a mapping ??? that best preserves ...

Manifold Learning A General Recipe 1. Define a matrix based on the relations between data points 2. Compute the eigenvectors & eigenvalues 3. Embed each sample

Manifold Learning A General Recipe Method Operator/Matrix Preserved Objective Function Variance of the dataset / Euclidean PCA Covariance matrix distances between data points Distances within the Laplacian Eigenmaps Graph Laplacian local neighbourhood of each data point Geodesic distance Geodesic distances ISOMAP matrix between data points Reconstruction Reconstruction weights within the LLE weights local neighbourhood of each data point

Manifold Learning Why does it work? • Rayleigh-Ritz Theorem: eigenvalues eigenvectors • Recall: – Scalar product: – Scalar product in H : – Norm: – Norm in H :

Manifold Learning Why does it work? Discrete Domain Continuous Domain • vectors • functions • • • • Schwarz’s Kernel Theorem: Each linear operator is given as an integration against a unique kernel. That kernel is the impulse response of the linear system to an impulse (a delta function).

Manifold Learning Why does it work? Discrete Domain Continuous Domain • vectors • functions • • • •

Manifold Learning Why does it work? Discrete Domain Continuous Domain • vectors • functions • • • • The matrix H defines: • which operator is applied • which (Hilbert) space we are working in • which quantity will be conserved

Laplacian Eigenmaps

Manifold Learning Laplacian Eigenmaps • Solve • Find the eigenvectors of the graph Laplacian • Equivalent to solving the Helmholtz Equation [ Belkin2003: M. Belkin, P. Niyogi: Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation , 15(6), 1373-1396. MIT Press, 2003]

Manifold Learning Laplacian Eigenmaps - Interpretation [ Chladni1787 ] [ Levy2010 ] [ Levy2010 ] [ Chladni1787: E. Chladni: Discoveries in the Theory of Sound, 1787.] [ Levy2010: B. Levy: Spectral Geometry Processing: ACM SIGGRAPH Course Notes , 2010.]

Non-linear Manifold Learning Laplacian Eigenmaps - Interpretation • Manifold learning as bending, stretching without cutting or creating wholes • Vibrational modes are preserved while bending the manifold

Endoscopic Video Manifolds (EVMs)

Endoscopic Video Manifolds Challenges • Clustering Uninformative Frames

Endoscopic Video Manifolds Clustering Uninformative Frames

Endoscopic Video Manifolds Clustering Uninformative Frames Informative frame & power spectrum Uninformative frame & power spectrum Informative frame Uninformative frame [ Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI , 2010.]

Endoscopic Video Manifolds Clustering Uninformative Frames [ Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI , 2010.]

Endoscopic Video Manifolds Challenges • Significant change in endoscope viewpoint • Small overlap between frames showing the same scene • Scenes do not necessarily contain distinctive features

Endoscopic Video Manifolds Clustering Endoscopic Scenes – Euclidean Distance Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 Cluster 7 Cluster 8 Cluster 9 Cluster 10 0.04 Cluster 1 Cluster 2 0.02 Cluster 3 Cluster 4 0 Cluster 5 Cluster 6 -0.02 Cluster 7 Cluster 8 -0.04 Cluster 9 Cluster 10 -0.06 0.1 -0.08 0.05 -0.1 0 -0.05 -0.12 -0.06 -0.04 -0.02 0 0.02 -0.1 0.04 0.06 0.08 338 frames 150 frames 137 frames 102 frames 98 frames 78 frames 71 frames 71 frames 44 frames 38 frames [ Belkin2003: M. Belkin, P. Niyogi: Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation , 15(6), 1373-1396. MIT Press, 2003] [ Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI , 2010.]

Endoscopic Video Manifolds Clustering Endoscopic Scenes – Euclidean Distances Cluster 3 0.04 Cluster 1 Cluster 2 0.02 Cluster 3 Cluster 4 0 Cluster 5 Cluster 6 -0.02 Cluster 7 Cluster 8 -0.04 Cluster 9 Cluster 10 -0.06 0.1 -0.08 0.05 -0.1 0 -0.05 -0.12 -0.06 -0.04 -0.02 0 0.02 -0.1 0.04 0.06 0.08

Endoscopic Video Manifolds Clustering Endoscopic Scenes - NCC Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 Cluster 7 Cluster 8 Cluster 9 Cluster 10 0.04 0.02 0 -0.02 Cluster 1 -0.04 Cluster 2 Cluster 3 -0.06 Cluster 4 Cluster 5 -0.08 Cluster 6 Cluster 7 -0.1 Cluster 8 -0.12 Cluster 9 Cluster 10 0.1 0.05 0 -0.05 0.04 0.02 0 -0.02 -0.04 -0.1 -0.06 -0.08 389 frames 137 frames 103 frames 98 frames 85 frames 82 frames 81 frames 64 frames 44 frames 44 frames [ Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI , 2010.]

Endoscopic Video Manifolds Clustering Endoscopic Scenes - NCC Euclidean Distance Normalized Cross Correlation [ Atasoy2010: S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G.Z. Yang, N. Navab: Endoscopic Video Manifolds, MICCAI , 2010.]

Endoscopic Video Manifolds Clustering Endoscopic Scenes - NCC

Endoscopic Video Manifolds Clustering Endoscopic Scenes with Temporal Constraints • Change the adjacency matrix to include temporal constraints [ Atasoy2011: S. Atasoy, D. Mateus, A. Meining, G.Z. Yang, N. Navab: Targeted Optical Biopsies for Surveillance Endoscopies, MICCAI, 2011]

Endoscopic Video Manifolds Clustering Endoscopic Scenes with Temporal Constraints Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 Cluster 7 Cluster 8 Cluster 9 Cluster 10 0.04 Cluster 1 0.02 Cluster 2 Cluster 3 Cluster 4 0 Cluster 5 Cluster 6 -0.02 Cluster 7 Cluster 8 -0.04 Cluster 9 Cluster 10 -0.06 -0.05 -0.08 0 -0.1 0.05 0.1 0.08 0.06 0.1 0.04 0.02 0 -0.02 0.15 -0.04 344 frames 143 frames 126 frames 120 frames 112 frames 88 frames 55 frames 53 frames 43 frames 43 frames [ Atasoy2011: S. Atasoy, D. Mateus, A. Meining, G.Z. Yang, N. Navab: Targeted Optical Biopsies for Surveillance Endoscopies, MICCAI, 2011]

Acknowledgements Prof. Nassir Prof. Guang-Zhong Prof. Alexander Dr. Diana Navab Yang Meining Mateus Thank you for your attention!!!