Estimating metrics suitable to an empirical manifold of shapes, - PowerPoint PPT Presentation

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Motivation Introduction Motivation ◮ Shape spaces : which metric ? (to define similarity/distance between shapes) ◮ Hausdorff distance ◮ Symmetric difference area ◮ Quotients by transformation groups (rotation, translation, scaling, affine...) ◮ Shape evolution, morphing : priors on probable deformations ? = ⇒ Which local metric on deformations ? (metric on the manifold of shapes) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Motivation Introduction Motivation ◮ Shape spaces : which metric ? (to define similarity/distance between shapes) ◮ Hausdorff distance ◮ Symmetric difference area ◮ Quotients by transformation groups (rotation, translation, scaling, affine...) ◮ Shape evolution, morphing : priors on probable deformations ? = ⇒ Which local metric on deformations ? (metric on the manifold of shapes) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Motivation Introduction Motivation ◮ Shape spaces : which metric ? (to define similarity/distance between shapes) ◮ Hausdorff distance ◮ Symmetric difference area ◮ Quotients by transformation groups (rotation, translation, scaling, affine...) ◮ Shape evolution, morphing : priors on probable deformations ? = ⇒ Which local metric on deformations ? (metric on the manifold of shapes) ◮ L 2 norm of instantaneous deformations ◮ L 2 + curvature, H 1 ◮ rigid motion more probable = ⇒ associated metric Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Motivation Introduction Motivation ◮ Shape spaces : which metric ? (to define similarity/distance between shapes) ◮ Hausdorff distance ◮ Symmetric difference area ◮ Quotients by transformation groups (rotation, translation, scaling, affine...) ◮ Shape evolution, morphing : priors on probable deformations ? = ⇒ Which local metric on deformations ? (metric on the manifold of shapes) ◮ L 2 norm of instantaneous deformations ◮ L 2 + curvature, H 1 ◮ rigid motion more probable = ⇒ associated metric L 2 inner product vs. rigidifying inner product Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Motivation Introduction Motivation ◮ Shape spaces : which metric ? (to define similarity/distance between shapes) ◮ Hausdorff distance ◮ Symmetric difference area ◮ Quotients by transformation groups (rotation, translation, scaling, affine...) ◮ Shape evolution, morphing : priors on probable deformations ? = ⇒ Which local metric on deformations ? (metric on the manifold of shapes) ◮ L 2 norm of instantaneous deformations ◮ L 2 + curvature, H 1 ◮ rigid motion more probable = ⇒ associated metric ◮ = ⇒ learn the suitable metric from examples (datasets of shapes) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Issues Issues ◮ Sparse sets of highly varying shapes ◮ e.g. human silhouettes ◮ high intrinsic dimension ( � 30) ◮ = ⇒ no dense training set Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Issues Issues ◮ Sparse sets of highly varying shapes ◮ e.g. human silhouettes ◮ high intrinsic dimension ( � 30) ◮ = ⇒ no dense training set Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Issues Issues ◮ Sparse sets of highly varying shapes ◮ e.g. human silhouettes ◮ high intrinsic dimension ( � 30) ◮ = ⇒ no dense training set Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Issues Issues ◮ Sparse sets of highly varying shapes ◮ e.g. human silhouettes ◮ high intrinsic dimension ( � 30) ◮ = ⇒ no dense training set Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Issues Issues ◮ Sparse sets of highly varying shapes ◮ e.g. human silhouettes ◮ high intrinsic dimension ( � 30) ◮ = ⇒ no dense training set ◮ to compare quantities defined on different shapes : need for correspondences ◮ match shape with different topologies ? ◮ very frequent topological changes Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Searching for solutions Main existing approaches and their limitations Approach 1 : mean + modes model Approach 2 : distance-based approaches, such as kernel methods Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) ◮ Mean M , shapes S i , warpings W M → S i Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) ◮ Mean M , shapes S i , warpings W M → S i Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) ◮ Mean M , shapes S i , warpings W M → S i Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) ◮ Mean M , shapes S i , warpings W M → S i ◮ PCA : diagonalize correlation matrix C : C ij = ˙ ˛ ¸ W M → S i ˛ W M → S j = ⇒ eigenmodes e k with eigenvalues λ k : best coordinate system Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) ◮ Mean M , shapes S i , warpings W M → S i ◮ PCA : diagonalize correlation matrix C : C ij = ˙ ˛ ¸ W M → S i ˛ W M → S j = ⇒ eigenmodes e k with eigenvalues λ k : best coordinate system ◮ any new deformation W of M : X W = α k e k + noise k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) α 2 X ◮ Mahalanobis distance : d ( M + W , ( S )) = k λ 2 k k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) α 2 X ◮ Mahalanobis distance : d ( M + W , ( S )) = k λ 2 k k ◮ associated inner product on deformations, in the tangent space of M : 1 X � W 1 | W 2 � = α 1 , k α 2 , k λ 2 k k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) α 2 X ◮ Mahalanobis distance : d ( M + W , ( S )) = k λ 2 k k ◮ associated inner product on deformations, in the tangent space of M : 1 X � W 1 | W 2 � = α 1 , k α 2 , k λ 2 k k ◮ defines a deformation cost � W � 2 = � W | W � Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) α 2 X ◮ probability p ( W ) ∝ exp ( − k ) : Gaussian distribution 2 λ 2 k k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) α 2 X ◮ probability p ( W ) ∝ exp ( − k ) : Gaussian distribution 2 λ 2 k k ◮ defines a Gaussian shape prior Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) X ◮ Empirical distribution : D emp = δ W M → Si i (possibly smoothed by a kernel) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) X ◮ Empirical distribution : D emp = δ W M → Si i (possibly smoothed by a kernel) ◮ Any inner product < | > P in tangent space of the mean ⇒ Gaussian distribution D P ( W ) ∝ exp( −� W � 2 = P ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) X ◮ Empirical distribution : D emp = δ W M → Si i (possibly smoothed by a kernel) ◮ Any inner product < | > P in tangent space of the mean ⇒ Gaussian distribution D P ( W ) ∝ exp( −� W � 2 = P ) ◮ Best P for Kullback-Leibler( D P |D emp ) : PCA! Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 1 : mean + modes model → example from my PhD thesis ֒ Automatic alignment − → and average shape computation Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Statistics (PCA) on deformation fields − → between the mean shape and each sample modes of deformation = deformation prior = Gaussian probabilistic model Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Example of application : image segmentation with shape prior without shape prior with shape prior Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Example of application : image segmentation with shape prior without shape prior with shape prior ◮ requires a mean shape (does not always make sense, e.g. person walking) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Example of application : image segmentation with shape prior without shape prior with shape prior ◮ requires a mean shape (does not always make sense, e.g. person walking) ◮ requires all deformations between the mean and samples : ⇒ relatively similar sample shapes (otherwise, not reliable) = Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ Kernel : symmetric definite positive function k ( x , y ) ◮ Expresses the similarity between x and y ◮ Typically, the Gaussian kernel : k ( x , y ) = exp ( − d ( x , y ) 2 ) ◮ For each point x i : k i ( y ) := k ( x i , y ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ Kernel : symmetric definite positive function k ( x , y ) ◮ Expresses the similarity between x and y ◮ Typically, the Gaussian kernel : k ( x , y ) = exp ( − d ( x , y ) 2 ) ◮ For each point x i : k i ( y ) := k ( x i , y ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ Kernel : symmetric definite positive function k ( x , y ) ◮ Expresses the similarity between x and y ◮ Typically, the Gaussian kernel : k ( x , y ) = exp ( − d ( x , y ) 2 ) ◮ For each point x i : k i ( y ) := k ( x i , y ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ Kernel : symmetric definite positive function k ( x , y ) ◮ Expresses the similarity between x and y ◮ Typically, the Gaussian kernel : k ( x , y ) = exp ( − d ( x , y ) 2 ) ◮ For each point x i : k i ( y ) := k ( x i , y ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ choice of a distance, of a kernel ? ◮ distance between 2 shapes : not much informative (wrt deformations) ◮ rebuild geometry of space of shapes from distances ? ◮ distances are not reliable/meaningful for far shapes ◮ = ⇒ needs for a representative neighborhood, i.e. a high dataset density (not affordable) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ choice of a distance, of a kernel ? ◮ distance between 2 shapes : not much informative (wrt deformations) ◮ rebuild geometry of space of shapes from distances ? ◮ distances are not reliable/meaningful for far shapes ◮ = ⇒ needs for a representative neighborhood, i.e. a high dataset density ◮ in a high-dimensional manifold, all distances are similar, and all points are on the boundary of the manifold ◮ = ⇒ cannot work, need for more information than distances Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Main idea ◮ consider deformations (not just distances) ◮ should not require high density of training set ◮ no magic (to handle/interpolate sparse sets) : add a prior Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Main idea ◮ consider deformations (not just distances) ◮ should not require high density of training set ◮ no magic (to handle/interpolate sparse sets) : add a prior ◮ prior chosen : transported deformations make sense, i.e. a deformation observed on one shape can be applied to other shapes Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Main idea ◮ consider deformations (not just distances) ◮ should not require high density of training set ◮ no magic (to handle/interpolate sparse sets) : add a prior ◮ prior chosen : transported deformations make sense, i.e. a deformation observed on one shape can be applied to other shapes ◮ transport requires correspondences Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Main idea ◮ consider deformations (not just distances) ◮ should not require high density of training set ◮ no magic (to handle/interpolate sparse sets) : add a prior ◮ prior chosen : transported deformations make sense, i.e. a deformation observed on one shape can be applied to other shapes ◮ transport requires correspondences ◮ but shape matching reliable only for close shapes Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Main idea ◮ consider deformations (not just distances) ◮ should not require high density of training set ◮ no magic (to handle/interpolate sparse sets) : add a prior ◮ prior chosen : transported deformations make sense, i.e. a deformation observed on one shape can be applied to other shapes ◮ transport requires correspondences ◮ but shape matching reliable only for close shapes ◮ = ⇒ compute correspondences between close shapes only, and combine small steps of reliable correspondences to build longer-distance correspondences Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Map ◮ Close shape matching ◮ Transport ◮ Metric estimation (statistics on transported deformations) ◮ Theoretical justifications Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Close shape matching Shape matching Simple case : two shapes, A and B, with one connected component Z � f � 2 + α �∇ f � 2 dA inf f : A → B A - shape sampling - dynamic time warping - theory & experiments : higher sampling rate on target A A (s+ds) (s) A f f (s) (s+ds) B B m(s) B m(s+ds) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Close shape matching Shape matching Simple case : two shapes, A and B, with one connected component Z � f � 2 + α �∇ f � 2 dA inf f : A → B A - shape sampling - dynamic time warping - theory & experiments : higher sampling rate on target Usual case : random topologies Usual cases = more complex (more than 10 connected components in this silhouette) but [ one connected component → connected components i = the same Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Close shape matching Further possible improvements ◮ as such, allows appearing points (mismatches) ◮ allows disappearing points : matching to ∅ with a fixed high cost ◮ pb : better matchings, but energy value loses meaning Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Close shape matching Further possible improvements ◮ as such, allows appearing points (mismatches) ◮ allows disappearing points : matching to ∅ with a fixed high cost ◮ pb : better matchings, but energy value loses meaning Drawbacks ◮ specific to planar curves ◮ not symmetric : m A → B � = m − 1 B → A Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Local transport Transport Local transport ◮ Set of shapes ( S i ) i ∈ I (e.g. from a video segmentation) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Local transport Transport Local transport ◮ Set of shapes ( S i ) i ∈ I (e.g. from a video segmentation) ◮ Two shapes S i and S j = ⇒ their correspondence field m i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Local transport Transport Local transport ◮ Set of shapes ( S i ) i ∈ I (e.g. from a video segmentation) ◮ Two shapes S i and S j = ⇒ their correspondence field m i → j ◮ Transport (translation, naive) : T L ∀ h : S j → X , j → i ( h ) : → X S i ` T L ´ j → i ( h ) ( s ) = h ( m i → j ( s )) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Local transport Transport Local transport ◮ Set of shapes ( S i ) i ∈ I (e.g. from a video segmentation) ◮ Two shapes S i and S j = ⇒ their correspondence field m i → j ◮ Transport (translation, naive) : T L ∀ h : S j → X , j → i ( h ) : S i → X T L ` ´ j → i ( h ) ( s ) = h ( m i → j ( s )) ⇒ reliability w L ◮ Associated cost : E ( m i → j ) = ` ´ i → j ∝ exp − α E ( m i → j ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Global transport ⇒ reliability w L ◮ Associated cost : E ( m i → j ) = ` ´ i → j ∝ exp − α E ( m i → j ) ◮ close shapes : reliable; distant shapes : not reliable ◮ = ⇒ search for paths of small steps in the training set ( S i ) ◮ graph : nodes = shapes, edges = transport, weights = transport cost ◮ shortest path between pairs of shapes : global transport Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Global transport ◮ Associated cost : E ( m i → j ) = ⇒ reliability w L i → j ∝ exp ` − α E ( m i → j ) ´ ◮ close shapes : reliable; distant shapes : not reliable ⇒ search for paths of small steps in the training set ( S i ) ◮ = ◮ graph : nodes = shapes, edges = transport, weights = transport cost ◮ shortest path between pairs of shapes : global transport ◮ compose : T G i → j = T L i n → j o T L o T L i 1 → i 2 o T L i n − 1 → i n o ... i → i 1 ◮ reliability : w G Y w L i → j = i k → i k +1 k ◮ use transport to propagate information Example : colored walker Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Correspondence field Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported arm rotation (translation) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported arm rotation (better) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Forearm rotation Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported forearm rotation Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported forearm rotation (better) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transport to another shape Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported forearm rotation (translation) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported forearm rotation (better) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported arm rotation (translation) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported arm rotation (better) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport L 2 inner-product > 0 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport L 2 inner-product = 0 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport L 2 inner-product < 0 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Remarks about transport Why ? ◮ transport : of deformations : needed to increase training set density Which ? ◮ “translation” : ok for short pathes ◮ transport : not obvious (muscles + T-shirt artifacts) ◮ criterion to assess transport quality / suitability ? ◮ transport : should be learned (from video sequences ?) ◮ path could depend on deformation transported What properties ? ◮ probability of a deformation transported : can differ ◮ inner product : no reason to be transport-invariant Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transport in differential geometry A connection ∇ is Riemannian if the parallel transport it defines preserves the metric g . Metric connection : ∇ X g ( · , · ) = 0 for all vector fields X on M ◮ not satisfied (probability of a deformation depends on the shape) = ⇒ not Riemannian : transport and metric are independent Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transport in differential geometry A connection ∇ is Riemannian if the parallel transport it defines preserves the metric g . Metric connection : ∇ X g ( · , · ) = 0 for all vector fields X on M ◮ not satisfied (probability of a deformation depends on the shape) = ⇒ not Riemannian : transport and metric are independent ⇒ connection Transport = Given transport, under few hypotheses (e.g. smoothness), it is possible to recover the associated infinitesimal connection : T h → 0 V γ ( h ) − V γ (0) ˛ = d γ dt T t → 0 ˛ ∇ X V = lim V γ ( t ) . γ ˛ h h → 0 ˛ t =0 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transport in differential geometry A connection ∇ is Riemannian if the parallel transport it defines preserves the metric g . Metric connection : ∇ X g ( · , · ) = 0 for all vector fields X on M ◮ not satisfied (probability of a deformation depends on the shape) = ⇒ not Riemannian : transport and metric are independent ⇒ connection Transport = Given transport, under few hypotheses (e.g. smoothness), it is possible to recover the associated infinitesimal connection : T h → 0 V γ ( h ) − V γ (0) ˛ = d γ dt T t → 0 ˛ ∇ X V = lim V γ ( t ) . γ ˛ h h → 0 ˛ t =0 Connection = ⇒ transport : Given a covariant derivative ∇ , the transport along a curve γ is obtained by integrating the condition ∇ ˙ γ = 0 . Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j ◮ statistics, for k fixed : PCA ◮ PCA with weights, and with H 1 -norm ⇒ eigenmodes e n (= principal deformations) with eigenvalues λ n ◮ = ⇒ defines an inner product P k = metric in the tangent space of the ◮ = shape S k ◮ P k varies smoothly as a function of k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Example of results Example of results : dancing sequence (9s, 24Hz), shape 1 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Example of results Example of results : shape 2 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Example of results Example of results : shape 3 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Theoretical justifications Theoretical justifications The best metric ? Searching for principal modes of deformations which vary smoothly (as a function of the shape S k ) ? ◮ vain quest : hairy ball theorem = ⇒ no best smooth direction field (or then it has to vanish sometimes) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Theoretical justifications Theoretical justifications The best metric ? Searching for principal modes of deformations which vary smoothly (as a function of the shape S k ) ? ◮ vain quest : hairy ball theorem = ⇒ no best smooth direction field (or then it has to vanish sometimes) Best metric for a given distribution (on one shape) ? ◮ = ⇒ PCA gives the best metric for a criterion based on Kullback-Leibler divergence between distributions Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Theoretical justifications Theoretical justifications The best metric ? Searching for principal modes of deformations which vary smoothly (as a function of the shape S k ) ? ◮ vain quest : hairy ball theorem = ⇒ no best smooth direction field (or then it has to vanish sometimes) Best metric for a given distribution (on one shape) ? ◮ = ⇒ PCA gives the best metric for a criterion based on Kullback-Leibler divergence between distributions Best metric for a given empirical manifold (all shapes together) ? ◮ needs a smoothness criterion ( = ⇒ transport) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Theoretical justifications Theoretical justifications The best metric ? Searching for principal modes of deformations which vary smoothly (as a function of the shape S k ) ? ◮ vain quest : hairy ball theorem = ⇒ no best smooth direction field (or then it has to vanish sometimes) Best metric for a given distribution (on one shape) ? ◮ = ⇒ PCA gives the best metric for a criterion based on Kullback-Leibler divergence between distributions Best metric for a given empirical manifold (all shapes together) ? ◮ needs a smoothness criterion ( = ⇒ transport) ◮ = ⇒ best metric for a criterion involving transport & K-L divergence. ⇒ best metric for another criterion involving transport & L 2 -norm of ◮ = distributions. Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k ◮ Empirical distributions : D emp i = P j w L i → j δ f i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k ◮ Empirical distributions : D emp i = P j w L i → j δ f i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k ◮ Empirical distributions : D emp i = P j w L i → j δ f i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k ◮ Empirical distributions : D emp i = P j w L i → j δ f i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k j w L ◮ Empirical distributions : D emp i = P i → j δ f i → j ◮ Transported distribution : via T i → k ( δ f ) = δ T i → k ( f ) . X w G ◮ Criterion : best ( P k ) for KL ( D P k | T i → k ( D emp i )) ik i , k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k ◮ Empirical distributions : D emp i = P j w L i → j δ f i → j ◮ Transported distribution : via T i → k ( δ f ) = δ T i → k ( f ) . X w G ◮ Criterion : best ( P k ) for KL ( D P k | T i → k ( D emp i )) ik i , k X KL ( D P k |D T ◮ = best ( P k ) for emp k ) k where D T X w k emp k = i → j δ f k i → j i , j ◮ Transported deformations to any shape S k : f k i → j = T G i → k ( f i → j ) with reliability weights w k i → j = w G i → k w L i → j ◮ = the one obtained by weighted PCA on transported deformations Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold (again!) ◮ empirical distributions : D emp i ◮ kernel-smoothed empirical distributions : D K emp i = g 0 i d µ ◮ g 0 i : density functions in the tangent space of the shape S i Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes