Learning on manifolds and graphs with intrinsic CNNs Michael - PowerPoint PPT Presentation

Learning on manifolds and graphs with intrinsic CNNs Michael Bronstein University of Lugano Tel Aviv University Intel Corporation Switzerland Israel Israel 3DDL NIPS Workshop, Barcelona, 9 December 2016 1/49

2/49

$100K $100 $20 2005 2010 2014 2/49

(Acquired by Intel in 2012) 4/49

Applications Markerless motion capture Gesture control 5/49

Basic problems: shape similarity and correspondence Isometric 6/49

Basic problems: shape similarity and correspondence Isometric Non-isometric 6/49

Basic problems: shape similarity and correspondence Isometric Non-isometric Partial 6/49

Basic problems: shape similarity and correspondence Isometric Non-isometric Partial Different representation 6/49

Task-specific features Correspondence 7/49

Task-specific features Correspondence Similarity ... 7/49

Deep learning in computer vision Error % 30 20 “Deep learning era” in vision 10 2.9% 2010 2011 2012 2013 2014 2015 2016 ImageNet ILSVRC Challenge 8/49

Deep learning in computer graphics Volumetric 1 Single view based 2 Multiple view based 3 1 Wu et al. 2015; 2 Wei et al. 2016; 3 Su et al. 2015 9/49

Extrinsic vs Intrinsic CNNs Extrinsic Intrinsic 10/49

What is convolution on manifolds? 11/49

Convolution Euclidean Non-Euclidean Spatial domain � π ( f ⋆ g )( x ) = f ( ξ ) g ( x − ξ ) dξ − π 12/49

Convolution Euclidean Non-Euclidean Spatial domain � π ( f ⋆ g )( x ) = f ( ξ ) g ( x − ξ ) dξ − π Spectral domain � ( f ⋆ g )( ω ) = ˆ f ( ω ) · ˆ g ( ω ) ‘Convolution Theorem’ 12/49

Convolution Euclidean Non-Euclidean Spatial domain � π ? ( f ⋆ g )( x ) = f ( ξ ) g ( x − ξ ) dξ − π Spectral domain ? � ( f ⋆ g )( ω ) = ˆ f ( ω ) · ˆ g ( ω ) ‘Convolution Theorem’ 12/49

Fourier analysis (Euclidean spaces) A function f : [ − π, π ] → R can be written as Fourier series � π 1 � f ( ξ ) e − ikξ dξ e ikx f ( x ) = 2 π − π ω ˆ ˆ ˆ = + + + . . . f 1 f 2 f 3 13/49

Fourier analysis (Euclidean spaces) A function f : [ − π, π ] → R can be written as Fourier series � π 1 � f ( ξ ) e − ikξ dξ e ikx f ( x ) = 2 π − π ω � �� � ˆ f k = � f,e ikx � L 2([ − π,π ]) ˆ ˆ ˆ = + + + . . . f 1 f 2 f 3 13/49

Fourier analysis (Euclidean spaces) A function f : [ − π, π ] → R can be written as Fourier series � π 1 � f ( ξ ) e − ikξ dξ e ikx f ( x ) = 2 π − π ω � �� � ˆ f k = � f,e ikx � L 2([ − π,π ]) ˆ ˆ ˆ = + + + . . . f 1 f 2 f 3 Fourier basis = Laplacian eigenfunctions: ∆ e ikx = k 2 e ikx We define Laplacian as a positive semi-definite operator ∆ = − d 2 dx 2 13/49

Fourier analysis (non-Euclidean spaces) A function f : X → R can be written as Fourier series � � f ( x ) = f ( ξ ) φ k ( ξ ) dξ φ k ( x ) X k ≥ 0 � �� � ˆ f k = � f,φ k � L 2( X ) ˆ ˆ ˆ = f 1 + f 2 + f 3 + . . . f φ 1 φ 2 φ 3 Fourier basis = Laplacian eigenfunctions: ∆ φ k ( x ) = λ k φ k ( x ) 14/49

Laplacian operator Laplacian ∆: L 2 ( X ) → L 2 ( X ) x ∆ f = − div( ∇ f ) “difference between f ( x ) and f average value of f around x ” 15/49

Laplacian operator Laplacian ∆: L 2 ( X ) → L 2 ( X ) x ∆ f = − div( ∇ f ) “difference between f ( x ) and f average value of f around x ” Intrinsic (expressed solely in terms of the Riemannian metric) 15/49

Laplacian operator Laplacian ∆: L 2 ( X ) → L 2 ( X ) x ∆ f = − div( ∇ f ) “difference between f ( x ) and f average value of f around x ” Intrinsic (expressed solely in terms of the Riemannian metric) Isometry-invariant 15/49

Laplacian operator Laplacian ∆: L 2 ( X ) → L 2 ( X ) x ∆ f = − div( ∇ f ) “difference between f ( x ) and f average value of f around x ” Intrinsic (expressed solely in terms of the Riemannian metric) Isometry-invariant Self-adjoint � ∆ f, g � L 2 ( X ) = � f, ∆ g � L 2 ( X ) 15/49

Laplacian operator Laplacian ∆: L 2 ( X ) → L 2 ( X ) x ∆ f = − div( ∇ f ) “difference between f ( x ) and f average value of f around x ” Intrinsic (expressed solely in terms of the Riemannian metric) Isometry-invariant Self-adjoint � ∆ f, g � L 2 ( X ) = � f, ∆ g � L 2 ( X ) ⇒ orthogonal eigenfunctions 15/49

Laplacian operator Laplacian ∆: L 2 ( X ) → L 2 ( X ) x ∆ f = − div( ∇ f ) “difference between f ( x ) and f average value of f around x ” Intrinsic (expressed solely in terms of the Riemannian metric) Isometry-invariant Self-adjoint � ∆ f, g � L 2 ( X ) = � f, ∆ g � L 2 ( X ) ⇒ orthogonal eigenfunctions Positive semidefinite 15/49

Laplacian operator Laplacian ∆: L 2 ( X ) → L 2 ( X ) x ∆ f = − div( ∇ f ) “difference between f ( x ) and f average value of f around x ” Intrinsic (expressed solely in terms of the Riemannian metric) Isometry-invariant Self-adjoint � ∆ f, g � L 2 ( X ) = � f, ∆ g � L 2 ( X ) ⇒ orthogonal eigenfunctions Positive semidefinite ⇒ non-negative eigenvalues 15/49

Convolution Euclidean Non-Euclidean Spatial domain � π ? ( f ⋆ g )( x ) = f ( ξ ) g ( x − ξ ) dξ − π Spectral domain � � ( f ⋆ g )( ω ) = ˆ ( f ⋆ g ) k = � f, φ k � L 2 ( X ) � g, φ k � L 2 ( X ) f ( ω ) · ˆ g ( ω ) ‘Convolution Theorem’ 16/49

Spectral convolution Filtered function ˜ Function f f Henaff, Bruna, LeCun 2015; Defferrard, Bresson, Vandergheynst 2016 17/49

Spectral convolution Filtered function ˜ Function f Same function, f same filter, another shape Henaff, Bruna, LeCun 2015; Defferrard, Bresson, Vandergheynst 2016 17/49

Spectral convolution Filtered function ˜ Function f Same function, f same filter, another shape Filter is basis dependent Henaff, Bruna, LeCun 2015; Defferrard, Bresson, Vandergheynst 2016 17/49

Spectral convolution Filtered function ˜ Function f Same function, f same filter, another shape Filter is basis dependent ⇒ does not generalize across domains! Henaff, Bruna, LeCun 2015; Defferrard, Bresson, Vandergheynst 2016 17/49

Convolution in the spatial domain A B C D E A B C D E F G H I J F G H I J K L M N O K L M N O P R S T U P R S T U V W X Y Z V W X Y Z A B C D E A B C D E F G H I J F G H I J K L M N O K L M N O P R S T U P R S T U V W X Y Z V W X Y Z Euclidean Non-Euclidean No canonical global system of coordinates 18/49

Convolution in the spatial domain A B C D E A B C D E F G H I J F G H I J K L M N O K L M N O P R S T U P R S T U V W X Y Z V W X Y Z A B C D E A B C D E F G H I J F G H I J K L M N O K L M N O P R S T U P R S T U V W X Y Z V W X Y Z Euclidean Non-Euclidean No canonical global system of coordinates No grid structure (no regular memory access) 18/49

Convolution in the spatial domain A B C D E A B C D E F G H I J F G H I J K L M N O K L M N O P R S T U P R S T U V W X Y Z V W X Y Z A B C D E A B C D E F G H I J F G H I J K L M N O K L M N O P R S T U P R S T U V W X Y Z V W X Y Z Euclidean Non-Euclidean No canonical global system of coordinates No grid structure (no regular memory access) No shift-invariance (patch operator is position-dependent) 18/49

Convolution Euclidean Non-Euclidean Spatial domain � π � ( f ⋆ g )( x ) = ( D ( x ) f )( u ) g ( u ) d u ( f ⋆ g )( x ) = f ( ξ ) g ( x − ξ ) dξ − π Spectral domain � ( f ⋆ g )( ω ) = ˆ � f ( ω ) · ˆ g ( ω ) ( f ⋆ g ) k = � f, φ k � L 2 ( X ) � g, φ k � L 2 ( X ) ‘Convolution Theorem’ 19/49

Patch operator � ( f ⋆ g )( x ) = d u × ( D ( x ) f )( u ) g ( u ) Masci, Boscaini, B, Vandergheynst 2015; Boscaini, Masci, Rodol` a, B 2016 20/49

Heat diffusion on manifolds f t = − c ∆ f Newton’s law of cooling: rate of change of the temperature of an object is proportional to the difference between its own temperature and the temperature of the surrounding c [m 2 /sec] = thermal diffusivity constant 21/49

Heat diffusion on manifolds � f t ( x, t ) = − ∆ f ( x, t ) f ( x, 0) = f 0 ( x ) f ( x, t ) = amount of heat at point x at time t f 0 ( x ) = initial heat distribution 22/49

Heat diffusion on manifolds � f t ( x, t ) = − ∆ f ( x, t ) f ( x, 0) = f 0 ( x ) f ( x, t ) = amount of heat at point x at time t f 0 ( x ) = initial heat distribution Solution of the heat equation expressed through the heat operator e − t ∆ f 0 ( x ) f ( x, t ) = 22/49

Heat diffusion on manifolds � f t ( x, t ) = − ∆ f ( x, t ) f ( x, 0) = f 0 ( x ) f ( x, t ) = amount of heat at point x at time t f 0 ( x ) = initial heat distribution Solution of the heat equation expressed through the heat operator � e − t ∆ f 0 ( x ) = � f 0 , φ k � L 2 ( X ) e − tλ k φ k ( x ) f ( x, t ) = k ≥ 0 22/49