Roto-Translation Equivariant Convolution Networks for Medical Image - PowerPoint PPT Presentation

Roto-Translation Equivariant Convolution Networks for Medical Image Analysis Erik J Bekkers, Remco Duits Dep. Math. & Computer Science CASA TU/e Lie Analysis Based on work by Bekkers 1 , Lafarge 2 , Veta 2 , Eppenhof 2 , Pluim 2 , Duits 1 (MICCAI 2018) _______2019/05/04________Imaging & Machine Learning_______Institut Henri Poincaré, Paris, France________ 1 Department of Mathematics and Computer Science, and 2 Department of Biomedical Engineering, Eindhoven University of Technology

Presentation outline 2

Presentation outline • Motivation 1: Geometric image analysis via orientation scores needs automation. • Motivation 2: Machine learning needs group equivariance. • Overview of related work. • Theoretical background: • Neural networks (NNs) • Convolutional neural networks (CNNs) • Group convolutional neural networks (G-CNNs) General Theorem on Equivariant linear operators Architecture G-CNNs. • Results on 3 different medical imaging applications. • Conclusion. 3

Equivariant image analysis via orientation scores 4

Invertible Orientation scores Image Image OS Orientation Score (real part) Curved & Torqued Geometry of roto-translation group SE(2) visible in Score: Image OS PDEs Moving frame Cake wavelets For tracking Gabor wavelets and regularization PhD Thesis R. Duits 2005 PhD Thesis E.M. Franken 2009 PhD Thesis E.J. Bekkers 2017 PhD Thesis J.M. Portegies 2018 5

SE(d) equivariant processing via orientation scores JMIV 2018 SIIMS 2016 JDCS 2016 JMIV 2018 6

Diffusion & Brownian Motions in Roto-Translation Group 2 nd Workshop IHP: “PDEs on the Homogeneous Space of Positions and Orientations.” R. Duits QAM-AMS 2008, 2010 ISKR BM3D DGA 2018 Entropy 2019 Ne Next: : Machine Learning to train such kernels and their offsets. 7 2014-onwards incl. IEEE-PAMI E.J. Bekkers & R. Duits et al.

Template matching via Group convolutions ICIAR 2014 & PAMI 2018 Bekkers-Loog-tHRomeny-Duits State-of-the-Art Results (99,8 % success rate) on 3 Detection Tasks . E.g. Optic Disc Detection via Template Matching using group convolutions in SE(2) image isotropic oriented Template structures structures Special Case 2 layer (optic disk) (vessels) score # Group CNN ! " ! 8

2 layer G-CNN: ICIAR 2014 & PAMI 2018 Bekkers-Loog-Duits • Minimize Functional: Convex-problem. training label patch Train parameters via • With Sub-Riemannian Brownian prior Generalized Cross-Validation: • Expansion in B-splines Output: 1 delta patch Accurate: Comparison to exact sol’s Duits 9

Motivation 1 Extend from 2 Layer G-CNN for template matching in OS to Deep Learning in OS 10

Related work Group equivariant networks Group convolution networks Steerable filter networks (domain extension) (co-domain extension) ℤ " LeCun et al 1990 translation networks Worrall et al. 2017 SE(2) irrep Mallat et al. 2013, 2015 SE(2) Scattering transform & SVM Marcos et al. 2017 SE(2) vector field networks Bekkers et al. 2014-2018 SE(2) via B-splines, 2 layer G-CNN Kondor 2018 SE(3) irrep, N-body nets via 90 o rotations + flips + theory! Cohen-Welling 2016 p4m Thomas et al. 2018 SE(3) irrep, point clouds via 90 o rotations + flips Dieleman et al. 2016 p4m Weiler et al. 2018 SE(3) irrep Weiler et al. 2017 SE(2) via circular harmonics Esteves SO(3)/SO(2) irrep Zhou et al. 2017 SE(2) via bilinear interpolation Kondor-Trivedi 2018 SO(d) irrep (on compact Bekkers et al. 2018 SE(2) via bilinear interpolation quotient sp.) Hoogeboom et al. 2018 S(2,6) hexagonal grids Winkels-Cohen 2018 SE(3,N) + m 90 o rotations + flips Continuous 90 o rotations Worrall-Brostow 2018 SE(3,N) Discrete Cohen et al. 2018 SO(3) via spherical harmonics 11

Motivation 2 Roto-Translation Covariant Convolution Neural Networks for Medical Image Analysis 12

Many computer vision and MedIA problems require invariance/equivariance properties (e.g. w.r.t. rotation) Rotation-invariant detection of pathological cells (mitotic figure) in histopathology 13

Transformation invariance Classification via CNN: CNN Label (e.g. 1: healthy cell, 0: pathological cell) Input image Transformation: “healthy cell” Geometric transformation by (e.g. rotation) 14

Transformation invariance by data augmentation Example: vessel segmentation Training set ( naïve example of only vertically aligned vessels ) Blood vessels (vertically aligned) Background A simple network will do Conv layer “vessel” “vessel” ?? “background” 15

Transformation invariance by data augmentation Example: vessel segmentation Expand training set with rotated copies -> data augmentation Blood vessels Background A more complex network is required Now you learn Conv layer rotated versions of “vessel” the same feature.. “background” 16

Redundancy in feature representations ImageNet Challenge Learned convolution filters in the first layer 17

Symmetries in Medical Imaging Problems with classical CNNs: - No guarantee of equivariance (other then translation). - Redundancy in feature representation. - Artificially create extra data samples by data augmentation. Solution: Group convolutional neural networks: G-CNNs Moreover, G-CNNs: - Exploit symmetries in data. - Weight sharing. - Perform better by not having to spend valuable network capacity on learning geometric properties. 18

Theory: From NNs to CNNs to G-CNNs NNs Add geometric structure (local spatial coherence, weight sharing, …) CNNs Adding more geometric structure (beyond translation equivariance) G-CNNs 19

Problem description Given a training set consisting of pairs of inputs and desired outputs , find a function , parameterized by , that best maps each input to a desired output. Here “best” is quantified by (local) minima of a loss: computed by stochastic gradient descent (backward propagation via chain-law) Training set Testing , 0: normal , 0: normal , 1: mitotic 0.14 … “probably a healthy cell” 20

Neural Network: • A Neural Network is a composition of operators: • In which each operator (“layer”) has the following form: 21

A classical neural network With “soft-max” Boltzmann distr. Class probability 1 3 5 7 9 0 0,2 Probability 2D function/image 2D array 1D vector 22

A classical neural network 23

A fully connected neural layer -Too many degrees of freedom -Does not exploit structure in data 24

A convolution layer + Localized transformations + Shift equivariance + Sparsification of the linear operator + Weight sharing 25

Theory: From NNs to CNNs to G-CNNs NNs CNNs G-CNNs 26

Input and output spaces Input vector Output vector Convolution layer 27

Convolution layer (cross-correlation layer) Translation by x 28

Convolution layer (cross-correlation layer) Translation by x template matching 29

Convolution layers on ℝ " A linear mapping parameterized by a collection of convolution filters 30

A typical CNN architecture LeCun et al. 1989-1998 31

Theory: From NNs to CNNs to G-CNNs NNs CNNs G-CNNs 32

The translation group 33

The roto-translation group SE(2) 34

Group Representations of SE(2) The representation of SE(2) on The representation of SE(2) on Shift-twist 35

Group correlation layers Lifting layer: G-correlation layer: 36

Group correlation layers SE(2) equivariance Lifting layer: G-correlation layer: 37

Theorem on Equivariant Linear Operators motivating the G-CNN Design 38

39

40

G-CNN design for rotation invariant patch classification Bekkers et al. “normal” (0) vs “mitotic” (1) MICCAI 2018 ICIAR 2014 (2 layers) Max-pooling over rotations only Equivariance. Better choices… Thm . equivariant linear operator design ! (in practice interleafed ReLu: not affecting equiv.) d e r e t c y e a r l n e v n n y n a o o o l v i c v t v c t n c n y u n o e g l o p o l c j n u c o t c - i - F u G - r t G G P o f i L Class probability Input image 41

Steerable Implementations possible via Fourier Transform on the homogeneous space of positions and orientations: 42

Results 43

Example (rotation equivariance and invariance) Invariance ! 44

Results histopathology Optical image of eye Electron microscopy Same capacity But no waist as N increases. 45

Conclusion 46

Conclusion 47