Non-rigid Registration Marcel Lthi Graphics and Vision Research - PowerPoint PPT Presentation

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Non-rigid Registration Marcel Lüthi Graphics and Vision Research Group Department of Mathematics and Computer Science University of Basel

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Outline • Non-rigid registration: The basic formulation • Exercise: Parametric registration in Scalismo • Advanced Priors • Likelihood functions • Exercise: ASMs in Scalismo • Optimization

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL The registration problem 𝑦 𝜒: Ω → Ω Ω Ω 𝜒(𝑦) Reference: Target: 𝐽 𝑆 : Ω → ℝ 𝐽 𝑈 : Ω → ℝ 3

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Why is it important? • Do automatic measurements • Compare shapes • Statistics 𝜒: Ω → Ω • Build statistical models • Transfer labels and annotations Ω Ω • Atlas based segmentation Maybe the most important problem in computer vision and medical image analysis

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Registration as analysis by synthesis Comparison: 𝑞 𝐽 𝑈 𝜄, 𝐽 𝑆 ) Prior 𝜒[𝜄] ∼ 𝑞(𝜄) 𝐽 𝑈 𝐽 𝑆 ∘ 𝜒[𝜄] Parameters 𝜄 Update using 𝑞(𝜄|𝐽 𝑈 , 𝐽 𝑆 ) Synthesis 𝜒[𝜄]

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL The registration problem Probabilistic formulation 𝜄 ∗ = arg max 𝑞 𝜄 𝐽 𝑈 , 𝐽 𝑆 = arg max 𝑞 𝜄 𝑞(𝐽 𝑈 |𝜄, 𝐽 𝑆 ) 𝜄 𝜄 Mapping 𝜒[𝜄 ∗ ] is trade-off that • how well does the mapping explain the target image (likelihood function) • matches the prior assumptions (prior distribution)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL The registration problem 𝜄 ∗ = arg max 𝑞 𝜄 𝐽 𝑈 , 𝐽 𝑆 = arg max 𝑞 𝜄 𝑞(𝐽 𝑈 |𝜄, 𝐽 𝑆 ) 𝜄 𝜄 𝜒[𝜄] Ω Ω

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL The registration problem 𝜄 ∗ = arg max 𝑞 𝜄 𝐽 𝑈 , 𝐽 𝑆 = arg max 𝑞 𝜄 𝑞(𝐽 𝑈 |𝜄, 𝐽 𝑆 ) 𝜄 𝜄 𝜒[𝜄] Ω Ω

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL The registration problem 𝜄 ∗ = arg max 𝑞 𝜄 𝐽 𝑈 , 𝐽 𝑆 = arg max 𝑞 𝜄 𝑞(𝐽 𝑈 |𝜄, 𝐽 𝑆 ) 𝜄 𝜄 𝜒[𝜄] Ω Ω

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL The registration problem Probabilistic formulation 𝜒 ∗ = arg max 𝑞 𝜒 𝐽 𝑈 , 𝐽 𝑆 = arg max 𝑞 𝜒 𝑞(𝐽 𝑈 |𝜒, 𝐽 𝑆 ) 𝜒 𝜒 Main questions: • How do we represent the mapping? • How do we define the prior? • What is the likelihood function? • How can we solve the optimization problem?

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Representation of the mapping 𝜒

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Representation of the mapping 𝜒

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Representation of the mapping 𝜒 Assumption: Images are rigidly aligned • Mapping can be represented as a displacement vector field: 𝑦 u( 𝑦) 𝜒 𝑦 = 𝑦 + 𝑣 𝑦 𝑣 ∶ Ω → ℝ 𝑒

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Representation of the mapping 𝜒 Assumption: Images are rigidly aligned • Mapping can be represented as a displacement vector field: 𝜒 𝑦 = 𝑦 + 𝑣 𝑦 𝑣 ∶ Ω → ℝ 𝑒 Observation: Knowledge of 𝑣 and 𝐽 𝑆 allows us to synthesize target image 𝐽 𝑈

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Registration as analysis by synthesis Comparison: 𝑞 𝐽 𝑈 𝜄, 𝐽 𝑆 ) Prior 𝜒[𝜄] ∼ 𝑞(𝜄) 𝐽 𝑈 𝐽 𝑆 ∘ 𝜒[𝜄] Parameters 𝜄 Update using 𝑞(𝜄|𝐽 𝑈 , 𝐽 𝑆 ) Synthesis 𝜒[𝜄]

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Priors Define the Gaussian process 𝑣 ∼ 𝐻𝑄 𝜈, 𝑙 with mean function 𝜈: Ω → ℝ 2 and covariance function 𝑙: Ω × Ω → ℝ 2×2 .

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Example prior: Smooth 2D deformations Zero mean: 𝜈 𝑦 = 0 0 Squared exponential covariance function (Gaussian kernel) s 1 exp − 𝑦 − 𝑦 ′ 2 0 2 𝜏 1 𝑙 𝑦, 𝑦 ′ = s 2 exp − 𝑦 − 𝑦 ′ 2 0 2 𝜏 2

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Example prior: Smooth 2D deformations 𝑡 1 = 𝑡 2 small, 𝜏 1 = 𝜏 2 large

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Example prior: Smooth 2D deformations 𝑡 1 = 𝑡 2 small, 𝜏 1 = 𝜏 2 small

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Example prior: Smooth 2D deformations 𝑡 1 = 𝑡 2 large, 𝜏 1 = 𝜏 2 large

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Parametric representation of Gaussian process Represent 𝐻𝑄(𝜈, 𝑙) using only the first 𝑠 components of its KL-Expansion 𝑠 𝑣 = 𝜈 + ෍ 𝛽 𝑗 𝜇 𝑗 𝜚 𝑗 , 𝛽 𝑗 ∼ 𝑂(0, 1) 𝑗=1 • We have a finite, parametric representation of the process. • We know the pdf for a deformation 𝑣 𝑠 1 2 /2) = 1 𝑎 exp(− 1 2 𝛽 2 ) 𝑞 𝑣[𝛽] = 𝑞 𝛽 = ෑ exp(−𝛽 𝑗 2𝜌 𝑗=1

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Registration as analysis by synthesis Comparison: 𝑞 𝐽 𝑈 𝜄, 𝐽 𝑆 ) Prior 𝜒[𝜄] ∼ 𝑞(𝜄) 𝐽 𝑈 𝐽 𝑆 ∘ 𝜒[𝜄] Parameters 𝜄 Update using 𝑞(𝜄|𝐽 𝑈 , 𝐽 𝑆 ) Synthesis 𝜒[𝜄]

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Likelihood function: Image registration Images are similar when the intensities match Assumptions: • Corresponding points have the same image intensity (up to i.i.d. noise) 𝑦 𝜒[𝜄](𝑦) 𝐽 𝑆 𝐽 𝑈 𝑞 𝐽 𝑈 (𝜒[𝜄](𝑦)) 𝐽 𝑆 , 𝜄, 𝑦 ∼ 𝑂 𝐽 𝑆 𝑦 , 𝜏 2

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Likelihood function: Image registration Images are similar when the intensities match Assumptions: • Corresponding points have the same image intensity (up to i.i.d. noise) 𝑦 Image term outside mapping function. Makes problem 𝜒[𝜄](𝑦) 𝐽 𝑆 really difficult 𝐽 𝑈 − 𝐽 𝑆 𝑦 ) 2 𝑎 exp − (𝐽 𝑈 𝜒 𝑦 1 𝑞 𝐽 𝑈 𝐽 𝑆 , 𝜄 = ෑ 𝑞 𝐽 𝑈 (𝜒[𝜄](𝑦)) 𝐽 𝑆 , 𝜄, 𝑦 = ෑ 𝜏 2 𝑦∈Ω 𝑦∈Ω

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Registration as analysis by synthesis Comparison: 𝑞 𝐽 𝑈 𝜄, 𝐽 𝑆 ) Prior 𝜒[𝜄] ∼ 𝑞(𝜄) 𝐽 𝑈 𝐽 𝑆 ∘ 𝜒[𝜄] Parameters 𝜄 Update using 𝑞(𝜄|𝐽 𝑈 , 𝐽 𝑆 ) Synthesis 𝜒[𝜄]

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Registration problem 𝜄 ∗ = arg max 𝑞 𝜒 𝜄 𝑞(𝐽 𝑈 |𝜒[𝜄], 𝐽 𝑆 ) 𝜄 𝛽 ∗ = 2 1 exp − 1 1 exp − 𝐽 𝑈 𝜒 𝜄 𝑦 − 𝐽 𝑆 (𝑦)) 2 𝜄 2 = arg max ෑ 𝜏 2 𝑎 1 𝑎 2 𝜄 𝑦 • Parametric problem, since: 𝑠 𝜒[𝜄](𝑦) = 𝑦 + 𝜈(𝑦) + ෍ 𝜄 𝑗 𝜇 𝑗 𝜚 𝑗 (𝑦) 𝑗=1 • Can be optimized using gradient descent

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Variational formulation 2 1 exp − 1 1 exp − 𝐽 𝑈 𝜒 𝜄 𝑦 − 𝐽 𝑆 (𝑦)) 𝛽 ∗ = 2 𝜄 2 arg max ෑ 𝜏 2 𝑎 1 𝑎 2 𝜄 𝑦 2 ln 1 exp − 1 + ln 1 exp − 𝐽 𝑈 𝜒 𝜄 𝑦 − 𝐽 𝑆 (𝑦)) 2 𝜄 2 arg max ෑ 𝜏 2 𝑎 1 𝑎 2 𝜄 𝑦 2 ln 1 − 1 2 𝜄 2 + ln 1 𝐽 𝑈 𝜒 𝜄 𝑦 − 𝐽 𝑆 (𝑦)) = arg max − ෍ 𝜏 2 𝑎 1 𝑎 2 𝜄 𝑦∈Ω 2 𝐽 𝑈 𝜒 𝜄 𝑦 − 𝐽 𝑆 (𝑦)) + 𝜇 2 𝜄 2 = arg min ෍ 𝜏 2 𝜄 𝑦∈Ω Regularizer Image metric

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL The registration problem Probabilistic formulation 𝜄 ∗ = arg min 𝜄 − ln 𝑞 𝐽 𝑈 𝐽 𝑆 , 𝜒 𝜄 − ln 𝑞 𝜒 𝜄 Variational formulation 𝜄 ∗ = arg min 𝜄 𝐸 𝐽 𝑈 , 𝐽 𝑆 , 𝜒[𝜄] + 𝜇𝑆[𝜒 𝜄 ]

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Exercise: Registration in Scalismo Type into the codepane: goto (“http://shapemodelling.cs.unibas.ch/exercises/Exercise14.html”) Scalismo 0.16: Check examples in https://github.com/unibas-gravis/pmm2018

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL A selection of Gaussian process priors

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Why are priors interesting? 𝜄 ∗ = arg max 𝑞 𝜒 𝜄 𝑞(𝐽 𝑈 |𝐽 𝑆 , 𝜒[𝜄]) 𝜄 𝜄

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2018 | BASEL Why are priors interesting? 𝜄 ∗ = arg max 𝑞 𝜒 𝜄 𝑞(𝐽 𝑈 |𝐽 𝑆 , 𝜒[𝜄]) 𝜄 𝜄