2D Face Image Analysis Probabilistic Morphable Models Summer - PowerPoint PPT Presentation

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL 2D Face Image Analysis Probabilistic Morphable Models Summer School, June 2017 Sandro Schönborn University of Basel

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Contents Landmarks Fitting Observed Landmarks in 2D Image Fitting Observed Image 2

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL 2D Face Image Analysis Morphable Model adaptation to explain image Bayesian Inference Setup 𝑄 𝜄 𝐽 ∝ ℓ 𝜄; 𝐽 𝑄(𝜄) Image Likelihood Image as observation 𝐺 Face & Feature point detection Fast bottom-up methods 3

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL 3D Face Reconstruction 4

� � > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Fitting as Probabilistic Inference • Probabilistic Inference Problem: 𝑄 𝜄 𝐽) = 𝑄 𝐽 𝜄)𝑄(𝜄) 𝑂 𝐽 = ∫ 𝑄 𝐽 𝜄)𝑄(𝜄)d𝜄 𝑂 𝐽 • Prior : 𝑄 𝜄 • Likelihood : 𝑄 𝐽 𝜄) Statistical face model Image is observation K 𝐽 Face shape & color (PPCA/GP models): 𝐶 𝑡 B = 𝜈 + 𝑉𝐸𝛽 𝛽~ 𝑂 0, 𝐽 J 𝐺 Scene: illumination, pose, camera 3 𝜄 , 𝜏 6 𝐽 7 ℓ 𝜄; 𝐽 = / 𝒪 𝐽 1 | 𝐽 1 / 𝑐 <= 𝐽 1 1∈: >∈? 5

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL MH Inference of the 3DMM • Target distribution is our “posterior”: M 𝜄 𝐽 = ℓ 𝜄; 𝐽 𝑄 𝜄 𝑄: 𝑄 • Unnormalized • Point-wise evaluation only • Parameters • Shape: 50 – 200, low-rank parameterized GP shape model • Color: 50 – 200, low-rank parameterized GP color model • Pose/Camera: 9 parameters, pin-hole camera model • Illumination: 9*3 Spherical Harmonics illumination/reflectance ≈ 300 dimensions (!!) 6

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Metropolis Algorithm MH Algorithm filters samples with stochastic accept/reject steps Proposal Accept with probability 𝛽 = min 𝑄(𝜄 O |𝐽) 𝑄(𝜄|𝐽) , 1 draw proposal 𝜄 O 𝜄′ 𝑄(𝜄 O |𝐽) 𝑅(𝜄 O |𝜄) Update 𝜄 ← 𝜄′ 1 − 𝛽 reject 𝜄 • Asymptotically generates samples 𝜄 1 ∼ 𝑄(𝜄|𝐽) : 𝜄 X , 𝜄 6 , 𝜄 7 , … • Markov chain Monte Carlo (MCMC) Method • Works with unnormalized , point-wise posterior 7

� � > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Proposals • Choose simple Gaussian random walk proposals (Metropolis) "𝑅 𝜄 O |𝜄 = 𝑂(𝜄 O |𝜄, Σ \ )" • Normal perturbations of current state • Block-wise to account for different parameter types 6 𝐹 ` ) • Shape 𝑂(𝜷′|𝜷, 𝜏 ^ 6 𝐹 b ) • Color 𝑂(𝜸′|𝜸, 𝜏 b O |𝜄 d , 𝜏 d 6 ) • Camera ∑ 𝑂(𝜄 d �d 6 𝐹 e ) O |𝜄 e , 𝜏 e,1 ∑ 𝑂(𝜄 e • Illumination 1 In practice, we often add • Large mixture distributions, e.g. more complicated proposals, e.g. shape scaling, a direct 2 3 𝑅 h 𝜄 O 𝜄 + 1 illumination estimation and e (𝜄 O |𝜄) 3 i 𝜇 1 𝑅 1 decorrelation 1 8

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Landmarks Fitting Face Model Target Landmarks Rendered Landmarks Projection Prior 𝑄 𝜄 l ∝ 𝑄 𝒚 l 𝒚 𝜄 Likelihood ℓ 𝜄; 𝒚 Variable Parameters • Pose Shape • 9

� > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL 3DMM Landmarks Likelihood Simple models: Independen Independent Gaus ussians ns • Observation of landmark locations in image • Single landmark position model: 6m 𝜄 = T op ∘ Pr ∘ T to ∘ ℎ 𝜷 7m 𝒚 1 𝒚 1 T to 𝒚 = 𝑆 z,{,| 𝒚 + 𝒖 𝑥 2 ∗ 𝑦 𝑨 6m = 𝑂 𝒚 6m 𝜄 , 𝜏 vt (T op ∘ Pr)(𝒚) = + 𝒖 ƒƒ 6m |𝒚 1 6 − ℎ 2 ∗ 𝑧 ℓ 1 𝜄; 𝒚 l 1 l 1 𝑨 • Independent model 6m } 1 = / ℓ 𝜄; 𝒚 6m ℓ 𝜄; {𝒚 l 1 l 1 1 • Independence and Gaussian are just simple models (questionable) 10

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Landmarks: Samples 11

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Results: Landmarks • Landmarks posterior: Manual labelling: 𝜏 vt = 4 pix Image: 512x512 • Certainty of pose fit • Influence of ear points? • Frontal better than sideview? Yaw, σ 𝐌𝐍 = 4pix wi with ears w/ w/o ears 1.4 ∘ ± 𝟏. 𝟘 ∘ −1.4 ∘ ± 𝟑. 𝟖 ∘ Frontal 24.8 ∘ ± 𝟑. 𝟔 ∘ 25.2 ∘ ± 𝟓. 𝟏 ∘ Sideview Di Distance st stdev wi with ears w/ w/o ears Frontal 22cm 125cm Sideview 35cm 35cm 12

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Face Model Fitting Reconstruction: Analysis-by-Synthesis Face Model Rendered Image 𝐽 𝜄 Target Image 𝐽 Parametric face model Likelihood ℓ 𝜄; 𝐽 ∝ 𝑄 𝐽 𝐽 𝜄 𝜄 = 𝜘, 𝛽, 𝛾 : : 𝜘 Scene Parameters, 𝛽 Face shape, 𝛾 Face color 13

� > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Independent Pixels Likelihood Standard choice Corresponds to least squares fitting 𝐺 K = , 𝜏 6 𝐽 7 ) , 𝜏 6 𝐽 7 ) ℓ 𝜄; 𝐽 𝒪( | 𝒪( | ∗ ∗ ⋯ 3 | 𝐽 1 𝜄 , 𝜏 6 𝐽 7 K = / 𝒪 𝐽 1 ℓ 𝜄; 𝐽 14 1∈:

� > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Background Model The face model covers only a small part of the complete target image K = / ℓ 1 𝜄; 𝐽 1 3 ℓ 𝜄; 𝐽 1∈: What to do outside face region? • Ignore → strong artifacts • Explicit model Shrinking Misalignment 15

� � � > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Explicit Background Model Add explicit likelihood for background Arbitrary background: The explicit background model K = / ℓ › 𝜄; 𝐽 1 3 3 ℓ 𝜄; 𝐽 / 𝑐 <= 𝐽 1 needs to be based on generic 1∈: >∈? and simple assumptions: Why is ignoring bad? Constant model K = / ℓ › 𝜄; 𝐽 1 3 ℓ 𝜄; 𝐽 3 = 1 𝑐 <= 𝐽 1 1∈: Histogram model Implicit background model is al alway ays present but might be inappropriate → better make it explicit! Schönborn et al. «Background modeling for generative image models», Computer Vision and Image Understanding, Volume 136, July 2015, Pages 117–127, doi:10.1016/j.cviu.2015.01.008 16

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Collective Likelihood • Independence is not a good assumption Too many observations (100k+): overconfident − Colors are correlated 𝑒 = • Model distribution of image distance Fit to empirical histogram or use model Can be any measure extracted on images ℎ(𝑒) • Most-likely solutions match the image with the expected noise level A perfect reconstruction is unlikely 17

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Posterior Samples: Fitting Result • Model instances with comparable reconstruction quality • Remaining uncertainty of model representation • Integration of uncertain detection directly into model adaptation 0.076 d I <d I > 0.075 RMS Image Distance 0.074 0.073 0.072 0 200000 400000 600000 800000 1e+06 Sample Posterior using collective likelihood 18

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Results: Image Yaw angle: 1.9 ∘ ± 0.2 ∘ 19

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Image: Samples 20

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Posterior Shape Variation Landmarks posterior, Image posterior, sd[mm] sd[mm] 21

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE PROBABILISTIC MORPHABLE MODELS | JUNE 2017 | BASEL Fitting Results LFW AFLW Images from: Huang, Gary B., et al. Labeled faces in the Images from: Köstinger, Martin, et al. "Annotated wild: A database for studying face recognition in facial landmarks in the wild: A large-scale, real-world unconstrained environments . Vol. 1. No. 2. Technical database for facial landmark localization." Computer Report 07-49, University of Massachusetts, Amherst, 2007. Vision Workshops (ICCV Workshops), 2011 IEEE 22 International Conference on . IEEE, 2011.