> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face Image Analysis Applications Probabilistic Morphable Models Summer School, June 2017 Thomas Vetter University Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face Identification by Image Comparison … done by pixel analysis ? But which pixel to compare with which ? Shape information tells us which pixel to compare 1
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by Synthesis 3D Image Image Description World model parameter Analysis Image Model Synthesis > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Change Your Image ... 2
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by Synthesis 3D Image Image Description World model parameter Analysis Image Model Synthesis > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE THE BIG QUESTION: How is this Image Model structured? Is it: 2D, an image based rendering model? Or 3D, a full 3D computer graphics model? Possibly, there is no final answer! 3
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Linear Object Class Idea Linear Object Classes and Image Synthesis from a Single Example Image. Thomas Vetter and Tomaso Poggio IEEE P AMI 1997, 19(7), 733-742. > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Separating shape and texture in 2D images 4
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE 2D Morphable Face Image Model > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Linear Object Class Idea 5
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Image based rendering 6
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Synthesis of novel views from a single face image. Thomas Vetter , IJCV 1998, 28(2), 103-116. > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Morphable 2D Face Model α 1 R + α 2 R + α 3 R + α 4 R = β 1 R + β 2 R + β 3 R + β 4 R 7
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Morphable 3D Face Model 1 2 3 4 R 3 1 2 4 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Morphable Models for Image Registration 1 2 3 R = 1 2 3 R = Rendering Function ρ = Parameters for Pose, Illumination, ... Optimization Problem: Find optimal α , β , ρ ! Output 8
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face Recognition 18 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Normalizing for pose, illumination and … ? Shape recovery Shape recovery Illumination inversion Illumination inversion 9
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Multi-Features Fitting Algorithm 1 2 3 4 5 anchor x edge x x x x pixel int. x x x spec. highl. x tex. const. x x prior x x x x > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face recognition Complex Changes in Appearance Images: CMU-PIE database. (2002) 10
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE 3D Morphable Model > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Identification by shape and texture coefficients only Gallery , Model- i i Fitting , Model- i i Fitting , Model- i i Fitting … Test Model- , compare Identity i i Fitting 11
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Correct Identification “1 out of 68” (%) gallery front side profile front 99.8 99.5 83.0 probe side 97.8 99.9 86.2 profile 79.5 85.7 98.3 total 92.3 95.0 89.0 CMU-PIE database: 4488 images of 68 individuals 3 poses x 22 illuminations = 66 images per individual > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face analysis Roger F. asian 0.34 caucasian 0.52 blue eyes 0.19 brown eyes 0.69 wide nose 0.70 male 0.52 mustache 0.13 gaze Hor 20° yaw 34° pitch -8° roll 4° 12
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Multi-PIE: Face recognition 100 90 3DGEM [16] 80 3DMM [17] 3DMM ours [18] 70 60 15° 30° 45° [16] Prabhu et al., “Unconstrained Pose -Invariant Face Recognition using 3D Generic E lastic Models”, PAMI 2011 [17] Schönborn et al., “A Monte Carlo Strategy to Integrate Detection and Model- Based Face Analysis”, GCPR 2013 [18] Egger et al., “ Pose Normalization for Eye Gaze Estimation and Facial Attribute Description ”, GCPR 2014 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Try a new hairstyle! 3D Geomety and Texture 3D Pose, Position Illumination, Foreground, Background 13
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Try a new hairstyle! 3D Geomety and Texture 3D Pose, Position Illumination, Foreground, Background > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Image Preprocessing for FRVT 2002 14
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Image Preprocessing for FRVT 2002 15
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Skin Detail Analysis for Face Recognition Skin Detail Analysis for Face Recognition Jean Sebastian Pierrard , Thomas Vetter CVPR 2007 16
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Overview Characterizing moles Appearance Blob detection Location Skin segmentation Importance Saliency measure Recognition Reference Systsem Morphable Model > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Data used Results based on subset of FERET-data base Gray scale Medium resolution (10-20k pixels face area) Mole sizes: 2-20 pixels 17
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Morphable Model for Correspondence Fitting Correspondence 3D reconst. Fitting > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE 3DMM maps visible region on a common reference Fitting Correspondence 3D reconst. Fitting 18
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Morphable Model for Correspondence II Rendering Fitting 3D reconst. Fitting > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Mole Detection: Shading Problem Template matching is sensitive to intensity gradients ! 19
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Illumination Compensation I x z x ( ), ( ) ( ) E x ic > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Mole Detection: Shading Problem 0.59 cc 0.82 cc 0.75 cc 0.56 cc Local fitting 20
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE False Positives Templates also match common facial features Sporadic hits due to hairstyle, beard, … We need to mask out non-skin regions / outliers 3DMM is not sufficient > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE (Skin) Texture Similarity Basic idea: Compare image texture with samples that are known to be skin tgt src src I N I q i Look-up best tgt N match p tgt src ( ) min seed E p N N I ts p q src seed q N | I q Example 21
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Skin Segmentation Texture similarity facilitates simple segmentation-by-thresholding method 1 if ( ) max ( ) E p E q ts ts Get threshold from in seed skin ( ) q I I p seed region: 0 otherwise Result still affected by shading seed "cheeks" I > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Segmentation Results Thresholding GrabCut 22
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Selection by Saliency > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Recognition Find matching pairs of moles in reference frame Identification score: weighted sum of saliencies from matched points 23
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face Recognition Based only on mole locations and saliency. > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Occlusion-aware 3D Morphable Face Models Bernhard Egger, Andreas Schneider, Clemens Blumer, Andreas Morel-Forster, Sandro Schönborn, Thomas Vetter 27th British Machine Vision Conference, September 2016 24
49 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face Image Analysis under Occlusion Source: AFLW Database Source: AR Face Database > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE There is nothing like: no background model 𝑚 𝑔𝑏𝑑𝑓 (𝜄; ෩ 𝑐(෩ ℓ 𝜄; 𝐽 = ෑ ℓ 𝜄; 𝐽 𝑦 = ෑ 𝐽 𝑗 ) ෑ 𝐽 𝑗` ) 𝑦 ∈ 𝐽 𝑗∈𝐺 𝑗`∈𝐶 “ Background Modeling for Generative Image Models ” Sandro Schönborn, Bernhard Egger, Andreas Forster, and Thomas Vetter Computer Vision and Image Understanding, Vol 113, 2015. 25
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Occlusion-aware Model 𝑨 ∙ 𝑚 𝑜𝑝𝑜−𝑔𝑏𝑑𝑓 𝜄; ෩ 1−𝑨 𝑚 𝜄; ሚ 𝑚 𝑔𝑏𝑑𝑓 𝜄; ෩ 𝐽, 𝑨 = ෑ 𝐽 𝑗 𝐽 𝑗 𝑗 53 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Inference 26
Recommend
More recommend
