> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face Image Analysis Applications Probabilistic Morphable Model Fitting Basel2018 Thomas Vetter University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face Identification by Image Comparison … done by pixel analysis ? But which pixel to compare with which ? Shape information tells us which pixel to compare 1
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by Synthesis 3D Image Image Description World model parameter Analysis Image Model Synthesis > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Change Your Image ... 2
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by Synthesis 3D Image Image Description World model parameter Analysis Image Model Synthesis > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE THE BIG QUESTION: How is this Image Model structured? Is it: 2D, an image based rendering model? or 3D, a full 3D computer graphics model? or … . Possibly, there is no final answer! 3
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Linear Object Class Idea Linear Object Classes and Image Synthesis from a Single Example Image. Thomas Vetter and Tomaso Poggio IEEE P AMI 1997, 19(7), 733-742. > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Separating shape and texture in 2D images 4
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE 2D Morphable Face Image Model > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Linear Object Class Idea 5
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Image based rendering 6
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Synthesis of novel views from a single face image. Thomas Vetter , IJCV 1998, 28(2), 103-116. > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Morphable 2D Face Model α 1 𝑆 + α 2 𝑆 + α 3 𝑆 + α 4 𝑆 + … = β 1 𝑆 + β 2 𝑆 + β 3 𝑆 + β 4 𝑆 + … 7
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Morphable 3D Face Model α 1 + α 2 + α 3 + α 4 + ⋯ R β 1 + β 2 + β 3 + β 4 + ⋯ > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Morphable Models for Image Registration α 1 + α 2 + α 3 + ⋯ R β 1 + β 2 + β 3 + ⋯ R = Rendering Function ρ = Parameters for Pose, Illumination, ... Optimization Problem: Find optimal α , β , ρ ! Output 8
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face Recognition 18 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Normalizing for pose, illumination and … ? Shape recovery Shape recovery Illumination inversion Illumination inversion 9
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face recognition Complex Changes in Appearance Images: CMU-PIE database. (2002) > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE 3D Morphable Model 10
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Identification by shape and texture coefficients only Gallery , Model- i i Fitting , Model- i i Fitting , Model- i i Fitting … Test Model- , compare Identity i i Fitting > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face analysis Roger F. asian 0.34 caucasian 0.52 blue eyes 0.19 brown eyes 0.69 wide nose 0.70 male 0.52 mustache 0.13 gaze Hor 20° yaw 34° pitch -8° roll 4° 11
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Multi-PIE: Face recognition 100 90 3DGEM [16] 80 3DMM [17] 3DMM ours [18] 70 60 15° 30° 45° [16] Prabhu et al., “Unconstrained Pose -Invariant Face Recognition using 3D Generic E lastic Models”, PAMI 2011 [17] Schönborn et al., “A Monte Carlo Strategy to Integrate Detection and Model- Based Face Analysis”, GCPR 2013 [18] Egger et al., “ Pose Normalization for Eye Gaze Estimation and Facial Attribute Description ”, GCPR 2014 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Try a new hairstyle! 3D Geomety and Texture 3D Pose, Position Illumination, Foreground, Background 12
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Try a new hairstyle! 3D Geomety and Texture 3D Pose, Position Illumination, Foreground, Background > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Image Preprocessing for FRVT 2002 13
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Image Preprocessing for FRVT 2002 14
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Skin Detail Analysis for Face Recognition Skin Detail Analysis for Face Recognition Jean Sebastian Pierrard , Thomas Vetter CVPR 2007 15
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Overview Characterizing moles Appearance Blob detection Location Skin segmentation Importance Saliency measure Recognition Reference Systsem Morphable Model > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Data used Results based on subset of FERET-data base Gray scale Medium resolution (10-20k pixels face area) Mole sizes: 2-20 pixels 16
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Morphable Model for Correspondence Fitting Correspondence 3D reconst. Fitting > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE 3DMM maps visible region on a common reference Fitting Correspondence 3D reconst. Fitting 17
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Morphable Model for Correspondence II Rendering Fitting 3D reconst. Fitting > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Mole Detection: Shading Problem Template matching is sensitive to intensity gradients ! 18
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Illumination Compensation I x z x ( ), ( ) E ( ) x ic > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Mole Detection: Shading Problem 0.59 cc 0.82 cc 0.56 cc 0.75 cc Local fitting 19
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE False Positives Templates also match common facial features Sporadic hits due to hairstyle, beard, … We need to mask out non-skin regions / outliers 3DMM is not sufficient > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Selection by Saliency 20
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Recognition Find matching pairs of moles in reference frame Identification score: weighted sum of saliencies from matched points > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face Recognition Based only on mole locations and saliency. 21
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Manipulation of Faces Modeler > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Modeling of 2D Images 22
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Face Exchange Tasks Scale & Orientation Color balance & Illumination (3DMM) (3DMM) Source Target Overlay target Remove outliers from Blend artificial occlusions source texture edges (3DMM) Difficult problem, even for humans. Has never be automated ! > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE 23
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Change Your Image ... > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Continuous Modeling in Face Space Caricature Original Average Anti Face 24
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Modeling the Appearance of Faces Which directions code for specific attributes ? > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Learning from Examples 25
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Attributes of Faces Gender Weight Original > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Portraits made to Measure Computer can learn to model faces according to „human“ categories. Aggressive Trustworthy 26
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Portraits made to Measure 100 90 80 Original Face Original Face Original Face 70 % Correct ratings 60 50 40 30 Aggressiveness Aggressiveness Aggressiveness Extroversion Extroversion Extroversion Likeability Likeability Likeability 20 10 0 Aggressiveness Extroversion Likeability Risk Seeking Social Skills Trustworthiness Personality traits Portraits made to measure: Mirella Walker and Thomas Vetter Journal of Vision, 9(11):12, 1-13, 2009 Risk Seeking Risk Seeking Risk Seeking Social Skills Social Skills Social Skills Trustworthiness Trustworthiness Trustworthiness . > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Expressions Original 27
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Simulation of Aging of Human Faces in Images > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Aging model: model predicts perceived age Predicted age 20 years 70 years Labeled / True age 28
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Ageing: linear shape model only > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Example-based aging Target Image Donor Image Shape and Skin of donor transferred to target 29
Recommend
More recommend
