Facial Expression Recognition Ian Endres April 10, 2009 Expression - PowerPoint PPT Presentation

Facial Expression Recognition Ian Endres April 10, 2009

Expression Overview ◮ Given image or video ◮ Identify 6 principal emotions: happiness, sadness, surprise, fear, anger, disgust

Applications ◮ Robotics ◮ User/operator feedback

Method ◮ 1996 - Hand initialization, parametric flow, hand selected predicates, simple hand written rules ◮ 2008 - Single image (Actor Dependent) Dimensionality reduction (using DLLE), SVM classification ◮ 2008 - Motion (Actor Independent) Automatic initialization, keypoint tracking, keypoint flow and displacement, nearest neighbor classification

Initialization ◮ Initialize initial frame, automatically track throughout ◮ Black - Mark box around features, head ◮ Yang ◮ Face detection - model skin with Gaussian color model ◮ Feature detection - Fit model to eye/mouth contours, iris edges

Motion Model (Black) Affine Model u ( x , y ) = a o + a 1 x + a 2 y v ( x , y ) = a 3 + a 4 x + a 5 y divergence = a 1 + a 5 curl = − a 2 + a 4 deformation = a 1 − a 5

Motion Model (Black) Planar, Curvature Models Additional planar terms u : p o x 2 + p 1 xy v : p o xy + p 1 y 2 Additional curvature term: v : cx 2

Model Fitting Write model as ( A ffine, P lanar, C urvature): ◮ Color constancy constraint: I ( x , t ) = I ( x − X ( x ) P , t + 1) ◮ Thus minimize over P : � x ∈ f ρ ( ∇ I · ( X ( x ) P ) + I t , σ ) ◮ Error norm ρ ( x , σ ) is used to ignore outliers

Motion Model (Yang) ◮ Energy Model ◮ Distance from neutral : potential energy ◮ Velocity from optical flow : kinetic energy

Features ◮ Black - Predicates based on flow model parameters Figure: Predicates for motion of 1) Mouth, eyebrows, eyes 2) Head ◮ Yang (Static) - Image embedding in R 2 , each person separate ◮ Yang (Motion) - Energy values for each image point

Learning/Inference ◮ Black - Detect expression change via conjunctions of predicates ◮ Yang (Static) ◮ Train 1v1 polynomial kernel SVM ◮ 20 training examples per expression per person ◮ Inference using these classifiers ◮ Yang (Motion) ◮ “Apex” of expression as region of constant kinetic energy ◮ Label using potential energy states, using exemplar/nearest neighbor method

Learning/Inference (Cont) Figure: 1) Predicate Rules for Black 2) Temporal behavior of expressions

Results (Black)

Results (Yang, Static)

Drawbacks/Limitations ◮ Artifical Datasets ◮ Explicit initialization ◮ Black ◮ No machine learning ◮ Yang ◮ Requires head motion and fast face motions to be avoided ◮ No quantitative results for motion!