Facial Action Unit Detection Using Kernel Partial Least Squares - PowerPoint PPT Presentation

Facial Action Unit Detection Using Kernel Partial Least Squares Tobias Gehrig and Hazım K. Ekenel | November 13, 2011 INSTITUTE FOR ANTHROPOMATICS, FACIAL IMAGE PROCESSING AND ANALYSIS GROUP Institute for Anthropomatics 1 KIT – University of the State of Baden-Wuerttemberg and 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares www.kit.edu FIPA Group National Laboratory of the Helmholtz Association

Motivation Why facial expression analysis? Natural communication of emotions, feelings, opinions, intentions, and cognitive states Affective states communicated faster through faces than with words Presumably better performance in any face analysis task for systems understanding many different facial attributes Institute for Anthropomatics 2 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Motivation Applications (cf. Bartlett and Whitehill 2010 [4]) Human-computer interaction, e.g. mobile service robots [22] Assistance systems for visually impaired or autistic persons [11] Driver safety [20] Online tutoring systems [21] Psychological studies Pain [1] or stress detection Institute for Anthropomatics 3 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Overview Motivation Proposed Approach System Overview Local Appearance-based Face Representation Partial Least Squares Evaluation of the proposed system Evaluation on single Dataset Evaluation across Datasets Evaluation of Non-additive AU Combinations Conclusion and Future Work Institute for Anthropomatics 4 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Overview Motivation Proposed Approach System Overview Local Appearance-based Face Representation Partial Least Squares Evaluation of the proposed system Evaluation on single Dataset Evaluation across Datasets Evaluation of Non-additive AU Combinations Conclusion and Future Work Institute for Anthropomatics 5 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Motivation How can facial expressions be described? Facial Action Coding System (FACS) (Ekman and Friesen 1978 [5]) Upper Face Action Units AU 1 AU 2 AU 4 AU 5 AU 6 AU 7 Inner Brow Raiser Outer Brow Raiser Brow Lowerer Upper Lid Raiser Cheek Raiser Lid Tightener Lower Face Action Units AU 9 AU 10 AU 11 AU 12 AU 15 AU 17 AU 18 Nasolabial Lip Corner Nose Wrinkler Upper Lip Raiser Lip Corner Puller Chin Raiser Lip Puckerer Deepener Depressor AU 20 AU 23 AU 24 AU 25 AU 26 AU 27 Lip Stretcher Lip Tightener Lip Pressor Lips Part Jaw Drop Mouth Stretch Images taken from Tian et al. 2005 [17] Institute for Anthropomatics 6 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Motivation AU Combinations Additive combination: AU 1 AU 2 AU 1+2 Non-additive combination: AU 1 AU 4 AU 1+4 Images taken from Tian et al. 2005 [17] Institute for Anthropomatics 7 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Motivation Applications (cf. Bartlett and Whitehill 2010 [4]) Human-computer interaction , e.g. mobile service robots [22] Assistance systems for visually impaired or autistic persons [11] Driver safety [20] Online tutoring systems [21] Psychological studies Pain [1] or stress detection Institute for Anthropomatics 8 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Design goals Detect additive and non-additive AU combinations Robustness against local appearance changes Real-time processing capability Compact representation Same representation for multiple face classification tasks Institute for Anthropomatics 9 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Related Work AU detection Support Vector Machines (SVM) Jiang et al. 2011 [8] Lucey et al. 2010 [9] Valstar et al. 2011 [19] Gehrig and Ekenel 2011 [6]) Gentle AdaBoost (Zhu et al. 2009 [23]) AdaBoost for feature selection + SVM classifier (Bartlett et al. 2006 [3]) Nearest Neighbor (Lucey et al. 2007 [10]) Neural Network (Tian et al. 2001 [16]) AdaBoost + dynamic Bayesian Network (DBN) (Tong et al. 2007 [18]) Institute for Anthropomatics 10 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Related Work Partial Least Squares Dimensionality reduction for person detection (Schwartz et al. 2009 [14]) Common space for multi-modal face recognition (Sharma and Jacobs 2011 [15]) Face recognition (Schwartz et al. 2010 [13]) Simultaneous age, gender and ethnicity estimation (Guo&Mu 2011 [7]) Institute for Anthropomatics 11 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Overview Motivation Proposed Approach System Overview Local Appearance-based Face Representation Partial Least Squares Evaluation of the proposed system Evaluation on single Dataset Evaluation across Datasets Evaluation of Non-additive AU Combinations Conclusion and Future Work Institute for Anthropomatics 12 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

System Overview 31 8 0 8 AU 1 80 25 10 Coeff. AU i per block AU N 64 MCT-based face eye-based block-based PLS & eye detection alignment DCT Institute for Anthropomatics 13 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

System Overview 31 8 0 8 AU 1 80 25 10 Coeff. AU i per block AU N 64 MCT-based face eye-based block-based PLS & eye detection alignment DCT Institute for Anthropomatics 14 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Local Appearance-based Face Representation based on Discrete Cosine Transform (Ekenel 2009) Each component in the feature vector is divided by its variance. Feature vectors for each block are normalized to have unit norm. All feature vectors for the individual blocks are concatenated to one big feature vector. Institute for Anthropomatics 15 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

System Overview 31 8 0 8 AU 1 80 25 10 Coeff. AU i per block AU N 64 MCT-based face eye-based block-based PLS & eye detection alignment DCT Institute for Anthropomatics 16 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Partial Least Squares (PLS) (cf. Rosipal 2011 [12]) X = TP T + E PLS models: Y = UQ T + F [ cov ( t , u )] 2 = | r | = | s | = 1 [ cov ( Xr , Ys )] 2 Optimization criteria: max Inner relation: U = TD + H ˆ Linear PLS regression estimate: Y = X test B B = X T U ( T T XX T U ) − 1 T T Y Regression matrix X : Input matrix ( n × N ) Y : Output matrix ( n × M ) T : Latent score matrix ( n × p ) U : Latent score matrix ( n × p ) P : Loading matrix ( N × p ) Q : Loading matrix ( M × p ) E : Residual matrix ( n × N ) F : Residual matrix ( n × M ) Institute for Anthropomatics 17 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Kernel Partial Least Squares (cf. Rosipal 2011 [12]) K = ΦΦ T Gram matrix: K i , j = k ( x i , x j ) = Φ ( x i ) T Φ ( x j ) ˆ Y = K test R Kernel PLS estimate: R = U ( T T KU ) − 1 T T Y K test = Φ test Φ T k linear ( x i , x j ) = x T Linear kernel function: i x j k gaussian ( x i , x j ) = exp (( | x i − x j | 2 ) / w ) Gaussian kernel function: Institute for Anthropomatics 18 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Overview Motivation Proposed Approach System Overview Local Appearance-based Face Representation Partial Least Squares Evaluation of the proposed system Evaluation on single Dataset Evaluation across Datasets Evaluation of Non-additive AU Combinations Conclusion and Future Work Institute for Anthropomatics 19 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group

Experimental Setup Leave-one-subject-out (LOSO) cross-validation Metrics: Two alternative forced choice (2AFC) score = area A ′ underneath the receiver-operator characteristic (ROC) curve Upper bound for the uncertainty of the A ′ : � A ′ ( 1 − A ′ ) s = min { n p , n n } Datasets: Constrained dataset: Extended Cohn-Kanade (CK+) dataset Less constrained dataset: GEMEP-FERA Configuration: PLS: 30 latent variables KPLS: 40 latent variables Gaussian kernel: w = 1024 Institute for Anthropomatics 20 13.11.2011 Tobias Gehrig and Hazım K. Ekenel: Facial Action Unit Detection Using Kernel Partial Least Squares FIPA Group